Hello All, we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ... freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 (sofia/internal/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid) Locked, Waiting on external entities ... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting? Thanks and regards Markus
If you take a gcore it'll generate a core dump. You could then dig into it with gdb to find the thread for that channel and see what lock it's waiting on. However the gcore will pause the process for a while, so it will impact on any other calls on that box. Have you tried reproducing it on 1.8? On Tue, 9 Oct 2018 at 20:36, Markus Bönke <mbodbg@gmx.net> wrote:
Hello All,
we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ... freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 ( sofia/internal/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid) Locked, Waiting on external entities ... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting?
Thanks and regards
Markus _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
So far I found out that the channel was kept open, because our ESL application never received a CHANNEL_HANGUP and CHANNEL_HANGUP_COMPLETE event. I’m running an ESL trace with tshark now. If it happens again I can see if those events are really not send by FS or they get lost in our app. If it turns out that FS sometimes is not sending the events we will upgrade to 1.8.2. Thanks and regards Markus
Am 10.10.2018 um 14:03 schrieb Steven Ayre <steveayre@gmail.com>:
If you take a gcore it'll generate a core dump. You could then dig into it with gdb to find the thread for that channel and see what lock it's waiting on. However the gcore will pause the process for a while, so it will impact on any other calls on that box.
Have you tried reproducing it on 1.8?
On Tue, 9 Oct 2018 at 20:36, Markus Bönke <mbodbg@gmx.net <mailto:mbodbg@gmx.net>> wrote: Hello All,
we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ... freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 (sofia/internal/anonymous@anonymous.invalid <mailto:sofia/internal/anonymous@anonymous.invalid>) State Change CS_REPORTING -> CS_DESTROY freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid <mailto:sofia/internal/anonymous@anonymous.invalid>) Locked, Waiting on external entities ... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting?
Thanks and regards
Markus _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/>
Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Hi, The message "Locked, Waiting on external entities" is not alarming by itself as it is always logged on hangup. But it should be followed by: [NOTICE] switch_core_session.c:1683 Session XXX (sofia/internal/YYY) Ended [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/YYY [CS_DESTROY] If it is not the case then, you have an app or a module that is holding a lock on the session and preventing its release. And this happens after the reporting state i.e. after the hangup_complete event and CDR posting. So there may be other causes for your missing events. Best Regards, -- Vallimamod Abdullah SIP Solutions vma@sip.solutions linkedin.com/in/vallimamod .
On 10 Oct 2018, at 15:54, Markus Bönke <mbodbg@gmx.net> wrote:
So far I found out that the channel was kept open, because our ESL application never received a CHANNEL_HANGUP and CHANNEL_HANGUP_COMPLETE event. I’m running an ESL trace with tshark now. If it happens again I can see if those events are really not send by FS or they get lost in our app. If it turns out that FS sometimes is not sending the events we will upgrade to 1.8.2.
Thanks and regards
Markus
Am 10.10.2018 um 14:03 schrieb Steven Ayre <steveayre@gmail.com>:
If you take a gcore it'll generate a core dump. You could then dig into it with gdb to find the thread for that channel and see what lock it's waiting on. However the gcore will pause the process for a while, so it will impact on any other calls on that box.
Have you tried reproducing it on 1.8?
On Tue, 9 Oct 2018 at 20:36, Markus Bönke <mbodbg@gmx.net> wrote: Hello All,
we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ... freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 (sofia/internal/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid) Locked, Waiting on external entities ... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting?
Thanks and regards
Markus _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
It;s not that it was never received, it was probably deadlocked and never sent. /b On Mon, Oct 15, 2018 at 7:46 AM Vallimamod Abdullah <vma@vallimamod.org> wrote:
Hi,
The message "Locked, Waiting on external entities" is not alarming by itself as it is always logged on hangup. But it should be followed by:
[NOTICE] switch_core_session.c:1683 Session XXX (sofia/internal/YYY) Ended [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/YYY [CS_DESTROY]
If it is not the case then, you have an app or a module that is holding a lock on the session and preventing its release.
And this happens after the reporting state i.e. after the hangup_complete event and CDR posting. So there may be other causes for your missing events.
Best Regards, -- Vallimamod Abdullah SIP Solutions vma@sip.solutions linkedin.com/in/vallimamod .
On 10 Oct 2018, at 15:54, Markus Bönke <mbodbg@gmx.net> wrote:
So far I found out that the channel was kept open, because our ESL application never received a CHANNEL_HANGUP and CHANNEL_HANGUP_COMPLETE event. I’m running an ESL trace with tshark now. If it happens again I can see if those events are really not send by FS or they get lost in our app. If it turns out that FS sometimes is not sending the events we will upgrade to 1.8.2.
Thanks and regards
Markus
Am 10.10.2018 um 14:03 schrieb Steven Ayre <steveayre@gmail.com>:
If you take a gcore it'll generate a core dump. You could then dig into it with gdb to find the thread for that channel and see what lock it's waiting on. However the gcore will pause the process for a while, so it will impact on any other calls on that box.
Have you tried reproducing it on 1.8?
On Tue, 9 Oct 2018 at 20:36, Markus Bönke <mbodbg@gmx.net> wrote: Hello All,
we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ...
freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 (sofia/internal/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY
freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid) Locked, Waiting on external entities
... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting?
Thanks and regards
Markus
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Brian West | Co-founder and Developer Need Commercial support? email sales@freeswitch.com FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045 <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g> Email: brian@freeswitch.com Mobile: 918-424-9378 Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/> [image: https://www.facebook.com/signalwireinc?src=email] <https://www.facebook.com/freeswitch> [image: https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
As there is no support for 1.6 anymore, we prepare to upgrade to 1.8.2. However in the 1.6 freeswitch logs there was nothing special to see, I only saw in the ESL trace that the CHANNEL_HANGLUP_COMPLETE event was missing. If we reproduce this on 1.8.2 is there anything special we should configure to better analyze if there is a deadlock? Best regards Markus
Am 22.10.2018 um 21:01 schrieb Brian West <brian@freeswitch.com>:
It;s not that it was never received, it was probably deadlocked and never sent.
/b
On Mon, Oct 15, 2018 at 7:46 AM Vallimamod Abdullah <vma@vallimamod.org <mailto:vma@vallimamod.org>> wrote: Hi,
The message "Locked, Waiting on external entities" is not alarming by itself as it is always logged on hangup. But it should be followed by:
[NOTICE] switch_core_session.c:1683 Session XXX (sofia/internal/YYY) Ended [NOTICE] switch_core_session.c:1687 Close Channel sofia/internal/YYY [CS_DESTROY]
If it is not the case then, you have an app or a module that is holding a lock on the session and preventing its release.
And this happens after the reporting state i.e. after the hangup_complete event and CDR posting. So there may be other causes for your missing events.
Best Regards, -- Vallimamod Abdullah SIP Solutions vma@sip.solutions linkedin.com/in/vallimamod <http://linkedin.com/in/vallimamod> .
On 10 Oct 2018, at 15:54, Markus Bönke <mbodbg@gmx.net <mailto:mbodbg@gmx.net>> wrote:
So far I found out that the channel was kept open, because our ESL application never received a CHANNEL_HANGUP and CHANNEL_HANGUP_COMPLETE event. I’m running an ESL trace with tshark now. If it happens again I can see if those events are really not send by FS or they get lost in our app. If it turns out that FS sometimes is not sending the events we will upgrade to 1.8.2.
Thanks and regards
Markus
Am 10.10.2018 um 14:03 schrieb Steven Ayre <steveayre@gmail.com <mailto:steveayre@gmail.com>>:
If you take a gcore it'll generate a core dump. You could then dig into it with gdb to find the thread for that channel and see what lock it's waiting on. However the gcore will pause the process for a while, so it will impact on any other calls on that box.
Have you tried reproducing it on 1.8?
On Tue, 9 Oct 2018 at 20:36, Markus Bönke <mbodbg@gmx.net <mailto:mbodbg@gmx.net>> wrote: Hello All,
we are running freeswitch 1.6.20, calls are controlled via ESL, CDRs are written with mod_xml_cdrl. Sometimes we see „hanging“ channels. In such a case the CDR via mod_xml_cdr is written and the last log entry for such a call is "Locked, Waiting on external entities“. ... freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_state_machine.c:610 (sofia/internal/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY freeswitch.log.2018-10-09-10-19-07.1:db2908e3-c06b-4293-88b1-e18cffb66263 2018-10-09 10:18:15.197345 [DEBUG] switch_core_session.c:1665 Session 166111 (sofia/internal/anonymous@anonymous.invalid) Locked, Waiting on external entities ... How can I proceed to further analyze the problem ? In the last log line I also see the session number (Session 166111) - is there a way to find out on which external entity it is waiting?
Thanks and regards
Markus _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/>
Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/>
Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/>
Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>
_________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/>
Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>
--
Brian West | Co-founder and Developer Need Commercial support? email sales@freeswitch.com <mailto:sales@freeswitch.com> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045 <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g> Email: brian@freeswitch.com <mailto:brian@freeswitch.com> Mobile: 918-424-9378 Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/> <https://www.facebook.com/freeswitch> <https://twitter.com/freeswitch> _________________________________________________________________________ Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
participants (4)
-
Brian West
-
Markus Bönke
-
Steven Ayre
-
Vallimamod Abdullah