[Freeswitch-users] FreeSWITCH Stops Responding

Wed Sep 1 22:05:35 UTC 2021

Here are two originate samples.

https://pastebin.com/YKRKrJLu

https://pastebin.com/TEX3yfX6

On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian at freeswitch.com> wrote:

> Can you try to create one where this happens without all this mess?
>
> On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary at ytel.com> wrote:
>
>> Here is the LUA code regarding creating sessions and bridging. It is
>> broken up into 5 sections.
>> The top part is the common functions, and the 4 sections below are
>> different call scenarios.
>> https://pastebin.com/qqGw3tms
>>
>>
>> On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian at freeswitch.com> wrote:
>>
>>> I you do sessions from your lua, I need to see how you do that.
>>>
>>> On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary at ytel.com> wrote:
>>>
>>>> Yes, it bridges the inbound call to an outbound one.
>>>>
>>>> (ProxyOutbound.luac)https://pastebin.com/RDUMy473
>>>>
>>>>
>>>> On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian at freeswitch.com>
>>>> wrote:
>>>>
>>>>> Guessing you create sessions in lua? ProxyOutbound.luac?
>>>>>
>>>>> On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian at freeswitch.com>
>>>>> wrote:
>>>>>
>>>>>> and to make sure when you uuid kill one it says it didn't exist but
>>>>>> show channels had it still listed correct?
>>>>>>
>>>>>> On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary at ytel.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have got the logs for a few calls that were stuck.
>>>>>>> (Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
>>>>>>> https://pastebin.com/NTnte7J7
>>>>>>>
>>>>>>> (Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
>>>>>>> https://pastebin.com/sAhFiGjY
>>>>>>>
>>>>>>> (Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
>>>>>>> https://pastebin.com/bmcdjzAr
>>>>>>>
>>>>>>> (Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
>>>>>>> https://pastebin.com/aGyY3m9F
>>>>>>>
>>>>>>> (Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)
>>>>>>> https://pastebin.com/0cnQaRhC
>>>>>>>
>>>>>>> (Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
>>>>>>> https://pastebin.com/atX0VXdZ
>>>>>>>
>>>>>>> Please let us know if there is anything else we can provide that
>>>>>>> would be useful. Thank you.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian at freeswitch.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Check the UUID of the session in question, and extract its logs,
>>>>>>>> and let's diagnose it
>>>>>>>>
>>>>>>>> On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary at ytel.com> wrote:
>>>>>>>>
>>>>>>>>>  Yes, that is what we are seeing when running uuid_kill.  We
>>>>>>>>> cannot "replicate" the issue, but it happens multiple times a day for us
>>>>>>>>> across our servers.   We have looked through logs across multiple
>>>>>>>>> systems, taken SIP traces to see if that showed anything.  Last
>>>>>>>>> we got a Core dump of the running process after the failure. Nothing seems
>>>>>>>>> to stand out as the cause.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Are there any other debug steps you can think of or what other
>>>>>>>>> information to look at?  We could capture all traffic on the server, or
>>>>>>>>> recompile with special debug flags whatever it is that might help.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Steps we have taken:
>>>>>>>>>
>>>>>>>>> sofia loglevel all 9
>>>>>>>>>
>>>>>>>>> sofia global siptrace on
>>>>>>>>>
>>>>>>>>> fsctl loglevel 7
>>>>>>>>>
>>>>>>>>> # Set environment variable
>>>>>>>>> TPORT_LOG=1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian at freeswitch.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> It's an issue we've been searching for, if you uuid_kill it, and
>>>>>>>>>> it says no session, it only means the record was NOT removed from the
>>>>>>>>>> sqlite database.
>>>>>>>>>>
>>>>>>>>>> Nobody can seem to replicate the steps required to get into that
>>>>>>>>>> condition, maybe you could look thru the logs and see if you can replicate
>>>>>>>>>> what ever happened on that lingering UUID
>>>>>>>>>>
>>>>>>>>>> /b
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary at ytel.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>  We recently started having a problem with our FreeSWITCH
>>>>>>>>>>> servers where calls are working well, with about 200 calls operating
>>>>>>>>>>> sometimes for a couple of days, and then something happens and all call
>>>>>>>>>>> progress stops. Then any new inbound calls that come into the server get
>>>>>>>>>>> stuck in initial status, new inbound get stuck at RINGING. This happens
>>>>>>>>>>> until the max sessions are hit and of course nothing else works from there.
>>>>>>>>>>> Our thought is something is creating a lock and stopping anything else from
>>>>>>>>>>> progressing.
>>>>>>>>>>>  We are seeing the stuck sessions using "show channels". Doing a
>>>>>>>>>>> uuid_kill on the calls seems to make no difference to the output of "show
>>>>>>>>>>> channels", but we have noticed in the logs that some events do happen to
>>>>>>>>>>> some of the calls.
>>>>>>>>>>>  We had ran into a similar issue when writing a custom C module
>>>>>>>>>>> and made some mistakes with session_rwlock etc. But in this case, all of
>>>>>>>>>>> this is using LUA.
>>>>>>>>>>> Some general information:
>>>>>>>>>>> FreeSWITCH Versions 1.10.6 and 1.10.1
>>>>>>>>>>> Modules used:
>>>>>>>>>>> LUA, xml_curl(dialplan), event_socket,
>>>>>>>>>>>
>>>>>>>>>>>  Most of this is just application playing audio files and or
>>>>>>>>>>> creating a new call to bridge to.
>>>>>>>>>>>
>>>>>>>>>>>  Really looking for some help on what debugging we can enable to
>>>>>>>>>>> help identify what is getting stuck to identify where a bug might be on our
>>>>>>>>>>> usage or possible issue with FreeSWITCH.
>>>>>>>>>>>
>>>>>>>>>>>  We have captured a core dump of the running process when it got
>>>>>>>>>>> into the bad state as well, if that is helpful, we can send it in.
>>>>>>>>>>>
>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>
>>>>>>>>>>> The FreeSWITCH project is sponsored by SignalWire
>>>>>>>>>>> https://signalwire.com
>>>>>>>>>>> Enhance your FreeSWITCH install with disruptive priced SMS and
>>>>>>>>>>> PSTN services.
>>>>>>>>>>> Build your next product on our scalable cloud platform.
>>>>>>>>>>>
>>>>>>>>>>> Join our online community to chat in real time
>>>>>>>>>>> https://signalwire.community
>>>>>>>>>>>
>>>>>>>>>>> Professional FreeSWITCH Services
>>>>>>>>>>> sales at freeswitch.com
>>>>>>>>>>> https://freeswitch.com
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> https://freeswitch.com/oss
>>>>>>>>>>> https://freeswitch.org/confluence
>>>>>>>>>>> https://cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> https://freeswitch.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>>>
>>>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI
>>>>>>>>>> 53045
>>>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>>>
>>>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>>>
>>>>>>>>>> Mobile: 918-424-9378
>>>>>>>>>>
>>>>>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>>>>>
>>>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Brian West | Co-founder and Developer
>>>>>>>>
>>>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>>>
>>>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>>>
>>>>>>>> Email: brian at freeswitch.com
>>>>>>>>
>>>>>>>> Mobile: 918-424-9378
>>>>>>>>
>>>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>>>
>>>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Brian West | Co-founder and Developer
>>>>>>
>>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>>
>>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>>
>>>>>> Email: brian at freeswitch.com
>>>>>>
>>>>>> Mobile: 918-424-9378
>>>>>>
>>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>>
>>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Brian West | Co-founder and Developer
>>>>>
>>>>> Need Commercial support? email sales at freeswitch.com
>>>>>
>>>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>>>
>>>>> Email: brian at freeswitch.com
>>>>>
>>>>> Mobile: 918-424-9378
>>>>>
>>>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>>>
>>>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>>>> <https://www.facebook.com/freeswitch> [image:
>>>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>>>
>>>>
>>>
>>> --
>>>
>>> Brian West | Co-founder and Developer
>>>
>>> Need Commercial support? email sales at freeswitch.com
>>>
>>> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
>>> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>>>
>>> Email: brian at freeswitch.com
>>>
>>> Mobile: 918-424-9378
>>>
>>> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>>>
>>> [image: https://www.facebook.com/signalwireinc?src=email]
>>> <https://www.facebook.com/freeswitch> [image:
>>> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>>>
>>
>
> --
>
> Brian West | Co-founder and Developer
>
> Need Commercial support? email sales at freeswitch.com
>
> FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
> <https://maps.google.com/?q=17345+Civic+Drive+%232531+Brookfield,+WI+53045&entry=gmail&source=g>
>
> Email: brian at freeswitch.com
>
> Mobile: 918-424-9378
>
> Website: https://www.FreeSWITCH.com <https://www.freeswitch.com/>
>
> [image: https://www.facebook.com/signalwireinc?src=email]
> <https://www.facebook.com/freeswitch> [image:
> https://twitter.com/freeswitch] <https://twitter.com/freeswitch>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20210901/764cde1c/attachment-0001.html>