[Freeswitch-users] FreeSwitch blocked

[Ken Rice]

the problem isnt freeswitch, but your event consumer isnt consuming the events fast enough. you should adjust your side of the equation.

the event queue is what drives everything

Sent from my iPhone

> On Dec 7, 2018, at 01:29, Melek Oktay <melekoktay at gmail.com> wrote:
> 
> Hi,
> 
> After deeply research about this issue, we understand Freeswitch-Core mechanism and three possible solutions for fixing these issue.
> 
> Here the link
> https://stackoverflow.com/questions/53609817/freeswitch-blocked 
> 
> Actually it would be be better third solution supported by Freeswitch, before deliver event to module, it create new thread for not waiting consumer thread (event hendler)
> 
>> On Fri, Dec 22, 2017 at 8:35 AM Melek Oktay <melekoktay at gmail.com> wrote:
>> 
>> Hi,
>> 
>> FreeSwitch software working well in a few days (~3 - 5 days), then new incoming call requests are accepted since FreeSwitch is blocked !! Ongoing calls continue their session, their calls seems not effected, but new calls are not accepted.  I got FreeSwitch snapshot and analyzed it in GDB.
>> 
>> I have 601 therads & most of them are waiting 
>> 
>> Thread 0x7f16bc55f700 (LWP 28544) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> 
>> When i apply "thread apply all bt" in gdb, I see most of the threads try to push events into queue (switch_queue_push )
>> 
>> Thread 600 (Thread 0x7f16bc55f700 (LWP 28544)):
>> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
>> #1  0x00007f180cf9b87d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68
>> #2  0x00007f180cf92dd0 in apr_queue_push (queue=queue at entry=0x7f180db157a8, data=data at entry=0x7f16d3d5ec20) at misc/apr_queue.c:166
>> #3  0x00007f180cc958fb in switch_queue_push (queue=0x7f180db157a8, data=data at entry=0x7f16d3d5ec20) at src/switch_apr.c:1134
>> #4  0x00007f180cd17850 in switch_event_queue_dispatch_event (eventp=0x7f16bc55ec48) at src/switch_event.c:384
>> #5  switch_event_fire_detailed (file=file at entry=0x7f180cfb07ea "src/switch_channel.c", func=func at entry=0x7f180cfb2ba0 <__func__.18348> "switch_channel_perform_set_running_state", line=line at entry=2260, event=event at entry=0x7f16bc55ec48, user_data=user_data at entry=0x0) at src/switch_event.c:1986
>> #6  0x00007f180cc9f118 in switch_channel_perform_set_running_state (channel=0x7f17e3e7de00, state=CS_NEW, file=0x7f180cfbc590 "src/switch_core_state_machine.c", func=<optimized out>, line=543) at src/switch_channel.c:2260
>> #7  0x00007f180ccc87d0 in switch_core_session_run (session=0x7f17e3e7fd28) at src/switch_core_state_machine.c:543
>> #8  0x00007f180ccc36de in switch_core_session_thread (thread=<optimized out>, obj=0x7f17e3e7fd28) at src/switch_core_session.c:1629
>> #9  0x00007f180ccbf47d in switch_core_session_thread_pool_worker (thread=0x7f17e3e9abb0, obj=0x80) at src/switch_core_session.c:1692
>> #10 0x00007f180cfa1910 in dummy_worker (opaque=0x7f17e3e9abb0) at threadproc/unix/thread.c:151
>> #11 0x00007f180c1e0064 in start_thread (arg=0x7f16bc55f700) at pthread_create.c:309
>> #12 0x00007f180b8b862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>> 
>> 
>> More interesting thing is below, when I look up event type, approximately all of  them are "SWITCH_EVENT_CHANNEL_STATE" and switch_queue (i think sofia_module queue is used in this scenario ) become full !!! nelts (number of elements ) and bounds values are equal, and there are 553 (full_waiters) waiters try to push , but no body try to consume it (empty_waiters = 0)
>> 
>> (gdb) print *(switch_queue_t *) 0x7f180db157a8
>> $1 = {
>>   data = 0x7f1805cfe038,
>>   nelts = 50000,
>>   in = 43000,
>>   out = 43000,
>>   bounds = 50000,
>>   full_waiters = 553,
>>   empty_waiters = 0,
>>   one_big_mutex = 0x7f180db157e8,
>>   not_empty = 0x7f180db15838,
>>   not_full = 0x7f180db15890,
>>   terminated = 0
>> }
>> 
>> (gdb) print *(switch_event_t *) 0x7f16d3d5ec20
>> $1 = {
>>   event_id = SWITCH_EVENT_CHANNEL_STATE,
>>   priority = SWITCH_PRIORITY_NORMAL,
>>   owner = 0x0,
>>   subclass_name = 0x0,
>>   headers = 0x7f16d3d5f750,
>>   last_header = 0x7f16d3d601d0,
>>   body = 0x0,
>>   bind_user_data = 0x0,
>>   event_user_data = 0x0,
>>   key = 0,
>>   next = 0x0,
>>   flags = 0
>> }
>> 
>> 
>> Why i am gonna getting this state? 
>> Any thoughts, tips, tricks would be much appreciated.
>> Regards,
>> Angel
> _________________________________________________________________________
> Professional FreeSWITCH Services
> sales at freeswitch.com
> https://freeswitch.com
> 
> Official FreeSWITCH Sites
> https://freeswitch.com/oss
> https://freeswitch.org/confluence
> https://cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> https://freeswitch.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20181208/d9ec4c8c/attachment.html>