<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Dec 20, 2018 at 10:05 AM Melek Oktay <<a href="mailto:melekoktay@gmail.com">melekoktay@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="margin-left:40px">
<i>the problem isnt freeswitch, but your event consumer isnt consuming the
events fast enough. you should adjust your side of the equation.<br>the event queue is what drives everything <br></i></div><div><br></div><div>Hi again,</div><div><br></div><div>I think FreeSwitch core mechanism should not be blocked when
user module
consuming thread is not fast enough. Per-module event queue with multi-thread consumer thread mechanism could solve this issue, or something else (i don't know).</div><div>Since user can add its module to the Freeswitch, s/he could not block core of freeswitch since its wrong action i think. <br></div><div><br></div><div>Regards<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sun, Dec 9, 2018 at 1:31 PM <<a href="mailto:freeswitch-users-request@lists.freeswitch.org" target="_blank">freeswitch-users-request@lists.freeswitch.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send FreeSWITCH-users mailing list submissions to<br>
<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:freeswitch-users-request@lists.freeswitch.org" target="_blank">freeswitch-users-request@lists.freeswitch.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:freeswitch-users-owner@lists.freeswitch.org" target="_blank">freeswitch-users-owner@lists.freeswitch.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of FreeSWITCH-users digest..."<br>
Today's Topics:<br>
<br>
1. Re: FreeSwitch blocked (Ken Rice)<br>
<br><br><br>---------- Forwarded message ----------<br>From: Ken Rice <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>><br>To: FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>Cc: <br>Bcc: <br>Date: Sat, 8 Dec 2018 03:03:30 -0600<br>Subject: Re: [Freeswitch-users] FreeSwitch blocked<br><div dir="auto">the problem isnt freeswitch, but your event consumer isnt consuming the events fast enough. you should adjust your side of the equation.<div><br></div><div>the event queue is what drives everything<br><br><div id="gmail-m_6796460693542688838gmail-m_6138720633776133053AppleMailSignature" dir="ltr">Sent from my iPhone</div><div dir="ltr"><br>On Dec 7, 2018, at 01:29, Melek Oktay <<a href="mailto:melekoktay@gmail.com" target="_blank">melekoktay@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div>After deeply research about this issue, we understand <b>Freeswitch-Core</b> mechanism and three possible solutions for fixing these issue.<div><br></div><div>Here the link</div><div><a href="https://stackoverflow.com/questions/53609817/freeswitch-blocked" target="_blank">https://stackoverflow.com/questions/53609817/freeswitch-blocked</a> </div><div><br></div><div>Actually it would be be better third solution supported by <b>Freeswitch</b>, before deliver event to module, it create new thread for not waiting consumer thread (event hendler)</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Dec 22, 2017 at 8:35 AM Melek Oktay <<a href="mailto:melekoktay@gmail.com" target="_blank">melekoktay@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>Hi,</div><div><br></div><div>FreeSwitch software working well in a few days (~3 - 5 days), then new incoming call requests are accepted since FreeSwitch is blocked !! Ongoing calls continue their session, their calls seems not effected, but new calls are not accepted. I got FreeSwitch snapshot and analyzed it in GDB.</div><div><br></div><div>I have 601 therads & most of them are waiting </div><div><br></div><div><font face="monospace, monospace">Thread 0x7f16bc55f700 (LWP 28544) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185</font><br></div><div><br></div><div>When i apply "<b>thread apply all bt</b>" in gdb, I see most of the threads try to push events into queue (<b style="font-family:monospace,monospace">switch_queue_push </b>)</div><div><br></div><div><div><font face="monospace, monospace">Thread 600 (Thread 0x7f16bc55f700 (LWP 28544)):</font></div><div><font face="monospace, monospace">#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185</font></div><div><font face="monospace, monospace">#1 0x00007f180cf9b87d in apr_thread_cond_wait (cond=<optimized out>, mutex=<optimized out>) at locks/unix/thread_cond.c:68</font></div><div><font face="monospace, monospace">#2 0x00007f180cf92dd0 in apr_queue_push (queue=queue@entry=0x7f180db157a8, data=data@entry=0x7f16d3d5ec20) at misc/apr_queue.c:166</font></div><div><font face="monospace, monospace">#3 0x00007f180cc958fb in <b>switch_queue_push </b>(queue=<font color="#0000ff"><b>0x7f180db157a8</b></font>, data=data@entry=<b><font color="#ff0000">0x7f16d3d5ec20</font></b>) at src/switch_apr.c:1134</font></div><div><font face="monospace, monospace">#4 0x00007f180cd17850 in switch_event_queue_dispatch_event (eventp=0x7f16bc55ec48) at src/switch_event.c:384</font></div><div><font face="monospace, monospace">#5 switch_event_fire_detailed (file=file@entry=0x7f180cfb07ea "src/switch_channel.c", func=func@entry=0x7f180cfb2ba0 <__func__.18348> "switch_channel_perform_set_running_state", line=line@entry=2260, event=event@entry=0x7f16bc55ec48, user_data=user_data@entry=0x0) at src/switch_event.c:1986</font></div><div><font face="monospace, monospace">#6 0x00007f180cc9f118 in switch_channel_perform_set_running_state (channel=0x7f17e3e7de00, state=CS_NEW, file=0x7f180cfbc590 "src/switch_core_state_machine.c", func=<optimized out>, line=543) at src/switch_channel.c:2260</font></div><div><font face="monospace, monospace">#7 0x00007f180ccc87d0 in switch_core_session_run (session=0x7f17e3e7fd28) at src/switch_core_state_machine.c:543</font></div><div><font face="monospace, monospace">#8 0x00007f180ccc36de in switch_core_session_thread (thread=<optimized out>, obj=0x7f17e3e7fd28) at src/switch_core_session.c:1629</font></div><div><font face="monospace, monospace">#9 0x00007f180ccbf47d in switch_core_session_thread_pool_worker (thread=0x7f17e3e9abb0, obj=0x80) at src/switch_core_session.c:1692</font></div><div><font face="monospace, monospace">#10 0x00007f180cfa1910 in dummy_worker (opaque=0x7f17e3e9abb0) at threadproc/unix/thread.c:151</font></div><div><font face="monospace, monospace">#11 0x00007f180c1e0064 in start_thread (arg=0x7f16bc55f700) at pthread_create.c:309</font></div><div><font face="monospace, monospace">#12 0x00007f180b8b862d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111</font></div></div><div><br></div><div><br></div><div>More interesting thing is below, when I look up event type, approximately all of them are "<span style="font-family:monospace,monospace">SWITCH_EVENT_CHANNEL_STAT</span><font face="arial, helvetica, sans-serif">E" and switch_queue (i think sofia_module queue is used in this scenario ) <b>become full</b> !!! <b>nelts</b> (number of elements ) and <b>bounds </b>values are equal, and there are 553 (full_waiters) waiters try to push , but no body try to consume it (empty_waiters = 0)</font></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">(gdb) print *(switch_queue_t *) <font color="#0000ff"><b>0x7f180db157a8</b></font></font></div><div><font face="monospace, monospace">$1 = {</font></div><div><font face="monospace, monospace"> data = 0x7f1805cfe038,</font></div><div><font face="monospace, monospace"> nelts = 50000,</font></div><div><font face="monospace, monospace"> in = 43000,</font></div><div><font face="monospace, monospace"> out = 43000,</font></div><div><font face="monospace, monospace"> bounds = 50000,</font></div><div><font face="monospace, monospace"> full_waiters = 553,</font></div><div><font face="monospace, monospace"> empty_waiters = 0,</font></div><div><font face="monospace, monospace"> one_big_mutex = 0x7f180db157e8,</font></div><div><font face="monospace, monospace"> not_empty = 0x7f180db15838,</font></div><div><font face="monospace, monospace"> not_full = 0x7f180db15890,</font></div><div><font face="monospace, monospace"> terminated = 0</font></div><div><font face="monospace, monospace">}</font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">(gdb) print *(switch_event_t *) <b><font color="#ff0000">0x7f16d3d5ec20</font></b></font></div><div><font face="monospace, monospace">$1 = {</font></div><div><font face="monospace, monospace"> event_id = SWITCH_EVENT_CHANNEL_STATE,</font></div><div><font face="monospace, monospace"> priority = SWITCH_PRIORITY_NORMAL,</font></div><div><font face="monospace, monospace"> owner = 0x0,</font></div><div><font face="monospace, monospace"> subclass_name = 0x0,</font></div><div><font face="monospace, monospace"> headers = 0x7f16d3d5f750,</font></div><div><font face="monospace, monospace"> last_header = 0x7f16d3d601d0,</font></div><div><font face="monospace, monospace"> body = 0x0,</font></div><div><font face="monospace, monospace"> bind_user_data = 0x0,</font></div><div><font face="monospace, monospace"> event_user_data = 0x0,</font></div><div><font face="monospace, monospace"> key = 0,</font></div><div><font face="monospace, monospace"> next = 0x0,</font></div><div><font face="monospace, monospace"> flags = 0</font></div><div><font face="monospace, monospace">}</font></div></div><div><br></div><div><br></div><div>Why i am gonna getting this state? </div><div><pre style="white-space:pre-wrap"><font face="arial, helvetica, sans-serif">Any thoughts, tips, tricks would be much appreciated.<br></font></pre><pre style="white-space:pre-wrap"><font face="arial, helvetica, sans-serif">Regards,</font></pre><pre style="white-space:pre-wrap"><font face="arial, helvetica, sans-serif">Angel</font></pre></div></div>
</blockquote></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_________________________________________________________________________</span><br><span>Professional FreeSWITCH Services</span><br><span><a href="mailto:sales@freeswitch.com" target="_blank">sales@freeswitch.com</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span><br><span></span><br><span>Official FreeSWITCH Sites</span><br><span><a href="https://freeswitch.com/oss" target="_blank">https://freeswitch.com/oss</a></span><br><span><a href="https://freeswitch.org/confluence" target="_blank">https://freeswitch.org/confluence</a></span><br><span><a href="https://cluecon.com" target="_blank">https://cluecon.com</a></span><br><span></span><br><span>FreeSWITCH-users mailing list</span><br><span><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a></span><br><span><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a></span><br><span>UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a></span><br><span><a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a></span></div></blockquote></div></div>_______________________________________________<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</blockquote></div>
</blockquote></div></div>