<div dir="ltr"><div>Hi,</div><div><br></div><div>Its hard to tell just looking at those examples. </div>The best way to deal with segmentation faults is to report them with all the details necessary to debug them and to try the latest version like Mike suggests, to rule out things already fixed.<div>If you open JIRA for them with the trace like you showed above and all the version details so we can match the code to the trace, then sometimes we can diagnose from that alone.</div><div><br></div><div>If the open source help is not enough for your needs and you want to get a full analysis of your usage footprint and assistance with large scale commercial deployments, y<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ou can also consider working with the pro services team at <a href="https://freeswitch.com" target="_blank">https://freeswitch.com</a> since sometimes it takes additional effort to track down things triggered by massive load.</span></div><div><br></div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 26, 2018 at 3:44 AM, Shaun Stokes <span dir="ltr"><<a href="mailto:shaun.stokes@itec-support.co.uk" target="_blank">shaun.stokes@itec-support.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-427635438174049953divtagdefaultwrapper" dir="ltr">
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
Hi All,</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
<br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt">We've experienced a number of segmentation faults</span><span style="font-size:12pt"> over time with various errors in the core dump and no clear indication of the exact cause
of the problem, we suspect this is load related.</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt"><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt">FreeSWITCH is typically handling more than 100 concurrent sessions but rarely more than 10 session per second, </span>these segmentation faults have even occurred in the early hours
of the morning with less than 10 session at the time which leads us to believe the bottleneck is on the number of simultaneous registrations\subscriptions which is the only variable which is roughly the same at all hours only decreasing and increasing slightly
in the evenings and mornings.</p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt"><br>
</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt">We've created more SIP profiles to distribute registrations to no avail, we're only able to prevent segmentation faults
</span>entirely<span style="font-size:12pt"> by</span><span style="font-size:12pt"> distributing registrations across more servers. The problems seem to occur when we have combination of:</span></p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt">- 10,000 subscriptions and 400 </span>registrations; reducing
<span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">
subscriptions</span> to 5000 resolves the issue.</p>
<p style="margin-top:0px;margin-bottom:0px">- 1200 subscriptions and 950 registrations; reducing registrations to 700 resolves the issue.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">We have a few FreeSWITCH servers which are dedicated for call routing only, these are processing in excess of 400 concurrent sessions and never have any issues.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Our server specs:</p>
<p style="margin-top:0px;margin-bottom:0px">Debian 8.9</p>
<p style="margin-top:0px;margin-bottom:0px"><span style="font-size:12pt">PostgreSQL 9.4 - Optimised for best performance</span><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">FreeSWITCH 1.6.20 - Using PostgreSQL backend</p>
<p style="margin-top:0px;margin-bottom:0px">32GB RAM</p>
<p style="margin-top:0px;margin-bottom:0px">Dual socket Intel Xeon X5670 2.93Ghz - With Hyper threading enabled</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Our plan is to implement SBCs in the future to handle subscriptions and registrations but in the short term we need to understand the limitations of FreeSWITCH, where are the bottlenecks, and are there any other
solutions.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Looking at SIP traces at the time of these segmentation faults there are more than 30 SIP messages per second mostly subscribe, 202, notify, registration and 200. If we look at all packets on the FreeSWITCH SIP
port including Application Data, ACK etc there are hundreds of packets per second.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">We initially suspected the bottleneck could be the FreeSWITCH backend database but moving to PostgreSQL has had no effect.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Has anyone come across similar issues before and\or is able to offer any advice?</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">We're currently looking at two possibilities, a software bottleneck with-in FreeSWITCH, or a bottleneck on the hardware possibly as a result of using dual socket CPUs. We are in the process of enabling node interleaving
on the hardware so we use SUMA (<span>Sufficiently Uniform Memory Access) instead of NUMA (<span>Non-Uniform Memory Access) which should theoretically reduce traffic on the QPI bridge slightly given that FreeSWITCH isn't optimised for NUMA.</span></span></p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Here's an example of some of some of the segmentation faults we've experienced:</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
<br>
</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
<span style="font-family:monospace">Program terminated with signal SIGSEGV, Segmentation fault.
<br>
#0 strlen () at ../sysdeps/x86_64/strlen.S:106 <br>
106 ../sysdeps/x86_64/strlen.<wbr>S: No such file or directory.</span></p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
<span style="font-family:monospace"><span style="font-family:monospace">(gdb) bt <br>
#0 strlen () at ../sysdeps/x86_64/strlen.S:106 <br>
#1 0x00007f92302f697e in __GI___strdup (s=0xecd42c18 <error: Cannot access memory at address 0xecd42c18>) at strdup.c:41
<br>
#2 0x00007f9231849ea4 in switch_ivr_generate_xml_cdr (session=0x7f91dd8146a8, xml_cdr=0x7f900b6da748) at src/switch_ivr.c:2803
<br>
#3 0x00007f91696fc7b0 in my_on_reporting (session=0x7f91dd8146a8) at mod_xml_cdr.c:228
<br>
#4 0x00007f92317801d5 in switch_core_session_reporting_<wbr>state (session=0x7f91dd8146a8) at src/switch_core_state_machine.<wbr>c:938
<br>
#5 0x00007f923177bf9c in switch_core_session_run (session=0x7f91dd8146a8) at src/switch_core_state_machine.<wbr>c:609
<br>
#6 0x00007f9231775737 in switch_core_session_thread (thread=0x7f8ed51f5170, obj=0x7f91dd8146a8) at src/switch_core_session.c:1648
<br>
#7 0x00007f9231775b25 in switch_core_session_thread_<wbr>pool_worker (thread=0x7f8ed51f5170, obj=0x7f8ed51f5000) at src/switch_core_session.c:1711
<br>
#8 0x00007f9231a9bea5 in dummy_worker (opaque=0x7f8ed51f5170) at threadproc/unix/thread.c:151
<br>
#9 0x00007f9230c85064 in start_thread (arg=0x7f900b6db700) at pthread_create.c:309
<br>
#10 0x00007f923035d62d in clone () at ../sysdeps/unix/sysv/linux/<wbr>x86_64/clone.S:111<br>
</span><br>
<br>
</span><span style="font-family:monospace">Program terminated with signal SIGSEGV, Segmentation fault.
<br>
#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:<wbr>66 <br>
66 ../nptl/pthread_mutex_<wbr>lock.c: No such file or directory. <br>
(gdb) bt <br>
#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:<wbr>66 <br>
#1 0x00007fd953bf9db8 in apr_thread_mutex_lock (mutex=0x19101fd8) at locks/unix/thread_mutex.c:92
<br>
#2 0x00007fd95389e663 in switch_mutex_lock (lock=0x19101fd8) at src/switch_apr.c:293
<br>
#3 0x00007fd9538a7f0c in switch_channel_test_flag (channel=0x7fd619101910, flag=CF_THREAD_SLEEPING) at src/switch_channel.c:1568
<br>
#4 0x00007fd9538ed448 in switch_core_session_read_lock (session=0x7fd627d19eb8) at src/switch_core_rwlock.c:91
<br>
#5 0x00007fd9538d78fc in switch_core_session_hupall_<wbr>matching_var_ans (var_name=0x7fd8b70229b1 "cc_member_pre_answer_uuid", <br>
var_val=0x7fd891c86680 "e38c1d71-e760-44bf-af63-<wbr>4b54dcbd01fc", cause=SWITCH_CAUSE_ORIGINATOR_<wbr>CANCEL, type=(SHT_UNANSWERED | SHT_ANSWERED)) at src/switch_core_session.c:231
<br>
#6 0x00007fd8b701eacd in callcenter_function (session=0x7fd90508dbc8, data=0x7fd6149e5a68 "Adjustments@<span style="font-family:monospace;font-size:16px"><a href="http://customerdomain.com" target="_blank">customerdomain.<wbr>com</a></span>") at mod_callcenter.c:3018
<br>
#7 0x00007fd9538df841 in switch_core_session_exec (session=0x7fd90508dbc8, application_interface=<wbr>0x2836ff0, arg=0x7fd6149e5a68 "<a href="mailto:Adjustments@customerdomain.com" target="_blank">Adjustments@customerdomain.<wbr>com</a>")
<br>
at src/switch_core_session.c:2802 <br>
#8 0x00007fd9538defe8 in switch_core_session_execute_<wbr>application_get_flags (session=0x7fd90508dbc8, app=0x7fd6149e5a58 "callcenter", <br>
arg=0x7fd6149e5a68 "Adjustments@<span style="font-family:monospace;font-size:16px"><a href="http://customerdomain.com" target="_blank">customerdomain.<wbr>com</a></span>", flags=0x0) at src/switch_core_session.c:2672
<br>
#9 0x00007fd9538e168b in switch_core_standard_on_<wbr>execute (session=0x7fd90508dbc8) at src/switch_core_state_machine.<wbr>c:353
<br>
#10 0x00007fd9538e31c6 in switch_core_session_run (session=0x7fd90508dbc8) at src/switch_core_state_machine.<wbr>c:650
<br>
#11 0x00007fd9538db737 in switch_core_session_thread (thread=0x7fd9048e9370, obj=0x7fd90508dbc8) at src/switch_core_session.c:1648
<br>
#12 0x00007fd9538dbb25 in switch_core_session_thread_<wbr>pool_worker (thread=0x7fd9048e9370, obj=0x7fd9048e9200) at src/switch_core_session.c:1711
<br>
#13 0x00007fd953c01ea5 in dummy_worker (opaque=0x7fd9048e9370) at threadproc/unix/thread.c:151
<br>
#14 0x00007fd952deb064 in start_thread (arg=0x7fd891c87700) at pthread_create.c:309
<br>
#15 0x00007fd9524c362d in clone () at ../sysdeps/unix/sysv/linux/<wbr>x86_64/clone.S:111<br>
</span><br>
</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
<br>
</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
Thanks,</p>
<p style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt;margin-top:0px;margin-bottom:0px">
Shaun</p>
</div>
<u></u>
<table>
<tbody>
<tr>
<td><u></u></td>
<td valign="top">
<div><a style="font-family:calibri;color:#1b3e94;font-size:11pt">Shaun Stokes - Infrastructure Analyst</a></div>
<div>
<table>
<tbody>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">T : </a></td>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">01453 700713</a></td>
</tr>
<tr>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">E : </a></td>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">shaun.stokes@itec-support.co.<wbr>uk</a></td>
</tr>
<tr>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">W : </a></td>
<td><a style="font-family:calibri;color:#1b3e94;font-size:11pt">www.itec-support.co.uk</a></td>
</tr>
</tbody>
</table>
</div>
</td>
</tr>
</tbody>
</table>
<div><a style="font-family:calibri;color:#1b3e94;font-size:11pt">Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF</a>
<br>
<a style="font-family:calibri;color:#1b3e94;font-size:11pt">Company No. 06908001</a>
</div>
<div><br>
<a style="font-family:calibri;color:Grey;font-size:11pt">CONFIDENTIALITY NOTICE</a>
<br>
<a style="font-family:calibri;color:Grey;font-size:11pt">This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying
or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately.</a>
<br>
<a style="font-family:calibri;color:Grey;font-size:11pt">The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk,
we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.</a>
</div>
</div>
<br>______________________________<wbr>______________________________<wbr>_____________<br>
Professional FreeSWITCH Services<br>
<a href="mailto:sales@freeswitch.com">sales@freeswitch.com</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="https://freeswitch.com/oss" rel="noreferrer" target="_blank">https://freeswitch.com/oss</a><br>
<a href="https://freeswitch.org/confluence" rel="noreferrer" target="_blank">https://freeswitch.org/<wbr>confluence</a><br>
<a href="https://cluecon.com" rel="noreferrer" target="_blank">https://cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.<wbr>freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/<wbr>mailman/listinfo/freeswitch-<wbr>users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.<wbr>freeswitch.org/mailman/<wbr>options/freeswitch-users</a><br>
<a href="https://freeswitch.com" rel="noreferrer" target="_blank">https://freeswitch.com</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Anthony Minessale II</div><div dir="ltr">Founder, FreeSWITCH.</div><div dir="ltr"><div><a href="http://freeswitch.com" target="_blank">http://freeswitch.com</a></div><div><br></div><div><br></div><div><span style="font-size:12.8px"><a href="https://youtu.be/l_hOxzCt6X4" target="_blank">https://youtu.be/l_hOxzCt6X4</a></span><br></div><div><a href="https://www.youtube.com/watch?v=oAxXgyx5jUw" target="_blank">https://www.youtube.com/watch?v=oAxXgyx5jUw</a><br></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div></div></div>
</div>