<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I’d try out the latest 1.8 release we made at ClueCon. The seg looks familiar but I would have to dig for specifics to be sure.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 26, 2018, at 4:44 AM, Shaun Stokes <<a href="mailto:shaun.stokes@itec-support.co.uk" class="">shaun.stokes@itec-support.co.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="divtagdefaultwrapper" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 12pt;" class="">Hi All,</div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 12pt;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class="">We've experienced a number of segmentation faults</span><span style="font-size: 12pt;" class=""> over time with various errors in the core dump and no clear indication of the exact cause of the problem, we suspect this is load related.</span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class=""><br class=""></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class="">FreeSWITCH is typically handling more than 100 concurrent sessions but rarely more than 10 session per second, </span>these segmentation faults have even occurred in the early hours of the morning with less than 10 session at the time which leads us to believe the bottleneck is on the number of simultaneous registrations\subscriptions which is the only variable which is roughly the same at all hours only decreasing and increasing slightly in the evenings and mornings.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class=""><br class=""></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class="">We've created more SIP profiles to distribute registrations to no avail, we're only able to prevent segmentation faults</span>entirely<span style="font-size: 12pt;" class=""> by</span><span style="font-size: 12pt;" class=""> distributing registrations across more servers. The problems seem to occur when we have combination of:</span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class="">- 10,000 subscriptions and 400 </span>registrations; reducing<span class="Apple-converted-space"> </span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;" class="">subscriptions</span><span class="Apple-converted-space"> </span>to 5000 resolves the issue.</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">- 1200 subscriptions and 950 registrations; reducing registrations to 700 resolves the issue.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">We have a few FreeSWITCH servers which are dedicated for call routing only, these are processing in excess of 400 concurrent sessions and never have any issues.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Our server specs:</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Debian 8.9</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="font-size: 12pt;" class="">PostgreSQL 9.4 - Optimised for best performance</span><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">FreeSWITCH 1.6.20 - Using PostgreSQL backend</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">32GB RAM</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Dual socket Intel Xeon X5670 2.93Ghz - With Hyper threading enabled</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Our plan is to implement SBCs in the future to handle subscriptions and registrations but in the short term we need to understand the limitations of FreeSWITCH, where are the bottlenecks, and are there any other solutions.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Looking at SIP traces at the time of these segmentation faults there are more than 30 SIP messages per second mostly subscribe, 202, notify, registration and 200. If we look at all packets on the FreeSWITCH SIP port including Application Data, ACK etc there are hundreds of packets per second.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">We initially suspected the bottleneck could be the FreeSWITCH backend database but moving to PostgreSQL has had no effect.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Has anyone come across similar issues before and\or is able to offer any advice?</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">We're currently looking at two possibilities, a software bottleneck with-in FreeSWITCH, or a bottleneck on the hardware possibly as a result of using dual socket CPUs. We are in the process of enabling node interleaving on the hardware so we use SUMA (<span class="">Sufficiently Uniform Memory Access) instead of NUMA (<span class="">Non-Uniform Memory Access) which should theoretically reduce traffic on the QPI bridge slightly given that FreeSWITCH isn't optimised for NUMA.</span></span></div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Here's an example of some of some of the segmentation faults we've experienced:</div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 12pt;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 12pt;" class=""><span style="font-family: monospace;" class="">Program terminated with signal SIGSEGV, Segmentation fault.<span class="Apple-converted-space"> </span><br class="">#0 strlen () at ../sysdeps/x86_64/strlen.S:106<span class="Apple-converted-space"> </span><br class="">106 ../sysdeps/x86_64/strlen.S: No such file or directory.</span></div><div style="margin-top: 0px; margin-bottom: 0px; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 12pt;" class=""><span style="font-family: monospace;" class=""><span style="font-family: monospace;" class="">(gdb) bt<span class="Apple-converted-space"> </span><br class="">#0 strlen () at ../sysdeps/x86_64/strlen.S:106<span class="Apple-converted-space"> </span><br class="">#1 0x00007f92302f697e in __GI___strdup (s=0xecd42c18 <error: Cannot access memory at address 0xecd42c18>) at strdup.c:41<span class="Apple-converted-space"> </span><br class="">#2 0x00007f9231849ea4 in switch_ivr_generate_xml_cdr (session=0x7f91dd8146a8, xml_cdr=0x7f900b6da748) at src/switch_ivr.c:2803<span class="Apple-converted-space"> </span><br class="">#3 0x00007f91696fc7b0 in my_on_reporting (session=0x7f91dd8146a8) at mod_xml_cdr.c:228<span class="Apple-converted-space"> </span><br class="">#4 0x00007f92317801d5 in switch_core_session_reporting_state (session=0x7f91dd8146a8) at src/switch_core_state_machine.c:938<span class="Apple-converted-space"> </span><br class="">#5 0x00007f923177bf9c in switch_core_session_run (session=0x7f91dd8146a8) at src/switch_core_state_machine.c:609<span class="Apple-converted-space"> </span><br class="">#6 0x00007f9231775737 in switch_core_session_thread (thread=0x7f8ed51f5170, obj=0x7f91dd8146a8) at src/switch_core_session.c:1648<span class="Apple-converted-space"> </span><br class="">#7 0x00007f9231775b25 in switch_core_session_thread_pool_worker (thread=0x7f8ed51f5170, obj=0x7f8ed51f5000) at src/switch_core_session.c:1711<span class="Apple-converted-space"> </span><br class="">#8 0x00007f9231a9bea5 in dummy_worker (opaque=0x7f8ed51f5170) at threadproc/unix/thread.c:151<span class="Apple-converted-space"> </span><br class="">#9 0x00007f9230c85064 in start_thread (arg=0x7f900b6db700) at pthread_create.c:309<span class="Apple-converted-space"> </span><br class="">#10 0x00007f923035d62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111<br class=""></span><br class=""><br class=""></span><span style="font-family: monospace;" class="">Program terminated with signal SIGSEGV, Segmentation fault.<span class="Apple-converted-space"> </span><br class="">#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66<span class="Apple-converted-space"> </span><br class="">66 ../nptl/pthread_mutex_lock.c: No such file or directory.<span class="Apple-converted-space"> </span><br class="">(gdb) bt<span class="Apple-converted-space"> </span><br class="">#0 __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:66<span class="Apple-converted-space"> </span><br class="">#1 0x00007fd953bf9db8 in apr_thread_mutex_lock (mutex=0x19101fd8) at locks/unix/thread_mutex.c:92<span class="Apple-converted-space"> </span><br class="">#2 0x00007fd95389e663 in switch_mutex_lock (lock=0x19101fd8) at src/switch_apr.c:293<span class="Apple-converted-space"> </span><br class="">#3 0x00007fd9538a7f0c in switch_channel_test_flag (channel=0x7fd619101910, flag=CF_THREAD_SLEEPING) at src/switch_channel.c:1568<span class="Apple-converted-space"> </span><br class="">#4 0x00007fd9538ed448 in switch_core_session_read_lock (session=0x7fd627d19eb8) at src/switch_core_rwlock.c:91<span class="Apple-converted-space"> </span><br class="">#5 0x00007fd9538d78fc in switch_core_session_hupall_matching_var_ans (var_name=0x7fd8b70229b1 "cc_member_pre_answer_uuid", <br class=""> var_val=0x7fd891c86680 "e38c1d71-e760-44bf-af63-4b54dcbd01fc", cause=SWITCH_CAUSE_ORIGINATOR_CANCEL, type=(SHT_UNANSWERED | SHT_ANSWERED)) at src/switch_core_session.c:231<span class="Apple-converted-space"> </span><br class="">#6 0x00007fd8b701eacd in callcenter_function (session=0x7fd90508dbc8, data=0x7fd6149e5a68 "<a href="mailto:Adjustments@customerdomain.com" class="">Adjustments@</a><span style="font-family: monospace; font-size: 16px;" class=""><a href="mailto:Adjustments@customerdomain.com" class="">customerdomain.com</a></span>") at mod_callcenter.c:3018<span class="Apple-converted-space"> </span><br class="">#7 0x00007fd9538df841 in switch_core_session_exec (session=0x7fd90508dbc8, application_interface=0x2836ff0, arg=0x7fd6149e5a68 "<a href="mailto:Adjustments@customerdomain.com" class="">Adjustments@customerdomain.com</a>")<span class="Apple-converted-space"> </span><br class=""> at src/switch_core_session.c:2802<span class="Apple-converted-space"> </span><br class="">#8 0x00007fd9538defe8 in switch_core_session_execute_application_get_flags (session=0x7fd90508dbc8, app=0x7fd6149e5a58 "callcenter", <br class=""> arg=0x7fd6149e5a68 "<a href="mailto:Adjustments@customerdomain.com" class="">Adjustments@</a><span style="font-family: monospace; font-size: 16px;" class=""><a href="mailto:Adjustments@customerdomain.com" class="">customerdomain.com</a></span>", flags=0x0) at src/switch_core_session.c:2672<span class="Apple-converted-space"> </span><br class="">#9 0x00007fd9538e168b in switch_core_standard_on_execute (session=0x7fd90508dbc8) at src/switch_core_state_machine.c:353<span class="Apple-converted-space"> </span><br class="">#10 0x00007fd9538e31c6 in switch_core_session_run (session=0x7fd90508dbc8) at src/switch_core_state_machine.c:650<span class="Apple-converted-space"> </span><br class="">#11 0x00007fd9538db737 in switch_core_session_thread (thread=0x7fd9048e9370, obj=0x7fd90508dbc8) at src/switch_core_session.c:1648<span class="Apple-converted-space"> </span><br class="">#12 0x00007fd9538dbb25 in switch_core_session_thread_pool_worker (thread=0x7fd9048e9370, obj=0x7fd9048e9200) at src/switch_core_session.c:1711<span class="Apple-converted-space"> </span><br class="">#13 0x00007fd953c01ea5 in dummy_worker (opaque=0x7fd9048e9370) at threadproc/unix/thread.c:151<span class="Apple-converted-space"> </span><br class="">#14 0x00007fd952deb064 in start_thread (arg=0x7fd891c87700) at pthread_create.c:309<span class="Apple-converted-space"> </span><br class="">#15 0x00007fd9524c362d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111<br class=""></span><br class=""></div></div></div></blockquote></div><br class=""></body></html>