<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Tom,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>This sounds very interesting. I’d
like to know a few things:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>First, if you disable the call recording
on the receiving end, will you still get the segfaults consistently? Just
curious to see what the receiver does if it only answers the calls and then
hangs up without the extra burden of recording the audio streams.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Second, how hard would it be to have the
originator and the receiver trade places? If you could, I’d like to
see the two machines switch roles, so that the machine currently acting as the
receiver makes the calls and the machine acting as the originator will now
receive calls. I’m wondering what will happen – will the
segfaults stay at the same machine or will they go over to the new receiver, or
will they go away altogether…? <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I know those are kinda brute force
suggestions but they might yield some interesting information:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>If the segfaults occur only on one
machine, regardless of whether it’s making or receiving calls then
obviously there’s something up with that machine.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>If the segfaults always occur at the
machine receiving the calls then, of course, we’ve got a more interesting
issue.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Would you mind putting your setup info,
scripts, etc. in to the pastebin? Maybe others could try to replicate
your symptoms and see what shakes out.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks for taking the initiative to do
this kind of testing. It will definitely help FS be a better, more stable
product.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>-MC<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>P.S. – I just saw Brian’s
emails on this thread, so be sure to check his suggestions as well!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'>
freeswitch-users-bounces@lists.freeswitch.org
[mailto:freeswitch-users-bounces@lists.freeswitch.org] <b><span
style='font-weight:bold'>On Behalf Of </span></b>tuhl@ix.netcom.com<br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, November 29, 2007
12:54 PM<br>
<b><span style='font-weight:bold'>To:</span></b>
freeswitch-users@lists.freeswitch.org<br>
<b><span style='font-weight:bold'>Subject:</span></b> [Freeswitch-users]
Capacity testing, seg fault</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Hi, <br>
<br>
I'm running some capacity tests on Freeswitch and can cause seg-faults fairly
quickly (<1 minute) at a 'light' load of 10 call originations per second.
Core dump backtrace is at the bottom, and my debugging shows what looks like
corrupted js_session. I'll open an issue on JIRA. I wanted to get opinions on
whether this is a valid architecture for testing capacity, and whether I'm
making a simple mistake.<br>
<br>
<b><span style='font-weight:bold'>Environment:<br>
</span></b>I have the trunk version installed on 2 servers and have one server
(the originator) calling the other (the receiver) using SIP, g711, with a Gig-E
ethernet switch between them. The originating server basically does a
session.originate, waitForAnswer, streamFile (10-second 8khz .wav), and hangup.
The receiver does a session.answer and a recordFile to a .wav file so I can go
back and check voice quality.<br>
<br>
My capacity testing engine is a Perl script which is using the RPC XML interface
to originate the calls on Freeswitch (I submit requests to do a 'jsrun play.js'
to a certain phone number, where play.js is a simple script which originates,
waitsforanswer, streamfile, hangup) . I can configure it to make a certain
number of originations per second and a certain number of total calls. I have
no Perl script running on the receiver - I just setup the dialplan to call a
.js which answers the call and records it. This testing setup is at an early
stage, so right now, to check pass/fail, I just verify that if I ran a
1000-call test on the originator, there should be 1000 .wav files that are all
about the same size, on the receiver at the end of the test, and no crashes.<br>
<br>
I've compiled with debug flags on, and I've set all *_DEBUG flags to 9 (I have
also run the tests after a recompile with debug flags off/0, and that didn't
make any difference). I've done all the ulimit commands that were in the last
few emails on this list. I'm running on FC6 on a Dell 2850 with dual 3.6ghz
Xeons (/proc/cpuinfo shows 4 processors), and 4G RAM. I've set max-sessions to
3000 and Session Rate to 100. 'top' is showing freeswitch at 60-80% on the
receiver during this test.<br>
<br>
<b><span style='font-weight:bold'>ISSUE: <br>
</span></b>My problem right now is on the receiver, which I wouldn't care about
because I'm most interested in the origination capacity of freeswitch, but with
my receiver crashing so quickly, I can't push the originator very hard. I setup
my originating engine to make 1000 total calls at 10 call originations per
second, each call lasting 10 seconds (which results in about 120 simultaneous
channels in use), and I get a seg fault on the receiver within about 500 calls
or 50 seconds. If I run a test with 1000 total calls at 6 call originations per
second, it will work, but if I run an overnight test with 20,000 total calls at
6 call originations per second, the receiver will sometimes seg-fault at around
15,000 calls, and sometimes it will not. Interestingly though, if I do a very
short but very high rate test of 100 total calls at 50 call originations per
second, that will usually work. But 500 total calls at 50 calls per second will
always seg-fault.<br>
<br>
Just so you know... I'm shooting for the holy grail of stable operation at 100
call originations per second. I know people have reported much better results
than I'm getting. Is something in my setup bad?<br>
<br>
Here's the core dump backtrace. I added some debug printf's in session_destroy
right before the call to destroy_speech_engine, and it looks like the jss has
been trampled - for example, jss->flags is always 0 for all my successful
calls, but right before it seg-faults, jss->flags is some large random
number. This happens every single time.<br>
<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00000000 in ?? ()<br>
(gdb) bt<br>
#0 0x00000000 in ?? ()<br>
#1 0x40040437 in switch_core_codec_destroy (codec=0x54ece168) at
src/switch_core_codec.c:245<br>
#2 0x40ee778b in destroy_speech_engine (jss=0x51206538) at
mod_spidermonkey.c:1652<br>
#3 0x40eeaa70 in session_destroy (cx=0x549c9920, obj=0x4eeac7f0) at
mod_spidermonkey.c:2723<br>
#4 0x417d1aa7 in js_FinalizeObject (cx=0x549c9920, obj=0x4eeac7f0) at
src/jsobj.c:2168<br>
#5 0x417b04d9 in js_GC (cx=0x549c9920, gcflags=0) at src/jsgc.c:1856<br>
#6 0x417af6ad in js_ForceGC (cx=0x549c9920, gcflags=0) at src/jsgc.c:1508<br>
#7 0x417830fd in js_DestroyContext (cx=0x549c9920, gcmode=JS_FORCE_GC) at
src/jscntxt.c:285<br>
#8 0x417727ac in JS_DestroyContext (cx=0x549c9920) at src/jsapi.c:956<br>
#9 0x40eec3c9 in js_parse_and_execute (session=0x464d9678,
input_code=0x9e31458 "capacity.js", ro=0x0) at
mod_spidermonkey.c:3296<br>
#10 0x40eec3f2 in js_dp_function (session=0x464d9678, data=0x9e31458
"capacity.js") at mod_spidermonkey.c:3302<br>
#11 0x40044341 in switch_core_session_exec (session=0x464d9678,
application_interface=0x40f26f80, arg=0x9e31458 "capacity.js")<br>
at src/switch_core_session.c:936<br>
#12 0x400455be in switch_core_standard_on_execute (session=0x464d9678) at
src/switch_core_state_machine.c:169<br>
#13 0x40046605 in switch_core_session_run (session=0x464d9678) at
src/switch_core_state_machine.c:406<br>
#14 0x4004381c in switch_core_session_thread (thread=0x9e31288, obj=0x464d9678)
at src/switch_core_session.c:681<br>
#15 0x4009047c in dummy_worker (opaque=0x9e31288) at
threadproc/unix/thread.c:138<br>
#16 0x007cb3db in start_thread () from /lib/libpthread.so.0<br>
#17 0x0072506e in clone () from /lib/libc.so.6<br>
<br>
<br>
Tom<br>
<br>
<br>
<o:p></o:p></span></font></p>
<x-sigsep>
<p></x-sigsep><font size=3 face="Times New Roman"><span style='font-size:12.0pt'>===============<br>
tuhl@ix.netcom.com<o:p></o:p></span></font></p>
</div>
</div>
</body>
</html>