<html>
<body>
Hi, <br><br>
I'm running some capacity tests on Freeswitch and can cause seg-faults
fairly quickly (<1 minute) at a 'light' load of 10 call originations
per second. Core dump backtrace is at the bottom, and my debugging shows
what looks like corrupted js_session. I'll open an issue on JIRA. I
wanted to get opinions on whether this is a valid architecture for
testing capacity, and whether I'm making a simple mistake.<br><br>
<b>Environment:<br>
</b>I have the trunk version installed on 2 servers and have one server
(the originator) calling the other (the receiver) using SIP, g711, with a
Gig-E ethernet switch between them. The originating server basically does
a session.originate, waitForAnswer, streamFile (10-second 8khz .wav), and
hangup. The receiver does a session.answer and a recordFile to a .wav
file so I can go back and check voice quality.<br><br>
My capacity testing engine is a Perl script which is using the RPC XML
interface to originate the calls on Freeswitch (I submit requests to do a
'jsrun play.js' to a certain phone number, where play.js is a simple
script which originates, waitsforanswer, streamfile, hangup) . I can
configure it to make a certain number of originations per second and a
certain number of total calls. I have no Perl script running on the
receiver - I just setup the dialplan to call a .js which answers the call
and records it. This testing setup is at an early stage, so right now, to
check pass/fail, I just verify that if I ran a 1000-call test on the
originator, there should be 1000 .wav files that are all about the same
size, on the receiver at the end of the test, and no crashes.<br><br>
I've compiled with debug flags on, and I've set all *_DEBUG flags to 9 (I
have also run the tests after a recompile with debug flags off/0, and
that didn't make any difference). I've done all the ulimit commands that
were in the last few emails on this list. I'm running on FC6 on a Dell
2850 with dual 3.6ghz Xeons (/proc/cpuinfo shows 4 processors), and 4G
RAM. I've set max-sessions to 3000 and Session Rate to 100. 'top' is
showing freeswitch at 60-80% on the receiver during this test.<br><br>
<b>ISSUE: <br>
</b>My problem right now is on the receiver, which I wouldn't care about
because I'm most interested in the origination capacity of freeswitch,
but with my receiver crashing so quickly, I can't push the originator
very hard. I setup my originating engine to make 1000 total calls at 10
call originations per second, each call lasting 10 seconds (which results
in about 120 simultaneous channels in use), and I get a seg fault on the
receiver within about 500 calls or 50 seconds. If I run a test with 1000
total calls at 6 call originations per second, it will work, but if I run
an overnight test with 20,000 total calls at 6 call originations per
second, the receiver will sometimes seg-fault at around 15,000 calls, and
sometimes it will not. Interestingly though, if I do a very short but
very high rate test of 100 total calls at 50 call originations per
second, that will usually work. But 500 total calls at 50 calls per
second will always seg-fault.<br><br>
Just so you know... I'm shooting for the holy grail of stable operation
at 100 call originations per second. I know people have reported much
better results than I'm getting. Is something in my setup bad?<br><br>
Here's the core dump backtrace. I added some debug printf's in
session_destroy right before the call to destroy_speech_engine, and it
looks like the jss has been trampled - for example, jss->flags is
always 0 for all my successful calls, but right before it seg-faults,
jss->flags is some large random number. This happens every single
time.<br><br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00000000 in ?? ()<br>
(gdb) bt<br>
#0 0x00000000 in ?? ()<br>
#1 0x40040437 in switch_core_codec_destroy (codec=0x54ece168) at
src/switch_core_codec.c:245<br>
#2 0x40ee778b in destroy_speech_engine (jss=0x51206538) at
mod_spidermonkey.c:1652<br>
#3 0x40eeaa70 in session_destroy (cx=0x549c9920, obj=0x4eeac7f0) at
mod_spidermonkey.c:2723<br>
#4 0x417d1aa7 in js_FinalizeObject (cx=0x549c9920, obj=0x4eeac7f0)
at src/jsobj.c:2168<br>
#5 0x417b04d9 in js_GC (cx=0x549c9920, gcflags=0) at
src/jsgc.c:1856<br>
#6 0x417af6ad in js_ForceGC (cx=0x549c9920, gcflags=0) at
src/jsgc.c:1508<br>
#7 0x417830fd in js_DestroyContext (cx=0x549c9920,
gcmode=JS_FORCE_GC) at src/jscntxt.c:285<br>
#8 0x417727ac in JS_DestroyContext (cx=0x549c9920) at
src/jsapi.c:956<br>
#9 0x40eec3c9 in js_parse_and_execute (session=0x464d9678,
input_code=0x9e31458 "capacity.js", ro=0x0) at
mod_spidermonkey.c:3296<br>
#10 0x40eec3f2 in js_dp_function (session=0x464d9678, data=0x9e31458
"capacity.js") at mod_spidermonkey.c:3302<br>
#11 0x40044341 in switch_core_session_exec (session=0x464d9678,
application_interface=0x40f26f80, arg=0x9e31458
"capacity.js")<br>
at src/switch_core_session.c:936<br>
#12 0x400455be in switch_core_standard_on_execute (session=0x464d9678) at
src/switch_core_state_machine.c:169<br>
#13 0x40046605 in switch_core_session_run (session=0x464d9678) at
src/switch_core_state_machine.c:406<br>
#14 0x4004381c in switch_core_session_thread (thread=0x9e31288,
obj=0x464d9678) at src/switch_core_session.c:681<br>
#15 0x4009047c in dummy_worker (opaque=0x9e31288) at
threadproc/unix/thread.c:138<br>
#16 0x007cb3db in start_thread () from /lib/libpthread.so.0<br>
#17 0x0072506e in clone () from /lib/libc.so.6<br><br>
<br>
Tom<br><br>
<x-sigsep><p></x-sigsep>
===============<br>
tuhl@ix.netcom.com</body>
</html>