[Freeswitch-users] Capacity testing, seg fault

[Brian West]

Tom,
	Also you're using lumenvox?

/b

On Nov 29, 2007, at 2:53 PM, tuhl at ix.netcom.com wrote:

> Hi,
>
> I'm running some capacity tests on Freeswitch and can cause seg- 
> faults fairly quickly (<1 minute) at a 'light' load of 10 call  
> originations per second. Core dump backtrace is at the bottom, and  
> my debugging shows what looks like corrupted js_session. I'll open  
> an issue on JIRA. I wanted to get opinions on whether this is a  
> valid architecture for testing capacity, and whether I'm making a  
> simple mistake.
>
> Environment:
> I have the trunk version installed on 2 servers and have one server  
> (the originator) calling the other (the receiver) using SIP, g711,  
> with a Gig-E ethernet switch between them. The originating server  
> basically does a session.originate, waitForAnswer, streamFile (10- 
> second 8khz .wav), and hangup. The receiver does a session.answer  
> and a recordFile to a .wav file so I can go back and check voice  
> quality.
>
> My capacity testing engine is a Perl script which is using the RPC  
> XML interface to originate the calls on Freeswitch (I submit  
> requests to do a 'jsrun play.js' to a certain phone number, where  
> play.js is a simple script which originates, waitsforanswer,  
> streamfile, hangup) . I can configure it to make a certain number of  
> originations per second and a certain number of total calls. I have  
> no Perl script running on the receiver - I just setup the dialplan  
> to call a .js which answers the call and records it. This testing  
> setup is at an early stage, so right now, to check pass/fail, I just  
> verify that if I ran a 1000-call test on the originator, there  
> should be 1000 .wav files that are all about the same size, on the  
> receiver at the end of the test, and no crashes.
>
> I've compiled with debug flags on, and I've set all *_DEBUG flags to  
> 9 (I have also run the tests after a recompile with debug flags off/ 
> 0, and that didn't make any difference). I've done all the ulimit  
> commands that were in the last few emails on this list. I'm running  
> on FC6 on a Dell 2850 with dual 3.6ghz Xeons (/proc/cpuinfo shows 4  
> processors), and 4G RAM. I've set max-sessions to 3000 and Session  
> Rate to 100. 'top' is showing freeswitch at 60-80% on the receiver  
> during this test.
>
> ISSUE:
> My problem right now is on the receiver, which I wouldn't care about  
> because I'm most interested in the origination capacity of  
> freeswitch, but with my receiver crashing so quickly, I can't push  
> the originator very hard. I setup my originating engine to make 1000  
> total calls at 10 call originations per second, each call lasting 10  
> seconds (which results in about 120 simultaneous channels in use),  
> and I get a seg fault on the receiver within about 500 calls or 50  
> seconds. If I run a test with 1000 total calls at 6 call  
> originations per second, it will work, but if I run an overnight  
> test with 20,000 total calls at 6 call originations per second, the  
> receiver will sometimes seg-fault at around 15,000 calls, and  
> sometimes it will not. Interestingly though, if I do a very short  
> but very high rate test of 100 total calls at 50 call originations  
> per second, that will usually work. But 500 total calls at 50 calls  
> per second will always seg-fault.
>
> Just so you know... I'm shooting for the holy grail of stable  
> operation at 100 call originations per second. I know people have  
> reported much better results than I'm getting. Is something in my  
> setup bad?
>
> Here's the core dump backtrace. I added some debug printf's in  
> session_destroy right before the call to destroy_speech_engine, and  
> it looks like the jss has been trampled - for example, jss->flags is  
> always 0 for all my successful calls, but right before it seg- 
> faults, jss->flags is some large random number. This happens every  
> single time.
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x00000000 in ?? ()
> (gdb) bt
> #0  0x00000000 in ?? ()
> #1  0x40040437 in switch_core_codec_destroy (codec=0x54ece168) at  
> src/switch_core_codec.c:245
> #2  0x40ee778b in destroy_speech_engine (jss=0x51206538) at  
> mod_spidermonkey.c:1652
> #3  0x40eeaa70 in session_destroy (cx=0x549c9920, obj=0x4eeac7f0) at  
> mod_spidermonkey.c:2723
> #4  0x417d1aa7 in js_FinalizeObject (cx=0x549c9920, obj=0x4eeac7f0)  
> at src/jsobj.c:2168
> #5  0x417b04d9 in js_GC (cx=0x549c9920, gcflags=0) at src/jsgc.c:1856
> #6  0x417af6ad in js_ForceGC (cx=0x549c9920, gcflags=0) at src/ 
> jsgc.c:1508
> #7  0x417830fd in js_DestroyContext (cx=0x549c9920,  
> gcmode=JS_FORCE_GC) at src/jscntxt.c:285
> #8  0x417727ac in JS_DestroyContext (cx=0x549c9920) at src/jsapi.c:956
> #9  0x40eec3c9 in js_parse_and_execute (session=0x464d9678,  
> input_code=0x9e31458 "capacity.js", ro=0x0) at mod_spidermonkey.c:3296
> #10 0x40eec3f2 in js_dp_function (session=0x464d9678, data=0x9e31458  
> "capacity.js") at mod_spidermonkey.c:3302
> #11 0x40044341 in switch_core_session_exec (session=0x464d9678,  
> application_interface=0x40f26f80, arg=0x9e31458 "capacity.js")
>     at src/switch_core_session.c:936
> #12 0x400455be in switch_core_standard_on_execute  
> (session=0x464d9678) at src/switch_core_state_machine.c:169
> #13 0x40046605 in switch_core_session_run (session=0x464d9678) at  
> src/switch_core_state_machine.c:406
> #14 0x4004381c in switch_core_session_thread (thread=0x9e31288,  
> obj=0x464d9678) at src/switch_core_session.c:681
> #15 0x4009047c in dummy_worker (opaque=0x9e31288) at threadproc/unix/ 
> thread.c:138
> #16 0x007cb3db in start_thread () from /lib/libpthread.so.0
> #17 0x0072506e in clone () from /lib/libc.so.6
>
>
> Tom
>
> ===============
> tuhl at ix.netcom.com
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org