[Freeswitch-users] Installation report, a python crash and a bottleneck

Fri Jun 13 07:55:18 PDT 2008

David,
	It has to work?  We have people running millions of min. a day thru  
FreeSWITCH and they have zero issues.  If you have to add the module  
to modules.conf and compile it.. that is outside the scope of what is  
well tested and supported.  Lua is more tested but still isn't in the  
default yet.  That might happen in 1.0.1.

By the tone of your comments you're going to take mod_python under  
your wing and make sure it works?  We do need more people to step up  
and help.  ;)

/b

On Jun 13, 2008, at 9:34 AM, David Knell wrote:

> Yes, but..
>
> Neither of Kresho's boxes are registering heavy CPU usage, which is  
> what
> one would expect if this were to do with the overhead of Python.  In
> addition,
> there's the crashes.
>
> On to a hobby horse of mine.  If FS is going to make it in the real
> world, then
> it has to work.  And that means that the bits which come with it need
> to work,
> and those which don't need to be separated off into a 'beta', 'don't
> expect
> much from this', 'FFS don't use unless you're prepared to fix it' or
> whatever
> repository.
>
> New users, such as Kresho, will use whichever scripting tool/language/
> interface suits their needs and that they're familiar with.  And
> they'll have the
> expectation that it'll work, and that expectation is perfectly
> reasonable.
> Keeping bits in the "release" source tree which don't work properly or
> which
> have failings which make them unsuitable for production use does FS
> (which is a great thing) a disservice.
>
> Cheers --
>
> Dave
>
>> Python is very heavy.. You should try lua.
>>
>> /b
>>
>> On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
>>
>>> Hi!
>>>
>>> I'm new to freeswitch and I like to report my success with it and a
>>> few failures. I'll be a little bit vague on some details because I
>>> must protect some business details. Sorry for that.
>>>
>>> I have no experience with asterisk, so many concepts were new to me.
>>>
>>> We run a hosted IVR system with a few hundred lines. We have a few
>>> servers running a SIP/VoiceXML application server and connect to the
>>> network with SIP/ISDN gateways.
>>>
>>> Recently we started an IVR with very short calls and very high CPS.
>>> Our existing software doesn't handle this scenario very well, so I
>>> started looking into alternatives.
>>>
>>> FreeSwitch caught my eye because of its support for multiple
>>> scripting languages. I love python and this feature put FS into the
>>> evaluation list. So I started on friday. I installed FS from the
>>> debian repositories on my ubuntu 8.04 laptop and tried some examples
>>> from the wiki ("Some thing to try out!"). I was very impressed that
>>> everything worked right out of the box.
>>>
>>> I was a little disappointed that mod_python wasn't included in the
>>> distribution so I checked out the source and compiled everything. An
>>> hour later I had another installation of FS.
>>>
>>> It took me a few hous to get the dialplan right. Because our service
>>> only runs IVRs and uses no switching, I removed everything from the
>>> default dialplan (mainly because it conflicted with the ANI numbers
>>> we get from the gateways).
>>>
>>> Another hour later, I had a simple IVR in python done: use a web
>>> service for a database lookup and play an appropriate prompt. I
>>> didn't use the database directly because I wanted the best possible
>>> comparison to what our current system does, and (our) VoiceXML can't
>>> use databases directly, but can use web services.
>>>
>>> In less than 1 working day I had everything running. Quite good.
>>>
>>> Time for load testing :-) Our old software handles around 20 CPS on
>>> my laptop. I inceased max_sessions to 5000 and sessions-per-second
>>> to 100 and started sipp. The result was quite bad - I could not get
>>> over 8 CPS! The processor barely noticed that FS was running, so I
>>> had no idea what the bottleneck was. I still don't. After fiddling
>>> with this for a while, I gave up and decided to try it on one of our
>>> production machines. Weekends are not very busy, so I took one
>>> offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz,
>>> 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04
>>> (server edition) so I just copied the binaries.
>>>
>>> I ran sipp from another machine, with the uac scenario and limiting
>>> the call duration to 4secs:
>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number>
>>> Theoretically, as each call lasts 4 seconds, the total calls number
>>> should never exceed 4x current CPS.
>>>
>>> These are the results:
>>>
>>> With up to 27 CPS everything was stable. The calls count was almost
>>> exactly 4 timee the CPS, indicating that new calls were ansewered
>>> immediately. This I also verified by calling in.
>>>
>>> Up to 30 CPS everything was stable for a while, but then the total
>>> calls number exploded to the limit set by sipp. The processor load
>>> was very reasonable, so I again I ran into the bottleneck mentioned
>>> above. After sipp hits the total call limit, it will not create new
>>> calls until some are released. So CPS oscillated between 0 and 30 as
>>> shown by sipp. CDRs show that there was an average of 27 CPS. At
>>> this point, when I called in, I got ringback tone for as long as the
>>> operator allows (60s) and then I was dropped. With a softphone I
>>> could reach the IVR after about 80 sec.
>>>
>>> When I set sipp to more than 30 CPS, the number of total calls
>>> exploded immediately.
>>>
>>> Experimenting some more, I found I could contain the explosion (and
>>> the instability in CPS) by limiting the number of total calls to 4x
>>> current cps when cps was up to 30. Thus, by starting sipp like this:
>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120
>>> I could go up to 30CPS and get reasonably stable real 30CPS. When
>>> calling in with a real phone, I would reach the IVR after 2-3
>>> seconds of ringback, which is acceptable. This simulation run for
>>> several hours without any other problems.
>>>
>>> With
>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160
>>> and setting cps to 40, the total calls count obviously never passed
>>> 160, but the cps shown by sipp became unstable, oscillating between
>>> 0 and 40.
>>>
>>> These results are only slightly better than our current SIP server.
>>>
>>> Today I put FreeSwitch into production. The unexpected thing here
>>> was that when FreeSwitch talked to our gateways instead of sipp, it
>>> crashed a few times. I don't associate these crashes with load
>>> because it happened equally on low and high load. Here's the log:
>>>
>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753
>>> switch_core_session_thread() Session 166 (sofia/internal/----
>>> deleted stuff -----) Ended
>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755
>>> switch_core_session_thread() Close Channel sofia/internal/----
>>> deleted stuff ----- [CS_HANGUP]
>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218
>>> print_trace() Obtained 10 stack frames.
>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1]
>>> [0xb7f69420]
>>> /usr/local/freeswitch/mod/mod_python.so [0xb011b46a]
>>> /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a]
>>> /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7]
>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907]
>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc]
>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>> /usr/lib/libpython2.5.so.1.0 [0xb001667f]
>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319
>>> switch_core_session_run() Thread has crashed for channel sofia/
>>> internal/---- deleted stuff -----
>>>
>>> It seems like mod_python is not quite ready for production yet :-) I
>>> have a few core dumps available on demand, cca 2MB each.
>>>
>>> Turning crash protection on didn't help. FreeSwitch would reject new
>>> calls, and wouldn't shutdown completely. I had to kill it.
>>>
>>> I would still like to replace our existing system with FreeSwitch
>>> because I find way more comfortable to work with. It has great
>>> potential and I'm sure it is being succesully deployed in many
>>> places as I write this.
>>>
>>> The python crash is probably a simple bug. The invisible bottleneck
>>> is what troubles me more. Any help would be greatly appreciated.
>>>
>>> Finally, this is all with FreeSwitch Version 1.0.pre4 (8760).
>>> -- 
>>> kresho
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

Brian West
sip:brian at freeswitch.org