[Freeswitch-users] Installation report, a python crash and a bottleneck

David Knell dave at 3c.co.uk
Fri Jun 13 08:49:48 PDT 2008 

Hi Brian,

> David,
> 	It has to work?  We have people running millions of min. a day thru
> FreeSWITCH and they have zero issues.

Yes, but that misses the point.  They run millions of minutes through a
well-tested subset of that which FS offers - we ran half a million mins
a day during May through a couple of FS boxes with the odd corrupt
CDR and occasional barf[1].

But that doesn't mean that FS works in its entirety.  It means that it  
can
be made to work, which is a very different thing.

>  If you have to add the module
> to modules.conf and compile it.. that is outside the scope of what is
> well tested and supported.  Lua is more tested but still isn't in the
> default yet.  That might happen in 1.0.1.

Here's the solution, then.  Someone who knows the answers could
document the module status in modules.conf; that way, anyone who
adds a less-well-supported module to their build will at least know
about it.  As a quick aside, mod_xml_curl isn't in the default build;
it is OK for production use, isn't it..?!

> By the tone of your comments you're going to take mod_python under
> your wing and make sure it works?  We do need more people to step up
> and help.  ;)

No, I'm not, I'm afraid - (a) I've too much on my plate to have a
meaningful stab at it, and (b) I'm quite happy with curly brackets to
indicate where my blocks start and end.  But if anyone wants a hand
with writing IVRs or doing call routing using event sockets and Perl,  
I'd
be very happy to help.

--Dave

[1] Mindful of the possibility of having already offended the  
Pythonistas
by disrespecting whitespace as a syntactic tool, I probably shouldn't
add that the * boxes which front-end that traffic worked without a  
glitch,
less I find myself on the wrong end of a stake, a gallon of petrol, a  
pile
of wood and a match ;-)


>
>
> /b
>
> On Jun 13, 2008, at 9:34 AM, David Knell wrote:
>
>> Yes, but..
>>
>> Neither of Kresho's boxes are registering heavy CPU usage, which is
>> what
>> one would expect if this were to do with the overhead of Python.  In
>> addition,
>> there's the crashes.
>>
>> On to a hobby horse of mine.  If FS is going to make it in the real
>> world, then
>> it has to work.  And that means that the bits which come with it need
>> to work,
>> and those which don't need to be separated off into a 'beta', 'don't
>> expect
>> much from this', 'FFS don't use unless you're prepared to fix it' or
>> whatever
>> repository.
>>
>> New users, such as Kresho, will use whichever scripting tool/ 
>> language/
>> interface suits their needs and that they're familiar with.  And
>> they'll have the
>> expectation that it'll work, and that expectation is perfectly
>> reasonable.
>> Keeping bits in the "release" source tree which don't work properly  
>> or
>> which
>> have failings which make them unsuitable for production use does FS
>> (which is a great thing) a disservice.
>>
>> Cheers --
>>
>> Dave
>>
>>> Python is very heavy.. You should try lua.
>>>
>>> /b
>>>
>>> On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
>>>
>>>> Hi!
>>>>
>>>> I'm new to freeswitch and I like to report my success with it and a
>>>> few failures. I'll be a little bit vague on some details because I
>>>> must protect some business details. Sorry for that.
>>>>
>>>> I have no experience with asterisk, so many concepts were new to  
>>>> me.
>>>>
>>>> We run a hosted IVR system with a few hundred lines. We have a few
>>>> servers running a SIP/VoiceXML application server and connect to  
>>>> the
>>>> network with SIP/ISDN gateways.
>>>>
>>>> Recently we started an IVR with very short calls and very high CPS.
>>>> Our existing software doesn't handle this scenario very well, so I
>>>> started looking into alternatives.
>>>>
>>>> FreeSwitch caught my eye because of its support for multiple
>>>> scripting languages. I love python and this feature put FS into the
>>>> evaluation list. So I started on friday. I installed FS from the
>>>> debian repositories on my ubuntu 8.04 laptop and tried some  
>>>> examples
>>>> from the wiki ("Some thing to try out!"). I was very impressed that
>>>> everything worked right out of the box.
>>>>
>>>> I was a little disappointed that mod_python wasn't included in the
>>>> distribution so I checked out the source and compiled everything.  
>>>> An
>>>> hour later I had another installation of FS.
>>>>
>>>> It took me a few hous to get the dialplan right. Because our  
>>>> service
>>>> only runs IVRs and uses no switching, I removed everything from the
>>>> default dialplan (mainly because it conflicted with the ANI numbers
>>>> we get from the gateways).
>>>>
>>>> Another hour later, I had a simple IVR in python done: use a web
>>>> service for a database lookup and play an appropriate prompt. I
>>>> didn't use the database directly because I wanted the best possible
>>>> comparison to what our current system does, and (our) VoiceXML  
>>>> can't
>>>> use databases directly, but can use web services.
>>>>
>>>> In less than 1 working day I had everything running. Quite good.
>>>>
>>>> Time for load testing :-) Our old software handles around 20 CPS on
>>>> my laptop. I inceased max_sessions to 5000 and sessions-per-second
>>>> to 100 and started sipp. The result was quite bad - I could not get
>>>> over 8 CPS! The processor barely noticed that FS was running, so I
>>>> had no idea what the bottleneck was. I still don't. After fiddling
>>>> with this for a while, I gave up and decided to try it on one of  
>>>> our
>>>> production machines. Weekends are not very busy, so I took one
>>>> offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz,
>>>> 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04
>>>> (server edition) so I just copied the binaries.
>>>>
>>>> I ran sipp from another machine, with the uac scenario and limiting
>>>> the call duration to 4secs:
>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number>
>>>> Theoretically, as each call lasts 4 seconds, the total calls number
>>>> should never exceed 4x current CPS.
>>>>
>>>> These are the results:
>>>>
>>>> With up to 27 CPS everything was stable. The calls count was almost
>>>> exactly 4 timee the CPS, indicating that new calls were ansewered
>>>> immediately. This I also verified by calling in.
>>>>
>>>> Up to 30 CPS everything was stable for a while, but then the total
>>>> calls number exploded to the limit set by sipp. The processor load
>>>> was very reasonable, so I again I ran into the bottleneck mentioned
>>>> above. After sipp hits the total call limit, it will not create new
>>>> calls until some are released. So CPS oscillated between 0 and 30  
>>>> as
>>>> shown by sipp. CDRs show that there was an average of 27 CPS. At
>>>> this point, when I called in, I got ringback tone for as long as  
>>>> the
>>>> operator allows (60s) and then I was dropped. With a softphone I
>>>> could reach the IVR after about 80 sec.
>>>>
>>>> When I set sipp to more than 30 CPS, the number of total calls
>>>> exploded immediately.
>>>>
>>>> Experimenting some more, I found I could contain the explosion (and
>>>> the instability in CPS) by limiting the number of total calls to 4x
>>>> current cps when cps was up to 30. Thus, by starting sipp like  
>>>> this:
>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120
>>>> I could go up to 30CPS and get reasonably stable real 30CPS. When
>>>> calling in with a real phone, I would reach the IVR after 2-3
>>>> seconds of ringback, which is acceptable. This simulation run for
>>>> several hours without any other problems.
>>>>
>>>> With
>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160
>>>> and setting cps to 40, the total calls count obviously never passed
>>>> 160, but the cps shown by sipp became unstable, oscillating between
>>>> 0 and 40.
>>>>
>>>> These results are only slightly better than our current SIP server.
>>>>
>>>> Today I put FreeSwitch into production. The unexpected thing here
>>>> was that when FreeSwitch talked to our gateways instead of sipp, it
>>>> crashed a few times. I don't associate these crashes with load
>>>> because it happened equally on low and high load. Here's the log:
>>>>
>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753
>>>> switch_core_session_thread() Session 166 (sofia/internal/----
>>>> deleted stuff -----) Ended
>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755
>>>> switch_core_session_thread() Close Channel sofia/internal/----
>>>> deleted stuff ----- [CS_HANGUP]
>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218
>>>> print_trace() Obtained 10 stack frames.
>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1]
>>>> [0xb7f69420]
>>>> /usr/local/freeswitch/mod/mod_python.so [0xb011b46a]
>>>> /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a]
>>>> /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7]
>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067)  
>>>> [0xb0076907]
>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c)  
>>>> [0xb00788bc]
>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>>> /usr/lib/libpython2.5.so.1.0 [0xb001667f]
>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319
>>>> switch_core_session_run() Thread has crashed for channel sofia/
>>>> internal/---- deleted stuff -----
>>>>
>>>> It seems like mod_python is not quite ready for production  
>>>> yet :-) I
>>>> have a few core dumps available on demand, cca 2MB each.
>>>>
>>>> Turning crash protection on didn't help. FreeSwitch would reject  
>>>> new
>>>> calls, and wouldn't shutdown completely. I had to kill it.
>>>>
>>>> I would still like to replace our existing system with FreeSwitch
>>>> because I find way more comfortable to work with. It has great
>>>> potential and I'm sure it is being succesully deployed in many
>>>> places as I write this.
>>>>
>>>> The python crash is probably a simple bug. The invisible bottleneck
>>>> is what troubles me more. Any help would be greatly appreciated.
>>>>
>>>> Finally, this is all with FreeSwitch Version 1.0.pre4 (8760).
>>>> -- 
>>>> kresho
>>>> _______________________________________________
>>>> Freeswitch-users mailing list
>>>> Freeswitch-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
> Brian West
> sip:brian at freeswitch.org
>
>
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

More information about the FreeSWITCH-users mailing list