[Freeswitch-users] Installation report, a python crash and a bottleneck

Brian West brian at freeswitch.org
Fri Jun 13 09:02:07 PDT 2008 

On Jun 13, 2008, at 10:49 AM, David Knell wrote:

> Hi Brian,
>
>> David,
>> 	It has to work?  We have people running millions of min. a day thru
>> FreeSWITCH and they have zero issues.
>
> Yes, but that misses the point.  They run millions of minutes  
> through a
> well-tested subset of that which FS offers - we ran half a million  
> mins
> a day during May through a couple of FS boxes with the odd corrupt
> CDR and occasional barf[1].
>

Did you report a bug?  Did you rule out bad ram?

> But that doesn't mean that FS works in its entirety.  It means that it
> can
> be made to work, which is a very different thing.
>
>> If you have to add the module
>> to modules.conf and compile it.. that is outside the scope of what is
>> well tested and supported.  Lua is more tested but still isn't in the
>> default yet.  That might happen in 1.0.1.
>
> Here's the solution, then.  Someone who knows the answers could
> document the module status in modules.conf; that way, anyone who
> adds a less-well-supported module to their build will at least know
> about it.  As a quick aside, mod_xml_curl isn't in the default build;
> it is OK for production use, isn't it..?!

Should be.

>
>
>> By the tone of your comments you're going to take mod_python under
>> your wing and make sure it works?  We do need more people to step up
>> and help.  ;)
>
> No, I'm not, I'm afraid - (a) I've too much on my plate to have a
> meaningful stab at it, and (b) I'm quite happy with curly brackets to
> indicate where my blocks start and end.  But if anyone wants a hand
> with writing IVRs or doing call routing using event sockets and Perl,
> I'd
> be very happy to help.

Welcome to my world.

>
>
> --Dave
>
> [1] Mindful of the possibility of having already offended the
> Pythonistas
> by disrespecting whitespace as a syntactic tool, I probably shouldn't
> add that the * boxes which front-end that traffic worked without a
> glitch,
> less I find myself on the wrong end of a stake, a gallon of petrol, a
> pile
> of wood and a match ;-)
>
>
>>
>>
>> /b
>>
>> On Jun 13, 2008, at 9:34 AM, David Knell wrote:
>>
>>> Yes, but..
>>>
>>> Neither of Kresho's boxes are registering heavy CPU usage, which is
>>> what
>>> one would expect if this were to do with the overhead of Python.  In
>>> addition,
>>> there's the crashes.
>>>
>>> On to a hobby horse of mine.  If FS is going to make it in the real
>>> world, then
>>> it has to work.  And that means that the bits which come with it  
>>> need
>>> to work,
>>> and those which don't need to be separated off into a 'beta', 'don't
>>> expect
>>> much from this', 'FFS don't use unless you're prepared to fix it' or
>>> whatever
>>> repository.
>>>
>>> New users, such as Kresho, will use whichever scripting tool/
>>> language/
>>> interface suits their needs and that they're familiar with.  And
>>> they'll have the
>>> expectation that it'll work, and that expectation is perfectly
>>> reasonable.
>>> Keeping bits in the "release" source tree which don't work properly
>>> or
>>> which
>>> have failings which make them unsuitable for production use does FS
>>> (which is a great thing) a disservice.
>>>
>>> Cheers --
>>>
>>> Dave
>>>
>>>> Python is very heavy.. You should try lua.
>>>>
>>>> /b
>>>>
>>>> On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> I'm new to freeswitch and I like to report my success with it  
>>>>> and a
>>>>> few failures. I'll be a little bit vague on some details because I
>>>>> must protect some business details. Sorry for that.
>>>>>
>>>>> I have no experience with asterisk, so many concepts were new to
>>>>> me.
>>>>>
>>>>> We run a hosted IVR system with a few hundred lines. We have a few
>>>>> servers running a SIP/VoiceXML application server and connect to
>>>>> the
>>>>> network with SIP/ISDN gateways.
>>>>>
>>>>> Recently we started an IVR with very short calls and very high  
>>>>> CPS.
>>>>> Our existing software doesn't handle this scenario very well, so I
>>>>> started looking into alternatives.
>>>>>
>>>>> FreeSwitch caught my eye because of its support for multiple
>>>>> scripting languages. I love python and this feature put FS into  
>>>>> the
>>>>> evaluation list. So I started on friday. I installed FS from the
>>>>> debian repositories on my ubuntu 8.04 laptop and tried some
>>>>> examples
>>>>> from the wiki ("Some thing to try out!"). I was very impressed  
>>>>> that
>>>>> everything worked right out of the box.
>>>>>
>>>>> I was a little disappointed that mod_python wasn't included in the
>>>>> distribution so I checked out the source and compiled everything.
>>>>> An
>>>>> hour later I had another installation of FS.
>>>>>
>>>>> It took me a few hous to get the dialplan right. Because our
>>>>> service
>>>>> only runs IVRs and uses no switching, I removed everything from  
>>>>> the
>>>>> default dialplan (mainly because it conflicted with the ANI  
>>>>> numbers
>>>>> we get from the gateways).
>>>>>
>>>>> Another hour later, I had a simple IVR in python done: use a web
>>>>> service for a database lookup and play an appropriate prompt. I
>>>>> didn't use the database directly because I wanted the best  
>>>>> possible
>>>>> comparison to what our current system does, and (our) VoiceXML
>>>>> can't
>>>>> use databases directly, but can use web services.
>>>>>
>>>>> In less than 1 working day I had everything running. Quite good.
>>>>>
>>>>> Time for load testing :-) Our old software handles around 20 CPS  
>>>>> on
>>>>> my laptop. I inceased max_sessions to 5000 and sessions-per-second
>>>>> to 100 and started sipp. The result was quite bad - I could not  
>>>>> get
>>>>> over 8 CPS! The processor barely noticed that FS was running, so I
>>>>> had no idea what the bottleneck was. I still don't. After fiddling
>>>>> with this for a while, I gave up and decided to try it on one of
>>>>> our
>>>>> production machines. Weekends are not very busy, so I took one
>>>>> offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz,
>>>>> 2G ecc ram and 10krpm disks. The server is also running ubuntu  
>>>>> 8.04
>>>>> (server edition) so I just copied the binaries.
>>>>>
>>>>> I ran sipp from another machine, with the uac scenario and  
>>>>> limiting
>>>>> the call duration to 4secs:
>>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number>
>>>>> Theoretically, as each call lasts 4 seconds, the total calls  
>>>>> number
>>>>> should never exceed 4x current CPS.
>>>>>
>>>>> These are the results:
>>>>>
>>>>> With up to 27 CPS everything was stable. The calls count was  
>>>>> almost
>>>>> exactly 4 timee the CPS, indicating that new calls were ansewered
>>>>> immediately. This I also verified by calling in.
>>>>>
>>>>> Up to 30 CPS everything was stable for a while, but then the total
>>>>> calls number exploded to the limit set by sipp. The processor load
>>>>> was very reasonable, so I again I ran into the bottleneck  
>>>>> mentioned
>>>>> above. After sipp hits the total call limit, it will not create  
>>>>> new
>>>>> calls until some are released. So CPS oscillated between 0 and 30
>>>>> as
>>>>> shown by sipp. CDRs show that there was an average of 27 CPS. At
>>>>> this point, when I called in, I got ringback tone for as long as
>>>>> the
>>>>> operator allows (60s) and then I was dropped. With a softphone I
>>>>> could reach the IVR after about 80 sec.
>>>>>
>>>>> When I set sipp to more than 30 CPS, the number of total calls
>>>>> exploded immediately.
>>>>>
>>>>> Experimenting some more, I found I could contain the explosion  
>>>>> (and
>>>>> the instability in CPS) by limiting the number of total calls to  
>>>>> 4x
>>>>> current cps when cps was up to 30. Thus, by starting sipp like
>>>>> this:
>>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120
>>>>> I could go up to 30CPS and get reasonably stable real 30CPS. When
>>>>> calling in with a real phone, I would reach the IVR after 2-3
>>>>> seconds of ringback, which is acceptable. This simulation run for
>>>>> several hours without any other problems.
>>>>>
>>>>> With
>>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160
>>>>> and setting cps to 40, the total calls count obviously never  
>>>>> passed
>>>>> 160, but the cps shown by sipp became unstable, oscillating  
>>>>> between
>>>>> 0 and 40.
>>>>>
>>>>> These results are only slightly better than our current SIP  
>>>>> server.
>>>>>
>>>>> Today I put FreeSwitch into production. The unexpected thing here
>>>>> was that when FreeSwitch talked to our gateways instead of sipp,  
>>>>> it
>>>>> crashed a few times. I don't associate these crashes with load
>>>>> because it happened equally on low and high load. Here's the log:
>>>>>
>>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753
>>>>> switch_core_session_thread() Session 166 (sofia/internal/----
>>>>> deleted stuff -----) Ended
>>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755
>>>>> switch_core_session_thread() Close Channel sofia/internal/----
>>>>> deleted stuff ----- [CS_HANGUP]
>>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218
>>>>> print_trace() Obtained 10 stack frames.
>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1]
>>>>> [0xb7f69420]
>>>>> /usr/local/freeswitch/mod/mod_python.so [0xb011b46a]
>>>>> /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a]
>>>>> /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7]
>>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067)
>>>>> [0xb0076907]
>>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c)
>>>>> [0xb00788bc]
>>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
>>>>> /usr/lib/libpython2.5.so.1.0 [0xb001667f]
>>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319
>>>>> switch_core_session_run() Thread has crashed for channel sofia/
>>>>> internal/---- deleted stuff -----
>>>>>
>>>>> It seems like mod_python is not quite ready for production
>>>>> yet :-) I
>>>>> have a few core dumps available on demand, cca 2MB each.
>>>>>
>>>>> Turning crash protection on didn't help. FreeSwitch would reject
>>>>> new
>>>>> calls, and wouldn't shutdown completely. I had to kill it.
>>>>>
>>>>> I would still like to replace our existing system with FreeSwitch
>>>>> because I find way more comfortable to work with. It has great
>>>>> potential and I'm sure it is being succesully deployed in many
>>>>> places as I write this.
>>>>>
>>>>> The python crash is probably a simple bug. The invisible  
>>>>> bottleneck
>>>>> is what troubles me more. Any help would be greatly appreciated.
>>>>>
>>>>> Finally, this is all with FreeSwitch Version 1.0.pre4 (8760).
>>>>> -- 
>>>>> kresho
>>>>> _______________________________________________
>>>>> Freeswitch-users mailing list
>>>>> Freeswitch-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>
>>>>
>>>> _______________________________________________
>>>> Freeswitch-users mailing list
>>>> Freeswitch-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>> Brian West
>> sip:brian at freeswitch.org
>>
>>
>>
>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

Brian West
sip:brian at freeswitch.org

More information about the FreeSWITCH-users mailing list