[Freeswitch-users] Installation report, a python crash and a bottleneck

Krešimir Tonković kresho at multimodus.hr
Fri Jun 13 11:17:51 PDT 2008 

Funny thing... I offered to pay for a fix for this bug by sending a mail to
consulting at freeswitch.org, but nobody responded. Now I spent some of that
money reimplementing the ivr in javascript. The rest is still there if
somebody wants it :-)

Regarding pythonistas and stake burnings - I just want the humongous number
of libraries that come with python.

I agree with David - pieces that aren't quite production ready should be
marked as such.

Regards,
Kresho

On Fri, Jun 13, 2008 at 7:05 PM, Brian West <brian at freeswitch.org> wrote:

> Also let me just say one last thing here.. As long as their is a bug
> on Jira about this issue we will get around to it... it's just not a
> top priority right now.
>
> /b
>
> On Jun 13, 2008, at 10:49 AM, David Knell wrote:
>
> > Hi Brian,
> >
> >> David,
> >>      It has to work?  We have people running millions of min. a day thru
> >> FreeSWITCH and they have zero issues.
> >
> > Yes, but that misses the point.  They run millions of minutes
> > through a
> > well-tested subset of that which FS offers - we ran half a million
> > mins
> > a day during May through a couple of FS boxes with the odd corrupt
> > CDR and occasional barf[1].
> >
> > But that doesn't mean that FS works in its entirety.  It means that it
> > can
> > be made to work, which is a very different thing.
> >
> >> If you have to add the module
> >> to modules.conf and compile it.. that is outside the scope of what is
> >> well tested and supported.  Lua is more tested but still isn't in the
> >> default yet.  That might happen in 1.0.1.
> >
> > Here's the solution, then.  Someone who knows the answers could
> > document the module status in modules.conf; that way, anyone who
> > adds a less-well-supported module to their build will at least know
> > about it.  As a quick aside, mod_xml_curl isn't in the default build;
> > it is OK for production use, isn't it..?!
> >
> >> By the tone of your comments you're going to take mod_python under
> >> your wing and make sure it works?  We do need more people to step up
> >> and help.  ;)
> >
> > No, I'm not, I'm afraid - (a) I've too much on my plate to have a
> > meaningful stab at it, and (b) I'm quite happy with curly brackets to
> > indicate where my blocks start and end.  But if anyone wants a hand
> > with writing IVRs or doing call routing using event sockets and Perl,
> > I'd
> > be very happy to help.
> >
> > --Dave
> >
> > [1] Mindful of the possibility of having already offended the
> > Pythonistas
> > by disrespecting whitespace as a syntactic tool, I probably shouldn't
> > add that the * boxes which front-end that traffic worked without a
> > glitch,
> > less I find myself on the wrong end of a stake, a gallon of petrol, a
> > pile
> > of wood and a match ;-)
> >
> >
> >>
> >>
> >> /b
> >>
> >> On Jun 13, 2008, at 9:34 AM, David Knell wrote:
> >>
> >>> Yes, but..
> >>>
> >>> Neither of Kresho's boxes are registering heavy CPU usage, which is
> >>> what
> >>> one would expect if this were to do with the overhead of Python.  In
> >>> addition,
> >>> there's the crashes.
> >>>
> >>> On to a hobby horse of mine.  If FS is going to make it in the real
> >>> world, then
> >>> it has to work.  And that means that the bits which come with it
> >>> need
> >>> to work,
> >>> and those which don't need to be separated off into a 'beta', 'don't
> >>> expect
> >>> much from this', 'FFS don't use unless you're prepared to fix it' or
> >>> whatever
> >>> repository.
> >>>
> >>> New users, such as Kresho, will use whichever scripting tool/
> >>> language/
> >>> interface suits their needs and that they're familiar with.  And
> >>> they'll have the
> >>> expectation that it'll work, and that expectation is perfectly
> >>> reasonable.
> >>> Keeping bits in the "release" source tree which don't work properly
> >>> or
> >>> which
> >>> have failings which make them unsuitable for production use does FS
> >>> (which is a great thing) a disservice.
> >>>
> >>> Cheers --
> >>>
> >>> Dave
> >>>
> >>>> Python is very heavy.. You should try lua.
> >>>>
> >>>> /b
> >>>>
> >>>> On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> I'm new to freeswitch and I like to report my success with it
> >>>>> and a
> >>>>> few failures. I'll be a little bit vague on some details because I
> >>>>> must protect some business details. Sorry for that.
> >>>>>
> >>>>> I have no experience with asterisk, so many concepts were new to
> >>>>> me.
> >>>>>
> >>>>> We run a hosted IVR system with a few hundred lines. We have a few
> >>>>> servers running a SIP/VoiceXML application server and connect to
> >>>>> the
> >>>>> network with SIP/ISDN gateways.
> >>>>>
> >>>>> Recently we started an IVR with very short calls and very high
> >>>>> CPS.
> >>>>> Our existing software doesn't handle this scenario very well, so I
> >>>>> started looking into alternatives.
> >>>>>
> >>>>> FreeSwitch caught my eye because of its support for multiple
> >>>>> scripting languages. I love python and this feature put FS into
> >>>>> the
> >>>>> evaluation list. So I started on friday. I installed FS from the
> >>>>> debian repositories on my ubuntu 8.04 laptop and tried some
> >>>>> examples
> >>>>> from the wiki ("Some thing to try out!"). I was very impressed
> >>>>> that
> >>>>> everything worked right out of the box.
> >>>>>
> >>>>> I was a little disappointed that mod_python wasn't included in the
> >>>>> distribution so I checked out the source and compiled everything.
> >>>>> An
> >>>>> hour later I had another installation of FS.
> >>>>>
> >>>>> It took me a few hous to get the dialplan right. Because our
> >>>>> service
> >>>>> only runs IVRs and uses no switching, I removed everything from
> >>>>> the
> >>>>> default dialplan (mainly because it conflicted with the ANI
> >>>>> numbers
> >>>>> we get from the gateways).
> >>>>>
> >>>>> Another hour later, I had a simple IVR in python done: use a web
> >>>>> service for a database lookup and play an appropriate prompt. I
> >>>>> didn't use the database directly because I wanted the best
> >>>>> possible
> >>>>> comparison to what our current system does, and (our) VoiceXML
> >>>>> can't
> >>>>> use databases directly, but can use web services.
> >>>>>
> >>>>> In less than 1 working day I had everything running. Quite good.
> >>>>>
> >>>>> Time for load testing :-) Our old software handles around 20 CPS
> >>>>> on
> >>>>> my laptop. I inceased max_sessions to 5000 and sessions-per-second
> >>>>> to 100 and started sipp. The result was quite bad - I could not
> >>>>> get
> >>>>> over 8 CPS! The processor barely noticed that FS was running, so I
> >>>>> had no idea what the bottleneck was. I still don't. After fiddling
> >>>>> with this for a while, I gave up and decided to try it on one of
> >>>>> our
> >>>>> production machines. Weekends are not very busy, so I took one
> >>>>> offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz,
> >>>>> 2G ecc ram and 10krpm disks. The server is also running ubuntu
> >>>>> 8.04
> >>>>> (server edition) so I just copied the binaries.
> >>>>>
> >>>>> I ran sipp from another machine, with the uac scenario and
> >>>>> limiting
> >>>>> the call duration to 4secs:
> >>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number>
> >>>>> Theoretically, as each call lasts 4 seconds, the total calls
> >>>>> number
> >>>>> should never exceed 4x current CPS.
> >>>>>
> >>>>> These are the results:
> >>>>>
> >>>>> With up to 27 CPS everything was stable. The calls count was
> >>>>> almost
> >>>>> exactly 4 timee the CPS, indicating that new calls were ansewered
> >>>>> immediately. This I also verified by calling in.
> >>>>>
> >>>>> Up to 30 CPS everything was stable for a while, but then the total
> >>>>> calls number exploded to the limit set by sipp. The processor load
> >>>>> was very reasonable, so I again I ran into the bottleneck
> >>>>> mentioned
> >>>>> above. After sipp hits the total call limit, it will not create
> >>>>> new
> >>>>> calls until some are released. So CPS oscillated between 0 and 30
> >>>>> as
> >>>>> shown by sipp. CDRs show that there was an average of 27 CPS. At
> >>>>> this point, when I called in, I got ringback tone for as long as
> >>>>> the
> >>>>> operator allows (60s) and then I was dropped. With a softphone I
> >>>>> could reach the IVR after about 80 sec.
> >>>>>
> >>>>> When I set sipp to more than 30 CPS, the number of total calls
> >>>>> exploded immediately.
> >>>>>
> >>>>> Experimenting some more, I found I could contain the explosion
> >>>>> (and
> >>>>> the instability in CPS) by limiting the number of total calls to
> >>>>> 4x
> >>>>> current cps when cps was up to 30. Thus, by starting sipp like
> >>>>> this:
> >>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120
> >>>>> I could go up to 30CPS and get reasonably stable real 30CPS. When
> >>>>> calling in with a real phone, I would reach the IVR after 2-3
> >>>>> seconds of ringback, which is acceptable. This simulation run for
> >>>>> several hours without any other problems.
> >>>>>
> >>>>> With
> >>>>> sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160
> >>>>> and setting cps to 40, the total calls count obviously never
> >>>>> passed
> >>>>> 160, but the cps shown by sipp became unstable, oscillating
> >>>>> between
> >>>>> 0 and 40.
> >>>>>
> >>>>> These results are only slightly better than our current SIP
> >>>>> server.
> >>>>>
> >>>>> Today I put FreeSwitch into production. The unexpected thing here
> >>>>> was that when FreeSwitch talked to our gateways instead of sipp,
> >>>>> it
> >>>>> crashed a few times. I don't associate these crashes with load
> >>>>> because it happened equally on low and high load. Here's the log:
> >>>>>
> >>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753
> >>>>> switch_core_session_thread() Session 166 (sofia/internal/----
> >>>>> deleted stuff -----) Ended
> >>>>> 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755
> >>>>> switch_core_session_thread() Close Channel sofia/internal/----
> >>>>> deleted stuff ----- [CS_HANGUP]
> >>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218
> >>>>> print_trace() Obtained 10 stack frames.
> >>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1]
> >>>>> [0xb7f69420]
> >>>>> /usr/local/freeswitch/mod/mod_python.so [0xb011b46a]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067)
> >>>>> [0xb0076907]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c)
> >>>>> [0xb00788bc]
> >>>>> /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368]
> >>>>> /usr/lib/libpython2.5.so.1.0 [0xb001667f]
> >>>>> 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319
> >>>>> switch_core_session_run() Thread has crashed for channel sofia/
> >>>>> internal/---- deleted stuff -----
> >>>>>
> >>>>> It seems like mod_python is not quite ready for production
> >>>>> yet :-) I
> >>>>> have a few core dumps available on demand, cca 2MB each.
> >>>>>
> >>>>> Turning crash protection on didn't help. FreeSwitch would reject
> >>>>> new
> >>>>> calls, and wouldn't shutdown completely. I had to kill it.
> >>>>>
> >>>>> I would still like to replace our existing system with FreeSwitch
> >>>>> because I find way more comfortable to work with. It has great
> >>>>> potential and I'm sure it is being succesully deployed in many
> >>>>> places as I write this.
> >>>>>
> >>>>> The python crash is probably a simple bug. The invisible
> >>>>> bottleneck
> >>>>> is what troubles me more. Any help would be greatly appreciated.
> >>>>>
> >>>>> Finally, this is all with FreeSwitch Version 1.0.pre4 (8760).
> >>>>> --
> >>>>> kresho
> >>>>> _______________________________________________
> >>>>> Freeswitch-users mailing list
> >>>>> Freeswitch-users at lists.freeswitch.org
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Freeswitch-users mailing list
> >>>> Freeswitch-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> >>>
> >>> _______________________________________________
> >>> Freeswitch-users mailing list
> >>> Freeswitch-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >> Brian West
> >> sip:brian at freeswitch.org <sip%3Abrian at freeswitch.org>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Freeswitch-users mailing list
> >> Freeswitch-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
> Brian West
> sip:brian at freeswitch.org <sip%3Abrian at freeswitch.org>
>
>
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 
kresho
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20080613/fd6fd283/attachment-0002.html

More information about the FreeSWITCH-users mailing list