Installation report, a python crash and a bottleneck
Hi! I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that. I have no experience with asterisk, so many concepts were new to me. We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways. Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives. FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box. I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS. It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways). Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services. In less than 1 working day I had everything running. Quite good. Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries. I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS. These are the results: With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in. Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec. When I set sipp to more than 30 CPS, the number of total calls exploded immediately. Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems. With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40. These results are only slightly better than our current SIP server. Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log: 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/internal/---- deleted stuff ----- It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each. Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it. I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this. The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated. Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho
Python is very heavy.. You should try lua. /b On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, but.. Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes. On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository. New users, such as Kresho, will use whichever scripting tool/language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice. Cheers -- Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues. If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1. By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;) /b On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1]. But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help. --Dave [1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Jun 13, 2008, at 10:49 AM, David Knell wrote:
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1].
Did you report a bug? Did you rule out bad ram?
But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
Should be.
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help.
Welcome to my world.
--Dave
[1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
Also let me just say one last thing here.. As long as their is a bug on Jira about this issue we will get around to it... it's just not a top priority right now. /b On Jun 13, 2008, at 10:49 AM, David Knell wrote:
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1].
But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help.
--Dave
[1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
Funny thing... I offered to pay for a fix for this bug by sending a mail to consulting@freeswitch.org, but nobody responded. Now I spent some of that money reimplementing the ivr in javascript. The rest is still there if somebody wants it :-) Regarding pythonistas and stake burnings - I just want the humongous number of libraries that come with python. I agree with David - pieces that aren't quite production ready should be marked as such. Regards, Kresho On Fri, Jun 13, 2008 at 7:05 PM, Brian West <brian@freeswitch.org> wrote:
Also let me just say one last thing here.. As long as their is a bug on Jira about this issue we will get around to it... it's just not a top priority right now.
/b
On Jun 13, 2008, at 10:49 AM, David Knell wrote:
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1].
But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help.
--Dave
[1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per-second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org <sip%3Abrian@freeswitch.org>
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org <sip%3Abrian@freeswitch.org>
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- kresho
Kresho, Thats because Anthony is on vacation and he didn't write mod_python. I should have responded to you and let you know that. He's the one that has to make the decision. Its something we have discussed in the past few weeks. As for production ready modules if you have to uncomment anything in modules.conf to compile then I would consider it on shaky ground unless otherwise noted. Even Traun said mod_python needs to be removed or rewritten as of last week. Which could happen when Anthony returns from vacation and reads your email sitting in the consulting queue. /b On Jun 13, 2008, at 1:17 PM, Kre?imir Tonkovi? wrote:
Funny thing... I offered to pay for a fix for this bug by sending a mail to consulting@freeswitch.org, but nobody responded. Now I spent some of that money reimplementing the ivr in javascript. The rest is still there if somebody wants it :-)
Regarding pythonistas and stake burnings - I just want the humongous number of libraries that come with python.
I agree with David - pieces that aren't quite production ready should be marked as such.
Regards, Kresho
On Fri, Jun 13, 2008 at 7:05 PM, Brian West <brian@freeswitch.org> wrote: Also let me just say one last thing here.. As long as their is a bug on Jira about this issue we will get around to it... it's just not a top priority right now.
/b
On Jun 13, 2008, at 10:49 AM, David Knell wrote:
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1].
But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help.
--Dave
[1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which
is
what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
Hi!
I'm new to freeswitch and I like to report my success with it and a few failures. I'll be a little bit vague on some details because I must protect some business details. Sorry for that.
I have no experience with asterisk, so many concepts were new to me.
We run a hosted IVR system with a few hundred lines. We have a few servers running a SIP/VoiceXML application server and connect to the network with SIP/ISDN gateways.
Recently we started an IVR with very short calls and very high CPS. Our existing software doesn't handle this scenario very well, so I started looking into alternatives.
FreeSwitch caught my eye because of its support for multiple scripting languages. I love python and this feature put FS into the evaluation list. So I started on friday. I installed FS from the debian repositories on my ubuntu 8.04 laptop and tried some examples from the wiki ("Some thing to try out!"). I was very impressed that everything worked right out of the box.
I was a little disappointed that mod_python wasn't included in the distribution so I checked out the source and compiled everything. An hour later I had another installation of FS.
It took me a few hous to get the dialplan right. Because our service only runs IVRs and uses no switching, I removed everything from the default dialplan (mainly because it conflicted with the ANI numbers we get from the gateways).
Another hour later, I had a simple IVR in python done: use a web service for a database lookup and play an appropriate prompt. I didn't use the database directly because I wanted the best possible comparison to what our current system does, and (our) VoiceXML can't use databases directly, but can use web services.
In less than 1 working day I had everything running. Quite good.
Time for load testing :-) Our old software handles around 20 CPS on my laptop. I inceased max_sessions to 5000 and sessions-per- second to 100 and started sipp. The result was quite bad - I could not get over 8 CPS! The processor barely noticed that FS was running, so I had no idea what the bottleneck was. I still don't. After fiddling with this for a while, I gave up and decided to try it on one of our production machines. Weekends are not very busy, so I took one offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, 2G ecc ram and 10krpm disks. The server is also running ubuntu 8.04 (server edition) so I just copied the binaries.
I ran sipp from another machine, with the uac scenario and limiting the call duration to 4secs: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> Theoretically, as each call lasts 4 seconds, the total calls number should never exceed 4x current CPS.
These are the results:
With up to 27 CPS everything was stable. The calls count was almost exactly 4 timee the CPS, indicating that new calls were ansewered immediately. This I also verified by calling in.
Up to 30 CPS everything was stable for a while, but then the total calls number exploded to the limit set by sipp. The processor load was very reasonable, so I again I ran into the bottleneck mentioned above. After sipp hits the total call limit, it will not create new calls until some are released. So CPS oscillated between 0 and 30 as shown by sipp. CDRs show that there was an average of 27 CPS. At this point, when I called in, I got ringback tone for as long as the operator allows (60s) and then I was dropped. With a softphone I could reach the IVR after about 80 sec.
When I set sipp to more than 30 CPS, the number of total calls exploded immediately.
Experimenting some more, I found I could contain the explosion (and the instability in CPS) by limiting the number of total calls to 4x current cps when cps was up to 30. Thus, by starting sipp like this: sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 I could go up to 30CPS and get reasonably stable real 30CPS. When calling in with a real phone, I would reach the IVR after 2-3 seconds of ringback, which is acceptable. This simulation run for several hours without any other problems.
With sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 and setting cps to 40, the total calls count obviously never passed 160, but the cps shown by sipp became unstable, oscillating between 0 and 40.
These results are only slightly better than our current SIP server.
Today I put FreeSwitch into production. The unexpected thing here was that when FreeSwitch talked to our gateways instead of sipp, it crashed a few times. I don't associate these crashes with load because it happened equally on low and high load. Here's the log:
2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 switch_core_session_thread() Session 166 (sofia/internal/---- deleted stuff -----) Ended 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 switch_core_session_thread() Close Channel sofia/internal/---- deleted stuff ----- [CS_HANGUP] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 print_trace() Obtained 10 stack frames. /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] [0xb7f69420] /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) [0xb0076907] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) [0xb00788bc] /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] /usr/lib/libpython2.5.so.1.0 [0xb001667f] 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 switch_core_session_run() Thread has crashed for channel sofia/ internal/---- deleted stuff -----
It seems like mod_python is not quite ready for production yet :-) I have a few core dumps available on demand, cca 2MB each.
Turning crash protection on didn't help. FreeSwitch would reject new calls, and wouldn't shutdown completely. I had to kill it.
I would still like to replace our existing system with FreeSwitch because I find way more comfortable to work with. It has great potential and I'm sure it is being succesully deployed in many places as I write this.
The python crash is probably a simple bug. The invisible bottleneck is what troubles me more. Any help would be greatly appreciated.
Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). -- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
What is really needed is an associated "cost' with the modules. Granted, that is a hard thing to get a defined metric for. Blanket statements like "Freeswitch does millions of calls a day, and 500 CPS, or whatever".. arent very useful to the rest of us without knowing the configuration they are using. Even if you rewrite your IVR in javascript, you still might have problems. Spidermonkey itself has some global contention in it too and is not going to scale linearly. But at what point is that even noticed? Given some predefined reference platform..it would be nice to look at a chart of modules mod_lua = 500 CPS mod_spidermonkey = 300 CPS mod_python = 50 CPS 2008/6/13 Brian West <brian@freeswitch.org>:
Kresho, Thats because Anthony is on vacation and he didn't write mod_python. I should have responded to you and let you know that. He's the one that has to make the decision. Its something we have discussed in the past few weeks. As for production ready modules if you have to uncomment anything in modules.conf to compile then I would consider it on shaky ground unless otherwise noted. Even Traun said mod_python needs to be removed or rewritten as of last week. Which could happen when Anthony returns from vacation and reads your email sitting in the consulting queue. /b On Jun 13, 2008, at 1:17 PM, Kre?imir Tonkovi? wrote:
Funny thing... I offered to pay for a fix for this bug by sending a mail to consulting@freeswitch.org, but nobody responded. Now I spent some of that money reimplementing the ivr in javascript. The rest is still there if somebody wants it :-)
Regarding pythonistas and stake burnings - I just want the humongous number of libraries that come with python.
I agree with David - pieces that aren't quite production ready should be marked as such.
Regards, Kresho
On Fri, Jun 13, 2008 at 7:05 PM, Brian West <brian@freeswitch.org> wrote:
Also let me just say one last thing here.. As long as their is a bug on Jira about this issue we will get around to it... it's just not a top priority right now.
/b
On Jun 13, 2008, at 10:49 AM, David Knell wrote:
Hi Brian,
David, It has to work? We have people running millions of min. a day thru FreeSWITCH and they have zero issues.
Yes, but that misses the point. They run millions of minutes through a well-tested subset of that which FS offers - we ran half a million mins a day during May through a couple of FS boxes with the odd corrupt CDR and occasional barf[1].
But that doesn't mean that FS works in its entirety. It means that it can be made to work, which is a very different thing.
If you have to add the module to modules.conf and compile it.. that is outside the scope of what is well tested and supported. Lua is more tested but still isn't in the default yet. That might happen in 1.0.1.
Here's the solution, then. Someone who knows the answers could document the module status in modules.conf; that way, anyone who adds a less-well-supported module to their build will at least know about it. As a quick aside, mod_xml_curl isn't in the default build; it is OK for production use, isn't it..?!
By the tone of your comments you're going to take mod_python under your wing and make sure it works? We do need more people to step up and help. ;)
No, I'm not, I'm afraid - (a) I've too much on my plate to have a meaningful stab at it, and (b) I'm quite happy with curly brackets to indicate where my blocks start and end. But if anyone wants a hand with writing IVRs or doing call routing using event sockets and Perl, I'd be very happy to help.
--Dave
[1] Mindful of the possibility of having already offended the Pythonistas by disrespecting whitespace as a syntactic tool, I probably shouldn't add that the * boxes which front-end that traffic worked without a glitch, less I find myself on the wrong end of a stake, a gallon of petrol, a pile of wood and a match ;-)
/b
On Jun 13, 2008, at 9:34 AM, David Knell wrote:
Yes, but..
Neither of Kresho's boxes are registering heavy CPU usage, which is what one would expect if this were to do with the overhead of Python. In addition, there's the crashes.
On to a hobby horse of mine. If FS is going to make it in the real world, then it has to work. And that means that the bits which come with it need to work, and those which don't need to be separated off into a 'beta', 'don't expect much from this', 'FFS don't use unless you're prepared to fix it' or whatever repository.
New users, such as Kresho, will use whichever scripting tool/ language/ interface suits their needs and that they're familiar with. And they'll have the expectation that it'll work, and that expectation is perfectly reasonable. Keeping bits in the "release" source tree which don't work properly or which have failings which make them unsuitable for production use does FS (which is a great thing) a disservice.
Cheers --
Dave
Python is very heavy.. You should try lua.
/b
On Jun 12, 2008, at 5:29 AM, Krešimir Tonković wrote:
> Hi! > > I'm new to freeswitch and I like to report my success with it > and a > few failures. I'll be a little bit vague on some details because I > must protect some business details. Sorry for that. > > I have no experience with asterisk, so many concepts were new to > me. > > We run a hosted IVR system with a few hundred lines. We have a few > servers running a SIP/VoiceXML application server and connect to > the > network with SIP/ISDN gateways. > > Recently we started an IVR with very short calls and very high > CPS. > Our existing software doesn't handle this scenario very well, so I > started looking into alternatives. > > FreeSwitch caught my eye because of its support for multiple > scripting languages. I love python and this feature put FS into > the > evaluation list. So I started on friday. I installed FS from the > debian repositories on my ubuntu 8.04 laptop and tried some > examples > from the wiki ("Some thing to try out!"). I was very impressed > that > everything worked right out of the box. > > I was a little disappointed that mod_python wasn't included in the > distribution so I checked out the source and compiled everything. > An > hour later I had another installation of FS. > > It took me a few hous to get the dialplan right. Because our > service > only runs IVRs and uses no switching, I removed everything from > the > default dialplan (mainly because it conflicted with the ANI > numbers > we get from the gateways). > > Another hour later, I had a simple IVR in python done: use a web > service for a database lookup and play an appropriate prompt. I > didn't use the database directly because I wanted the best > possible > comparison to what our current system does, and (our) VoiceXML > can't > use databases directly, but can use web services. > > In less than 1 working day I had everything running. Quite good. > > Time for load testing :-) Our old software handles around 20 CPS > on > my laptop. I inceased max_sessions to 5000 and sessions-per-second > to 100 and started sipp. The result was quite bad - I could not > get > over 8 CPS! The processor barely noticed that FS was running, so I > had no idea what the bottleneck was. I still don't. After fiddling > with this for a while, I gave up and decided to try it on one of > our > production machines. Weekends are not very busy, so I took one > offline. It's a HP proliant server with 1 quad-core xeon on 2 GHz, > 2G ecc ram and 10krpm disks. The server is also running ubuntu > 8.04 > (server edition) so I just copied the binaries. > > I ran sipp from another machine, with the uac scenario and > limiting > the call duration to 4secs: > sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> > Theoretically, as each call lasts 4 seconds, the total calls > number > should never exceed 4x current CPS. > > These are the results: > > With up to 27 CPS everything was stable. The calls count was > almost > exactly 4 timee the CPS, indicating that new calls were ansewered > immediately. This I also verified by calling in. > > Up to 30 CPS everything was stable for a while, but then the total > calls number exploded to the limit set by sipp. The processor load > was very reasonable, so I again I ran into the bottleneck > mentioned > above. After sipp hits the total call limit, it will not create > new > calls until some are released. So CPS oscillated between 0 and 30 > as > shown by sipp. CDRs show that there was an average of 27 CPS. At > this point, when I called in, I got ringback tone for as long as > the > operator allows (60s) and then I was dropped. With a softphone I > could reach the IVR after about 80 sec. > > When I set sipp to more than 30 CPS, the number of total calls > exploded immediately. > > Experimenting some more, I found I could contain the explosion > (and > the instability in CPS) by limiting the number of total calls to > 4x > current cps when cps was up to 30. Thus, by starting sipp like > this: > sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 120 > I could go up to 30CPS and get reasonably stable real 30CPS. When > calling in with a real phone, I would reach the IVR after 2-3 > seconds of ringback, which is acceptable. This simulation run for > several hours without any other problems. > > With > sipp <FS server ip> -sn uac -d 4000 -s <ivr_number> -l 160 > and setting cps to 40, the total calls count obviously never > passed > 160, but the cps shown by sipp became unstable, oscillating > between > 0 and 40. > > These results are only slightly better than our current SIP > server. > > Today I put FreeSwitch into production. The unexpected thing here > was that when FreeSwitch talked to our gateways instead of sipp, > it > crashed a few times. I don't associate these crashes with load > because it happened equally on low and high load. Here's the log: > > 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:753 > switch_core_session_thread() Session 166 (sofia/internal/---- > deleted stuff -----) Ended > 2008-06-09 09:34:33 [NOTICE] switch_core_session.c:755 > switch_core_session_thread() Close Channel sofia/internal/---- > deleted stuff ----- [CS_HANGUP] > 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:218 > print_trace() Obtained 10 stack frames. > /usr/local/freeswitch/lib/libfreeswitch.so.1 [0xb7e413b1] > [0xb7f69420] > /usr/local/freeswitch/mod/mod_python.so [0xb011b46a] > /usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0xfa) [0xb002a50a] > /usr/lib/libpython2.5.so.1.0(PyObject_Call+0x37) [0xafff38d7] > /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x4067) > [0xb0076907] > /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] > /usr/lib/libpython2.5.so.1.0(PyEval_EvalFrameEx+0x601c) > [0xb00788bc] > /usr/lib/libpython2.5.so.1.0(PyEval_EvalCodeEx+0x748) [0xb007a368] > /usr/lib/libpython2.5.so.1.0 [0xb001667f] > 2008-06-09 09:34:33 [CRIT] switch_core_state_machine.c:319 > switch_core_session_run() Thread has crashed for channel sofia/ > internal/---- deleted stuff ----- > > It seems like mod_python is not quite ready for production > yet :-) I > have a few core dumps available on demand, cca 2MB each. > > Turning crash protection on didn't help. FreeSwitch would reject > new > calls, and wouldn't shutdown completely. I had to kill it. > > I would still like to replace our existing system with FreeSwitch > because I find way more comfortable to work with. It has great > potential and I'm sure it is being succesully deployed in many > places as I write this. > > The python crash is probably a simple bug. The invisible > bottleneck > is what troubles me more. Any help would be greatly appreciated. > > Finally, this is all with FreeSwitch Version 1.0.pre4 (8760). > -- > kresho > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- kresho _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
You really can't do a metric like that. Its all going to depend on what you're doing in your scripts how fast you can go. If you really want speed you need to be using outbound event socket which is like Fast AGI but 10 times more powerful. On Jun 13, 2008, at 1:56 PM, Matt Porter wrote:
What is really needed is an associated "cost' with the modules. Granted, that is a hard thing to get a defined metric for.
Blanket statements like "Freeswitch does millions of calls a day, and 500 CPS, or whatever".. arent very useful to the rest of us without knowing the configuration they are using. Even if you rewrite your IVR in javascript, you still might have problems. Spidermonkey itself has some global contention in it too and is not going to scale linearly. But at what point is that even noticed?
Given some predefined reference platform..it would be nice to look at a chart of modules
mod_lua = 500 CPS mod_spidermonkey = 300 CPS mod_python = 50 CPS
Brian West sip:brian@freeswitch.org
you could do a metric for CPS, with an (almost) empty script. which would give you only the "startup" cost of a module. And thats going to be very different for the modules. But, i know where you are coming from.. we maintain a fork of OpenVXI we get the same kind of loaded questions.. Q: "how many ports can we do per box?" A: "depends on what your scripts do" Love those questions ;) On Fri, Jun 13, 2008 at 2:03 PM, Brian West <brian@freeswitch.org> wrote:
You really can't do a metric like that. Its all going to depend on what you're doing in your scripts how fast you can go. If you really want speed you need to be using outbound event socket which is like Fast AGI but 10 times more powerful.
On Jun 13, 2008, at 1:56 PM, Matt Porter wrote:
What is really needed is an associated "cost' with the modules. Granted, that is a hard thing to get a defined metric for.
Blanket statements like "Freeswitch does millions of calls a day, and 500 CPS, or whatever".. arent very useful to the rest of us without knowing the configuration they are using. Even if you rewrite your IVR in javascript, you still might have problems. Spidermonkey itself has some global contention in it too and is not going to scale linearly. But at what point is that even noticed?
Given some predefined reference platform..it would be nice to look at a chart of modules
mod_lua = 500 CPS mod_spidermonkey = 300 CPS mod_python = 50 CPS
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I must also apologize about being a bit defensive on this issue. Its one we know about have been talking about but with Anthony gone on vacation and MikeJ being sick its been a rough job for me to keep up with Cluecon, IRC, Jira, Consulting emails and the mailing lists plus about the time all this was going down I had this really crazy woman beating on my door which this morning screaming for me to call the cops because her boy friend was armed with a taser and she had her baby in her arms so as you can tell my blood pressure today is a bit high in the first place. I mean really what are the odds.... needless to say when I called the cops and said "he's armed with a taser" they sent the big boys after him. I mean can Friday the 13th get any worse? :P /b On Jun 13, 2008, at 2:17 PM, Matt Porter wrote:
you could do a metric for CPS, with an (almost) empty script. which would give you only the "startup" cost of a module. And thats going to be very different for the modules.
But, i know where you are coming from.. we maintain a fork of OpenVXI we get the same kind of loaded questions.. Q: "how many ports can we do per box?" A: "depends on what your scripts do"
Love those questions ;)
On Jun 13, 2008, at 10:27 PM, Brian West wrote:
I had this really crazy woman beating on my door which this morning screaming for me to call the cops because her boy friend was armed with a taser and she had her baby in her arms so as you can tell my blood pressure today is a bit high in the first place. I mean really what are the odds.... needless to say when I called the cops and said "he's armed with a taser" they sent the big boys after him. I mean can Friday the 13th get any worse? :P
I think so. You could have been the boyfriend..! --Dave
This problem that most of you are ignoring is that there are several things that affect the performance of FreeSwitch Hardware, Ram, Actual Switch Configuration... And can FreeSwitch do 500 CPS, Yes it can... What config can I confirm will work 1) Dell 1950, Dual E5335 2Ghz Quad core CPUs, Gentoo Linux, 4G Ram 2) -nosql, /usr/local/freeswitch/db mounted in a ram drive 3) sofia profile has presence tracking turned off, and has auth calls turned off 4) Custom Dialplan module written in C accessing separate Database using a proprietary LCR Algorithm for the routing 5) RTP in no media mode where possible for the end points Actual CPS volumes on a single box in this configuration have exceeded 500 cps and actual concurrent 2 leg calls has exceeded 4000 And this configuration runs extremely stable... (this configuration is also commercially available and supported email me off list for details)
From: Brian West <brian@freeswitch.org> Reply-To: <freeswitch-users@lists.freeswitch.org> Date: Fri, 13 Jun 2008 14:03:47 -0500 To: <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Installation report, a python crash and a bottleneck
You really can't do a metric like that. Its all going to depend on what you're doing in your scripts how fast you can go. If you really want speed you need to be using outbound event socket which is like Fast AGI but 10 times more powerful.
On Jun 13, 2008, at 1:56 PM, Matt Porter wrote:
What is really needed is an associated "cost' with the modules. Granted, that is a hard thing to get a defined metric for.
Blanket statements like "Freeswitch does millions of calls a day, and 500 CPS, or whatever".. arent very useful to the rest of us without knowing the configuration they are using. Even if you rewrite your IVR in javascript, you still might have problems. Spidermonkey itself has some global contention in it too and is not going to scale linearly. But at what point is that even noticed?
Given some predefined reference platform..it would be nice to look at a chart of modules
mod_lua = 500 CPS mod_spidermonkey = 300 CPS mod_python = 50 CPS
Brian West sip:brian@freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
Brian West
-
David Knell
-
Ken Rice
-
Krešimir Tonković
-
Matt Porter