Testing Freeswitch performance led to strange behavior
In the process of trying to use Freeswitch in a production environment I conducted a number of performance tests using various servers. It was then that I noticed some strange behavior from FS. When I stripped down the scenario I was using to a simple bridge scenario, I stumbled upon a strange behavior. The scenario as I stated is quite simple. |---------| |---------| | |------- Call from sipp------> | | | sipp | | FS | | | <------ Call back to sipp----| | |---------| |---------| I did not use an RTP stream for my calls just to test the signaling alone. The sipp scenario is the standard uac.xml scenario that can be found integrated to sipp with the following options : Test FS 1: sipp <FS_IP>:5060 -s 55555555 -i <SIPP_IP> -mi <SIPP_IP> -ci <SIPP_IP> -r 10 -d 5000 -l 100 -m 1000 -sf uac.xml Calls : 1000 Successful calls : 1000 Idle CPU during tests : ~(35-60) % (35 during the generation of new calls, 60 during the -l limit imposed by the test) Note : 985 of them had a duration (billsec) of 10 and 15 of them had a duration of 11. I tried raising the call rate and limit... Test FS 1: sipp <FS_IP>:5060 -s 55555555 -i <SIPP_IP> -mi <SIPP_IP> -ci <SIPP_IP> -r 20 -d 5000 -l 200 -m 1000 -sf uac.xml Calls : 1000 Successful calls : 1000 Idle CPU during tests : ~0-30 % (0 during the generation of new calls, 30 during the -l limit imposed by the test) THIS IS WHAT MAKES ME WONDER : The distribution of the durations (billsec - not complete durations) : 183 calls with 10 secs billed duration 110 calls with 11 secs billed duration 238 calls with 12 secs billed duration 447 calls with 13 secs billed duration 22 calls with 14 secs billed duration The sipp scenario is simple "hangup the phone after 10 secs". So, why am I seeing these? Of course that has something to do with the stress the machine has been put through during the second test. But I can see it happening to less stressful conditions (i.e. 15 calls per second) to a smaller extend. I captured one of these calls and verified that when the sipp client hangs up exactly 10 secs after the call start, FS receives the BYE and replies with 200 OK. BUT it does not hang the second leg in a timely manner i.e. it sends a BYE to the sipp server side 1-4 seconds AFTER that. That explains the 11, 12, 13, 14 secs durations seen on the second test. What is more interesting is that I would expect to see in the CDRs the first and second leg to have different durations (since the a leg BYE was received and aknowledged by FS in the correct time) i.e. 10 and 14 secs accordingly. But what I get is the same duration for both legs (14 secs). This in my opinion is very dangerous on production environments (you get charged by your provider more seconds that you charge your clients - or - you falsely charge your clients with bigger durations although they hunged up corectly (and you acknowledged it)). NOTE No 1 : All the performance recommendations found in the wiki has been applied. In fact only the essential modules that could make this scenario work were loaded. NOTE No 2 : I tried using asterisk (as a point of reference - don't get me wrong - I am not trying to start a flame war here). And it succeeded doing on the same hardware 60 calls/sec with a channel limit of 400 sim. calls using only 50% of the cpu (maximum). No under any circumstances I have seen the behavior above (this inability to hang call legs in a timely manner). Even when I pushed asterisk to the limits (80 calls per second 600 max call limit) and it started failing on some calls it never failed to hangup the calls for both legs on exactly 10 secs. NOTE No 3 : As you can tell I was using a very small machine for my tests. When I moved the same tests to larger installations (Quad Core Opterons and Xeons) I got proportional results to the above. NOTE No 4 : The tests were performed in a LAN environment and since there was no RTP involved I think there were no bandwidth issues there. NOTE No 5 : The tests were performed using numerous SVN versions (latest : 13610), the stable version and the 1.0.4pre8 version. NOTE No 6 : Using the -hp switch made no noticeable change in behavior. I am not trying to complain for FS's performance (far from it). I am just somewhat disappointed seeing it performing in such a strange manner when under stress. I would prefer a design that drops the calls after a certain threshold than a design that incorrectly handles them all (I am aware of the max sessions per second in switch.conf.cml - but I am starting to see this behavior even with the cpu idling at about 80%). I don't know if anyone else had the same experience when testing Freeswitch. I can happily supply with all the test details (config files, captures etc) to all interested parties. -- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com R & D email: regs@kinetix.gr -------------------------------------------
On Jun 4, 2009, at 9:32 AM, Apostolos Pantsiopoulos wrote:
NOTE No 1 : All the performance recommendations found in the wiki has been applied. In fact only the essential modules that could make this scenario work were loaded.
What are you testing against? What OS, Hardware, Distro and such?
NOTE No 2 : I tried using asterisk (as a point of reference - don't get me wrong - I am not trying to start a flame war here). And it succeeded doing on the same hardware 60 calls/sec with a channel limit of 400 sim. calls using only 50% of the cpu (maximum). No under any circumstances I have seen the behavior above (this inability to hang call legs in a timely manner). Even when I pushed asterisk to the limits (80 calls per second 600 max call limit) and it started failing on some calls it never failed to hangup the calls for both legs on exactly 10 secs.
Load testing is a science and you can do it wrong most of the time unless you know exactly what you're doing. If you're going against the default dialplan its heavy and not something I would load test against.
NOTE No 3 : As you can tell I was using a very small machine for my tests. When I moved the same tests to larger installations (Quad Core Opterons and Xeons) I got proportional results to the above.
What are you testing on now? Hope its 64bit. Brian West brian@freeswitch.org -- Meet us at ClueCon! http://www.cluecon.com
Brian West wrote:
On Jun 4, 2009, at 9:32 AM, Apostolos Pantsiopoulos wrote:
NOTE No 1 : All the performance recommendations found in the wiki has been applied. In fact only the essential modules that could make this scenario work were loaded.
What are you testing against? What OS, Hardware, Distro and such?
The small server tests were made on a 5-year old PC (32 bit, 3 Ghz P4, Cetnos 5.3). The large server 1 : Quad-Core AMD Opteron(tm) Processor 2350 HE (64 bit, Centos 5.3) The large server 2 : Dual-Core AMD Opteron(tm) Processor 2214 HE (64 bit, Centos 5.3) The large server 3 : Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (32 bit, Centos 5.3)
NOTE No 2 : I tried using asterisk (as a point of reference - don't get me wrong - I am not trying to start a flame war here). And it succeeded doing on the same hardware 60 calls/sec with a channel limit of 400 sim. calls using only 50% of the cpu (maximum). No under any circumstances I have seen the behavior above (this inability to hang call legs in a timely manner). Even when I pushed asterisk to the limits (80 calls per second 600 max call limit) and it started failing on some calls it never failed to hangup the calls for both legs on exactly 10 secs.
Load testing is a science and you can do it wrong most of the time unless you know exactly what you're doing. If you're going against the default dialplan its heavy and not something I would load test against.
The dialplan : <?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include> <context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="^.*$"> <!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMA"/> <action application="bridge" data="sofia/gateway/sipp01/$1"/> </condition> </extension> </context> </include> I think it is the simplest that can be used in this scenario.
NOTE No 3 : As you can tell I was using a very small machine for my tests. When I moved the same tests to larger installations (Quad Core Opterons and Xeons) I got proportional results to the above.
What are you testing on now? Hope its 64bit.
Most of the platforms were 64 bit (although the results that I posted were from the small 32-bit server, the results from the 64-bit servers were proportional to those). In other words we needed a large call/sec rate for the high end servers but in any case the same phenomenon occured at around 60% idle cpu. -- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com R & D email: regs@kinetix.gr -------------------------------------------
The dialplan :
<?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include>
<context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="^.*$">
You forgot the parens around .* It should be expression="^(.*)$" if you plan to use $1 later in the extension...
<!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMA"/> <action application="bridge" data="sofia/gateway/sipp01/$1"/>
... like here ^^^^^^^ :) -MC
</condition> </extension> </context>
</include>
Michael Collins wrote:
The dialplan :
<?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include>
<context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="^.*$">
You forgot the parens around .* It should be expression="^(.*)$" if you plan to use $1 later in the extension...
<!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMA"/> <action application="bridge" data="sofia/gateway/sipp01/$1"/>
... like here ^^^^^^^ :) -MC
You are right! Although, I don't think that would change the outcome of my test :)
</condition> </extension> </context>
</include>
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
FS uses async rtp timers so you may want to set rtp-timer-name=none in the profile param to simulate asterisk conditions. Also keep in mind that asterisk as an atvantage in a tiny crappy 32 bit single cpu box because that was what was popular when it was designed and the chance for race conditions is minimal because there is only 1 cpu. As you scale up to a 8 core 64 bit xeon you will set a drastic difference. I will be happy to investigate this issue a bit if you'd like but i do not have any box like you describe so if I can't find anything you may have to lend us your lab. On Thu, Jun 4, 2009 at 12:47 PM, regs@kinetix.gr <regs@kinetix.gr> wrote:
Michael Collins wrote:
The dialplan :
<?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include>
<context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="^.*$">
You forgot the parens around .* It should be expression="^(.*)$" if you plan to use $1 later in the extension...
<!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMA"/> <action application="bridge" data="sofia/gateway/sipp01/$1"/>
... like here ^^^^^^^ :) -MC
You are right! Although, I don't think that would change the outcome of my test :)
</condition> </extension> </context>
</include>
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Anthony Minessale wrote:
FS uses async rtp timers so you may want to set rtp-timer-name=none in the profile param to simulate asterisk conditions.
I tried that - although I am not using rtp in my scenario - with the same results.
Also keep in mind that asterisk as an atvantage in a tiny crappy 32 bit single cpu box because that was what was popular when it was designed and the chance for race conditions is minimal because there is only 1 cpu. As you scale up to a 8 core 64 bit xeon you will set a drastic difference.
Yes I know that this machine is not well suited for today's test needs. But the issue occurs in every machine as long as it is pushed near (but not quite near) to its limits. I have the same odd durations using a 64 bit low end server. In this case I could achieve a better call/sec rate than that of the crappy PC but around 50-60 calls/sec the same problem showed up. I also used a Mosso Cloud Server (quad core - 64-bit) and the same thing happened at a higher rate.
I will be happy to investigate this issue a bit if you'd like but i do not have any box like you describe so if I can't find anything you may have to lend us your lab.
I would appreciate it if you did. After all there this might be a problem that has not surfaced yet but someday will as more and more production boxes start using FS. So it would be better to investigate it now. I don't think lending you access to my old P4 PC would help you very much :) If you have access to a normal 2-4 core system you can easily start raising the sipp parameters until it starts happening. However if you really think it is appropriate to use my test machines I'd be happy to grant access to our low-end Opteron machine (just send me a personal email). I cannot grant you access to larger systems because they are used in production. I used the embedded sipp scenarios : on the UAS side : sipp -i <UAS_IP> -mi <UAS_IP> -ci <UAS_IP> -mp 8000 -sn uas on the UAC side : sipp <FS_IP>:5060 -s 44050505-i <UAC_IP> -mi <UAC_IP> -ci <UAC_IP> -r 70 -d 5000 -l 500 -m 2000 -sn uac The dialplan : <?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include> <context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="(^.*)$"> <!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMU"/> <!-- <action application="set" data="proxy_media=true"/> --> <action application="bridge" data="sofia/gateway/sipp01/$1"/> </condition> </extension> </context> </include> If you need anything else from the config just notify me. In order to verify that at some point the calls start having a duration larger than the scenario's 5secs you can tcpdump on the sipp machine or turn on the cdrs logging (I know that it degrades performance, but as I said it is not a matter of when exactly it starts happening, it is a matter that it DOES start happening).
On Thu, Jun 4, 2009 at 12:47 PM, regs@kinetix.gr <mailto:regs@kinetix.gr> <regs@kinetix.gr <mailto:regs@kinetix.gr>> wrote:
Michael Collins wrote: > > > The dialplan : > > <?xml version="1.0" encoding="utf-8"?> > <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> > <include> > > <context name="mydialplan"> > <extension name="dial1"> > <condition field="destination_number" > expression="^.*$"> > > > You forgot the parens around .* > It should be expression="^(.*)$" if you plan to use $1 later in the > extension... > > > > <!-- Dial Back --> > <action application="set" > data="absolute_codec_string=PCMA"/> > <action application="bridge" > data="sofia/gateway/sipp01/$1"/> > > ... like here ^^^^^^^ > :) > -MC
You are right! Although, I don't think that would change the outcome of my test :) > > > > </condition> > </extension> > </context> > > </include> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 <http://iax:guest@conference.freeswitch.org/888> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com R & D email: regs@kinetix.gr -------------------------------------------
I am writing this to let you know that this behavior persists in the 1.0.4pre9. Could the calls/sec issue be due to the single threaded nature of Sofia? Because I am getting the feeling that the number of simultaneous channels doesn't really burdens FS, but many Calls/sec does. Apostolos Pantsiopoulos wrote:
Anthony Minessale wrote:
FS uses async rtp timers so you may want to set rtp-timer-name=none in the profile param to simulate asterisk conditions.
I tried that - although I am not using rtp in my scenario - with the same results.
Also keep in mind that asterisk as an atvantage in a tiny crappy 32 bit single cpu box because that was what was popular when it was designed and the chance for race conditions is minimal because there is only 1 cpu. As you scale up to a 8 core 64 bit xeon you will set a drastic difference.
Yes I know that this machine is not well suited for today's test needs. But the issue occurs in every machine as long as it is pushed near (but not quite near) to its limits. I have the same odd durations using a 64 bit low end server. In this case I could achieve a better call/sec rate than that of the crappy PC but around 50-60 calls/sec the same problem showed up. I also used a Mosso Cloud Server (quad core - 64-bit) and the same thing happened at a higher rate.
I will be happy to investigate this issue a bit if you'd like but i do not have any box like you describe so if I can't find anything you may have to lend us your lab.
I would appreciate it if you did. After all there this might be a problem that has not surfaced yet but someday will as more and more production boxes start using FS. So it would be better to investigate it now. I don't think lending you access to my old P4 PC would help you very much :) If you have access to a normal 2-4 core system you can easily start raising the sipp parameters until it starts happening. However if you really think it is appropriate to use my test machines I'd be happy to grant access to our low-end Opteron machine (just send me a personal email). I cannot grant you access to larger systems because they are used in production.
I used the embedded sipp scenarios :
on the UAS side :
sipp -i <UAS_IP> -mi <UAS_IP> -ci <UAS_IP> -mp 8000 -sn uas
on the UAC side :
sipp <FS_IP>:5060 -s 44050505-i <UAC_IP> -mi <UAC_IP> -ci <UAC_IP> -r 70 -d 5000 -l 500 -m 2000 -sn uac
The dialplan :
<?xml version="1.0" encoding="utf-8"?> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> <include>
<context name="mydialplan"> <extension name="dial1"> <condition field="destination_number" expression="(^.*)$"> <!-- Dial Back --> <action application="set" data="absolute_codec_string=PCMU"/> <!-- <action application="set" data="proxy_media=true"/> --> <action application="bridge" data="sofia/gateway/sipp01/$1"/> </condition> </extension> </context>
</include>
If you need anything else from the config just notify me.
In order to verify that at some point the calls start having a duration larger than the scenario's 5secs you can tcpdump on the sipp machine or turn on the cdrs logging (I know that it degrades performance, but as I said it is not a matter of when exactly it starts happening, it is a matter that it DOES start happening).
On Thu, Jun 4, 2009 at 12:47 PM, regs@kinetix.gr <mailto:regs@kinetix.gr> <regs@kinetix.gr <mailto:regs@kinetix.gr>> wrote:
Michael Collins wrote: > > > The dialplan : > > <?xml version="1.0" encoding="utf-8"?> > <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML --> > <include> > > <context name="mydialplan"> > <extension name="dial1"> > <condition field="destination_number" > expression="^.*$"> > > > You forgot the parens around .* > It should be expression="^(.*)$" if you plan to use $1 later in the > extension... > > > > <!-- Dial Back --> > <action application="set" > data="absolute_codec_string=PCMA"/> > <action application="bridge" > data="sofia/gateway/sipp01/$1"/> > > ... like here ^^^^^^^ > :) > -MC
You are right! Although, I don't think that would change the outcome of my test :) > > > > </condition> > </extension> > </context> > > </include> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Freeswitch-users mailing list > Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 <http://iax:guest@conference.freeswitch.org/888> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
------------------------------------------------------------------------
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------- Apostolos Pantsiopoulos Kinetix Tele.com R & D email: regs@kinetix.gr -------------------------------------------
participants (5)
-
Anthony Minessale
-
Apostolos Pantsiopoulos
-
Brian West
-
Michael Collins
-
regs＠kinetix.gr