Hi guys, Doing a bit of testing / benchmarking with FS 1.04 (and 1.05 SVN). Found out some interesting things, I think. The setup is like this: SIPP Client -> FS -> SIPP Server The dialplan is as simple as it gets: <extension name="test-out"> <condition field="network_addr" expression="^130\.244\.190\.150$"/> <condition field="destination_number" expression="^(\d+)$"> <action application="set" data="proxy_media=true"/> <action application="set" data="hangup_after_bridge=true"/> <action application="bridge" data="{sip_contact_user=transit}sofia/gateway/${distributor(gwg1)}/$1|sofia/gateway/${distributor(gwg2)}/$1"/> </condition> </extension> For the rest it's running CSV cdr's, commented out all modules I'm not using, etc etc all that I could find on the wiki and the Interwebs. Hardware / OS: from the shelve quadcore Xeon, debian 64-bit, 12GB memory. SIPP is running a 500ms RTP pcap and the other side echos back. I had a few test setups then: 1: FS SVN, 1 sofia profile where the gateways were configured and the server_IP:5060 was used. 2. FS SVN, 2 sofia profiles where the gateways where in an seperate profile (server_IP:5070) and the "customer facing" side was the original profile. 3. FS 1.04, same as above 4. FS 1.04, 4 sofia profiles, distributor to spread load over 2 incoming and 2 outgoing profiles. Now the interesting thing was that under 1 I could go up, almost without any CPU load, to 50cps. As soon as I went over this, calls where handled slower and "ongoing" calls would pile up untill it became really slow. CPU load went to 100% on the FS process (both user and system time). Lots of interupts and context switches. No throughput anymore untill I lower and wait till the "buffer" is empty and FS is keeping up again. Under 2, I was able to increase the CPS to about 100 with the same effect. 3 then went much better, I was able to increase CPS to about 200 cps and response times in SIPP went up slighty untill it just hits some kind of limit and calls are handled slower. 4 is pretty cool. Here I can run 2 sipp clients both doing 150cps to the gateways that are spread on the distribution module, so I spread traffic over 2 profiles. With 300 cps in total, FS is keeping up and I have 30% idle CPU. However, increasing to over 300cps gives problems again, even though I have idle CPU left! All in all, I have a feeling that a single sip profile can't run more than a certain limit untill it gets into some problem. Depending on if I'm running SVN or 1.04 that limit seems to be 50cps or a bit higher. After that limit it starts piling up "ongoing" calls, by taking time to handle them and when that limit gets too high it's too late. All in all really fine, I just set the system wide limit to a little under that "threshold". But when I'm running just UNDER the threshold it's not CPU that's a problem. Theoretically I should be able to run (based on the CPU usage at 300cps) about 400cps. When running at 300 I get SOME failed calls and I see "switch_core_state_machine.c:525 a9a60636-0cea-11df-85a1-09c991f2afc5 Timeout waiting for next instruction in CS_NEW!" in the console. I didn't find much on how people do high cps setups and it feels a bit like a "friday afternoon solution" to run multiple sofia profiles on the same machine in order to max out the system. Maybe I'm missing something and I know it's not an exact science this, but I'm not sure "all is OK" because I'm not slowly getting to a 100% cpu (or disk / network) usage, I hit some kind of limit after which stuff goes wrong. Anyone any input! /Robin
2010. január 29. 16.31.29 Robin Vleij dátummal ezt írta:
Hi guys,
Doing a bit of testing / benchmarking with FS 1.04 (and 1.05 SVN). Found out some interesting things, I think.
The setup is like this:
SIPP Client -> FS -> SIPP Server
The dialplan is as simple as it gets:
<extension name="test-out"> <condition field="network_addr" expression="^130\.244\.190\.150$"/> <condition field="destination_number" expression="^(\d+)$"> <action application="set" data="proxy_media=true"/> <action application="set" data="hangup_after_bridge=true"/> <action application="bridge" data="{sip_contact_user=transit}sofia/gateway/${distributor(gwg1)}/$1|sofia /gateway/${distributor(gwg2)}/$1"/> </condition> </extension>
For the rest it's running CSV cdr's, commented out all modules I'm not using, etc etc all that I could find on the wiki and the Interwebs. Hardware / OS: from the shelve quadcore Xeon, debian 64-bit, 12GB memory. SIPP is running a 500ms RTP pcap and the other side echos back.
I had a few test setups then:
1: FS SVN, 1 sofia profile where the gateways were configured and the server_IP:5060 was used.
2. FS SVN, 2 sofia profiles where the gateways where in an seperate profile (server_IP:5070) and the "customer facing" side was the original profile.
3. FS 1.04, same as above
4. FS 1.04, 4 sofia profiles, distributor to spread load over 2 incoming and 2 outgoing profiles.
Now the interesting thing was that under 1 I could go up, almost without any CPU load, to 50cps. As soon as I went over this, calls where handled slower and "ongoing" calls would pile up untill it became really slow. CPU load went to 100% on the FS process (both user and system time). Lots of interupts and context switches. No throughput anymore untill I lower and wait till the "buffer" is empty and FS is keeping up again.
Under 2, I was able to increase the CPS to about 100 with the same effect.
3 then went much better, I was able to increase CPS to about 200 cps and response times in SIPP went up slighty untill it just hits some kind of limit and calls are handled slower.
4 is pretty cool. Here I can run 2 sipp clients both doing 150cps to the gateways that are spread on the distribution module, so I spread traffic over 2 profiles. With 300 cps in total, FS is keeping up and I have 30% idle CPU. However, increasing to over 300cps gives problems again, even though I have idle CPU left!
All in all, I have a feeling that a single sip profile can't run more than a certain limit untill it gets into some problem. Depending on if I'm running SVN or 1.04 that limit seems to be 50cps or a bit higher. After that limit it starts piling up "ongoing" calls, by taking time to handle them and when that limit gets too high it's too late. All in all really fine, I just set the system wide limit to a little under that "threshold". But when I'm running just UNDER the threshold it's not CPU that's a problem. Theoretically I should be able to run (based on the CPU usage at 300cps) about 400cps.
When running at 300 I get SOME failed calls and I see
"switch_core_state_machine.c:525 a9a60636-0cea-11df-85a1-09c991f2afc5 Timeout waiting for next instruction in CS_NEW!"
in the console.
I didn't find much on how people do high cps setups and it feels a bit like a "friday afternoon solution" to run multiple sofia profiles on the same machine in order to max out the system.
Maybe I'm missing something and I know it's not an exact science this, but I'm not sure "all is OK" because I'm not slowly getting to a 100% cpu (or disk / network) usage, I hit some kind of limit after which stuff goes wrong.
Anyone any input!
/Robin
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
CPU usage is not the only thing that limit your calls. Have you set the recommended ulimit settings and / or started fs with the -waste option ? http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations
Also keep in mind that the industry standard is 50 which is the capacity to take over for the real standard of 25 in a fail-over scenario. So you should be happy you even get 300cps for free. The sofia stack can be improved but we are not the creators of this sip stack. There is little to no work being done on that project right now and we are happy with what we have until we can get the lead dev to work on improving it with us when he has the time. On Sat, Jan 30, 2010 at 5:33 AM, Puskás Zsolt <errotan@gmail.com> wrote:
2010. január 29. 16.31.29 Robin Vleij dátummal ezt írta:
Hi guys,
Doing a bit of testing / benchmarking with FS 1.04 (and 1.05 SVN). Found out some interesting things, I think.
The setup is like this:
SIPP Client -> FS -> SIPP Server
The dialplan is as simple as it gets:
<extension name="test-out"> <condition field="network_addr" expression="^130\.244\.190\.150$"/> <condition field="destination_number" expression="^(\d+)$"> <action application="set" data="proxy_media=true"/> <action application="set" data="hangup_after_bridge=true"/> <action application="bridge"
data="{sip_contact_user=transit}sofia/gateway/${distributor(gwg1)}/$1|sofia
/gateway/${distributor(gwg2)}/$1"/> </condition> </extension>
For the rest it's running CSV cdr's, commented out all modules I'm not using, etc etc all that I could find on the wiki and the Interwebs. Hardware / OS: from the shelve quadcore Xeon, debian 64-bit, 12GB memory. SIPP is running a 500ms RTP pcap and the other side echos back.
I had a few test setups then:
1: FS SVN, 1 sofia profile where the gateways were configured and the server_IP:5060 was used.
2. FS SVN, 2 sofia profiles where the gateways where in an seperate profile (server_IP:5070) and the "customer facing" side was the original profile.
3. FS 1.04, same as above
4. FS 1.04, 4 sofia profiles, distributor to spread load over 2 incoming and 2 outgoing profiles.
Now the interesting thing was that under 1 I could go up, almost without any CPU load, to 50cps. As soon as I went over this, calls where handled slower and "ongoing" calls would pile up untill it became really slow. CPU load went to 100% on the FS process (both user and system time). Lots of interupts and context switches. No throughput anymore untill I lower and wait till the "buffer" is empty and FS is keeping up again.
Under 2, I was able to increase the CPS to about 100 with the same effect.
3 then went much better, I was able to increase CPS to about 200 cps and response times in SIPP went up slighty untill it just hits some kind of limit and calls are handled slower.
4 is pretty cool. Here I can run 2 sipp clients both doing 150cps to the gateways that are spread on the distribution module, so I spread traffic over 2 profiles. With 300 cps in total, FS is keeping up and I have 30% idle CPU. However, increasing to over 300cps gives problems again, even though I have idle CPU left!
All in all, I have a feeling that a single sip profile can't run more than a certain limit untill it gets into some problem. Depending on if I'm running SVN or 1.04 that limit seems to be 50cps or a bit higher. After that limit it starts piling up "ongoing" calls, by taking time to handle them and when that limit gets too high it's too late. All in all really fine, I just set the system wide limit to a little under that "threshold". But when I'm running just UNDER the threshold it's not CPU that's a problem. Theoretically I should be able to run (based on the CPU usage at 300cps) about 400cps.
When running at 300 I get SOME failed calls and I see
"switch_core_state_machine.c:525 a9a60636-0cea-11df-85a1-09c991f2afc5 Timeout waiting for next instruction in CS_NEW!"
in the console.
I didn't find much on how people do high cps setups and it feels a bit like a "friday afternoon solution" to run multiple sofia profiles on the same machine in order to max out the system.
Maybe I'm missing something and I know it's not an exact science this, but I'm not sure "all is OK" because I'm not slowly getting to a 100% cpu (or disk / network) usage, I hit some kind of limit after which stuff goes wrong.
Anyone any input!
/Robin
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
CPU usage is not the only thing that limit your calls. Have you set the recommended ulimit settings and / or started fs with the -waste option ?
http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
For clarification, is it correct that your getting worse numbers for sustainable cps on SVN then on 1.0.4? I would be interested in the numbers you would get with bypass_media=true instead of proxy_media=true and with neither setting set as well. Also, make sure your logging level is low and try putting the db dir on a ram disk. Thanks for the info. As tony said, there are clearly some bottlenecks in the sofia library, but if you really need to pass media, your test is not very accurate for real life, and the results are likely not very useful to you. You should use a more realistic length of call. In bypass media I would suspect that length of call matters very little. In proxy media or normal mode, the performance of the box is much more of a calculation on number of calls than cps as a result of the context switching from having to move the media and you will see that this plays a much more significant role on realistic call lengths (unless you really have lots of 1/2 second calls). Some other tips. While this extension may be trivial, what else is there in your dialplan context? Anything above that extension could cause a significant impact. Do you have any of the presence features enabled? These do significantly impact call handling performance even if your sipp scenarios do not send any of those packets. Mike On Jan 30, 2010, at 11:57 AM, Anthony Minessale wrote:
Also keep in mind that the industry standard is 50 which is the capacity to take over for the real standard of 25 in a fail-over scenario. So you should be happy you even get 300cps for free.
The sofia stack can be improved but we are not the creators of this sip stack. There is little to no work being done on that project right now and we are happy with what we have until we can get the lead dev to work on improving it with us when he has the time.
On Sat, Jan 30, 2010 at 5:33 AM, Puskás Zsolt <errotan@gmail.com> wrote: 2010. január 29. 16.31.29 Robin Vleij dátummal ezt írta:
Hi guys,
Doing a bit of testing / benchmarking with FS 1.04 (and 1.05 SVN). Found out some interesting things, I think.
The setup is like this:
SIPP Client -> FS -> SIPP Server
The dialplan is as simple as it gets:
<extension name="test-out"> <condition field="network_addr" expression="^130\.244\.190\.150$"/> <condition field="destination_number" expression="^(\d+)$"> <action application="set" data="proxy_media=true"/> <action application="set" data="hangup_after_bridge=true"/> <action application="bridge" data="{sip_contact_user=transit}sofia/gateway/${distributor(gwg1)}/$1|sofia /gateway/${distributor(gwg2)}/$1"/> </condition> </extension>
For the rest it's running CSV cdr's, commented out all modules I'm not using, etc etc all that I could find on the wiki and the Interwebs. Hardware / OS: from the shelve quadcore Xeon, debian 64-bit, 12GB memory. SIPP is running a 500ms RTP pcap and the other side echos back.
I had a few test setups then:
1: FS SVN, 1 sofia profile where the gateways were configured and the server_IP:5060 was used.
2. FS SVN, 2 sofia profiles where the gateways where in an seperate profile (server_IP:5070) and the "customer facing" side was the original profile.
3. FS 1.04, same as above
4. FS 1.04, 4 sofia profiles, distributor to spread load over 2 incoming and 2 outgoing profiles.
Now the interesting thing was that under 1 I could go up, almost without any CPU load, to 50cps. As soon as I went over this, calls where handled slower and "ongoing" calls would pile up untill it became really slow. CPU load went to 100% on the FS process (both user and system time). Lots of interupts and context switches. No throughput anymore untill I lower and wait till the "buffer" is empty and FS is keeping up again.
Under 2, I was able to increase the CPS to about 100 with the same effect.
3 then went much better, I was able to increase CPS to about 200 cps and response times in SIPP went up slighty untill it just hits some kind of limit and calls are handled slower.
4 is pretty cool. Here I can run 2 sipp clients both doing 150cps to the gateways that are spread on the distribution module, so I spread traffic over 2 profiles. With 300 cps in total, FS is keeping up and I have 30% idle CPU. However, increasing to over 300cps gives problems again, even though I have idle CPU left!
All in all, I have a feeling that a single sip profile can't run more than a certain limit untill it gets into some problem. Depending on if I'm running SVN or 1.04 that limit seems to be 50cps or a bit higher. After that limit it starts piling up "ongoing" calls, by taking time to handle them and when that limit gets too high it's too late. All in all really fine, I just set the system wide limit to a little under that "threshold". But when I'm running just UNDER the threshold it's not CPU that's a problem. Theoretically I should be able to run (based on the CPU usage at 300cps) about 400cps.
When running at 300 I get SOME failed calls and I see
"switch_core_state_machine.c:525 a9a60636-0cea-11df-85a1-09c991f2afc5 Timeout waiting for next instruction in CS_NEW!"
in the console.
I didn't find much on how people do high cps setups and it feels a bit like a "friday afternoon solution" to run multiple sofia profiles on the same machine in order to max out the system.
Maybe I'm missing something and I know it's not an exact science this, but I'm not sure "all is OK" because I'm not slowly getting to a 100% cpu (or disk / network) usage, I hit some kind of limit after which stuff goes wrong.
Anyone any input!
/Robin
On 1/31/10 2:57 AM, Michael Jerris wrote: Hi Mike!
For clarification, is it correct that your getting worse numbers for sustainable cps on SVN then on 1.0.4? I would be interested in the
Yes, 1.04 offered a lot better performance in my test scenario, using the same call scripts. I agree that the scenario isn't really realistic, I should test with at least 30 second calls.
numbers you would get with bypass_media=true instead of proxy_media=true
Thought about the same, and funnily enough that didn't help. But in the long list of things I tried I don't remember if I tried it on 1.04, without media proxying. I expect a much higher performance just bypassing the media. I'll try again tomorrow on 1.04 and post the results.
and with neither setting set as well. Also, make sure your logging level is low and try putting the db dir on a ram disk. Thanks for the info. As
I check disk performance using iostat and they're on 2% usage. It's a 10k rpm sas disk we're writing to, with ram cache on the raid controller. Also -nosql didn't have any influence at all on performance.
that length of call matters very little. In proxy media or normal mode, the performance of the box is much more of a calculation on number of calls than cps as a result of the context switching from having to move
Exactly. I'll adjust the lengt of the call to 30 or 60 seconds and then try to see how many calls we can have. That's in the end what matters really and not cps on it's own. Have to create a 60 second pcap first. :)
calls). Some other tips. While this extension may be trivial, what else is there in your dialplan context? Anything above that extension could
Nothing, it's a single entry dialplan and that entry is thus on the top.
cause a significant impact. Do you have any of the presence features
All presence features disabled.
enabled? These do significantly impact call handling performance even if your sipp scenarios do not send any of those packets.
Yep. :) Thanks for the input so far! Maybe after I get some final results I put it on the wiki for future reference to load tests? /robin
On 1/30/10 5:57 PM, Anthony Minessale wrote: Hi Anthony,
Also keep in mind that the industry standard is 50 which is the capacity to take over for the real standard of 25 in a fail-over scenario. So you should be happy you even get 300cps for free.
Yeah, no discussion there. I think 300 on standard hardware seems really good. Especially compared to some commercial product we've seen / read about. :) It was just that it doesn't feel like the hardware is fully used, but some invisible wall we hit, ie bug maybe. That's why I'm asking if anyone has seen this. Whatever limit we'll have, we'll set in the system wide limits and we're done, so it's no show-stopper in putting it in production. I've also done a long term test with a few thousand ongoing calls at 50cps over the weekend, see how it goes with memory usage and such.
and we are happy with what we have until we can get the lead dev to work on improving it with us when he has the time.
OK. /robin
On 1/30/10 12:33 PM, Puskás Zsolt wrote: Hi!
CPU usage is not the only thing that limit your calls. Have you set the recommended ulimit settings and / or started fs with the -waste option ?
Yes, I did read the wiki & docs and do have the right setup for max performance. The -waste option didn't seem to do much in the 1.05 setup I was testing in, but I can try again on 1.04 tomorrow when I continue testing. /Robin
participants (4)
-
Anthony Minessale
-
Michael Jerris
-
Puskás Zsolt
-
Robin Vleij