If you're going to have that many listeners then it would be best to use something like shoutcast to broadcast the stream out to a local stream on various different boxes... then tie the callers into a stream... when they have questions uuid_transfer them into the conf.. then back to the stream when done. This would scale to very large numbers because you could split it out into 100's of boxes if needed. /b On Dec 17, 2009, at 1:29 PM, Brian wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

I didn't realize there was a policy about load testing questions. What forum should I have used for this? I didn't get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on. Thanks, Brian. From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability. When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS? Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page. All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production......... On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote: Hi Mike, I didn't get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version. However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems. I've read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test - more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I'm doing wrong, but I don't see what it could be. Brian. From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability I would be curious what the same tests produce with svn trunk of FreeSWITCH. Mike On Dec 16, 2009, at 4:49 PM, Brian wrote: Hi, I'm new to FreeSWITCH and I'm testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn't scale nearly as well as I'd hoped (based on what I've read on how FreeSWITCH is supposed to be generally very scalable). Here's my server setup is this: FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I've set file logging to "notice" level. My conference profile is configured to suppress several events, hoping that it would improve performance. Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences: Scenario 1: 1 conference, 1 speaker, audio failed at approx 300 listeners (mute) Scenario 2: 4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system). Scenario 3: 16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system). Looking at the output from "top", it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%. I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case? Thanks, Brian. _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners. Brian. From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 3:49 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability We didn't post it anywhere but we just get overwhelmed with them and many of them are unfounded and take up a lot of time to track down. That does not mean you have not found a real problem but the first step is trying trunk. On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> wrote: I didn't realize there was a policy about load testing questions. What forum should I have used for this? I didn't get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on. Thanks, Brian. From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability. When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS? Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page. All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production......... On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote: Hi Mike, I didn't get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version. However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems. I've read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test - more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I'm doing wrong, but I don't see what it could be. Brian. From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability I would be curious what the same tests produce with svn trunk of FreeSWITCH. Mike On Dec 16, 2009, at 4:49 PM, Brian wrote: Hi, I'm new to FreeSWITCH and I'm testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn't scale nearly as well as I'd hoped (based on what I've read on how FreeSWITCH is supposed to be generally very scalable). Here's my server setup is this: FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I've set file logging to "notice" level. My conference profile is configured to suppress several events, hoping that it would improve performance. Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences: Scenario 1: 1 conference, 1 speaker, audio failed at approx 300 listeners (mute) Scenario 2: 4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system). Scenario 3: 16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system). Looking at the output from "top", it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%. I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case? Thanks, Brian. _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

I've got FS running on a 64 bit OS, and here is more info on the test procedure. I've got one server (primary) that hosts the speaker call (this is meant to be a primary conference with a few speakers, but my test simplifies this to just one speaker). I've got a second server (secondary) that hosts the conference that all the listeners go into, and I have two other servers that I use automate the listener calls. The goal is to have several secondary servers to scale the listener side of things, but for this initial test I've only got one secondary server. The primary server dials into the secondary conference server so that the listeners can hear the speaker conference on the primary server. The automated listener servers start dialing into the listener conference at a combined rate of 5 calls per second (i.e. 2.5 calls per second each). The play an audio loop that represents noise on their end, which since they are listeners, should be ignored anyway. As I ramp up the automated listener calls, I manually call into the conference from either my SIP phone, or from a land line using a DID that I have directed to the conference. All calls are using SIP with uLaw 8000hz codec. Also, I've set up the profile for the listener conference to disable many of the events: <profile name="listener"> <param name="domain" value="$${domain}"/> <param name="rate" value="8000"/> <param name="moh-sound" value="moh.wav"/> <param name="suppress-events" value="start-talking,stop-talking,energy-level,volume-level,gain-level,mute- detect,energy-level-member,volume-in-member,volume-out-member,lock,unlock,fl oor-change"/> <param name="caller-controls" value="listener_controls"/> </profile> I do have caller controls for the listener, since in my production I will need to generate and handle events for listener DTMF. To compare FreeSWITCH vs Asterisk, I just swap out the secondary conference server and everything else stays the same. Brian. From: Brian West [mailto:brian@freeswitch.org] Sent: Thursday, December 17, 2009 5:20 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability What exactly are you doing I know it goes better than that.. are you using 64bit? / b On Dec 17, 2009, at 3:41 PM, Brian wrote: I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners. Brian.

What exactly is your test process? you should try increasing the interval in the conference profile to a bigger time slice maybe 30 40 or 60ms you could also increase the ptime to match as well. like brian said you could use mod_shout to broadcast the single speaker to icecast and let people listen with itunes/winamp On Thu, Dec 17, 2009 at 3:41 PM, Brian <brian@proximosystems.com> wrote:

I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners.

Brian.

*From:* Anthony Minessale [mailto:anthony.minessale@gmail.com] *Sent:* Thursday, December 17, 2009 3:49 PM

*To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_conference scalability

We didn't post it anywhere but we just get overwhelmed with them and many of them are unfounded and take up a lot of time to track down. That does not mean you have not found a real problem but the first step is trying trunk.

On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> wrote:

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

*From:* Anthony Minessale [mailto:anthony.minessale@gmail.com] *Sent:* Thursday, December 17, 2009 2:42 PM

*To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

*From:* Michael Jerris [mailto:mike@jerris.com] *Sent:* Thursday, December 17, 2009 10:18 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

Hearing that Asterisk (1.4) scales 2x like FS is not common, sounds like a configuration error. If not, I already see the title of the next Digium blog entry: "FreeSwitch scalability myth finally ends: The worst Asterisk version ever (1.4) beating the crap of the best and latest FS." Anyway, you should compare FS trunk to Asterisk 1.6.2 to see who wins the final conference battle! :-) François. On Thu, 2009-12-17 at 16:41 -0500, Brian wrote:

I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners.

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 3:49 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

We didn't post it anywhere but we just get overwhelmed with them and many of them are unfounded and take up a lot of time to track down. That does not mean you have not found a real problem but the first step is trying trunk.

On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> wrote:

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM

To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

It was of course just bad humor, I love both projects for what they are, and I agree that both have their own advantages and inconvenients. For example, accessing that same conference from a dahdi card could be another goal where Asterisk would be at an advantage, as chan_dahdi is still superior (in the more tested sense) than openzap+mod_openzap. I just use both projects separately or together depending on what's needed! I'm no banker nor do I understand the code, but many thanks for all those unpaid contributions providing an excellent alternative for free telephony. Your names really deserve being engraved in google's cache for eternity. :-) But still, I would like to see those numbers... François. On Fri, 2009-12-18 at 10:34 -0600, Anthony Minessale wrote:

Conferencing is hardly the best place to judge performance. Quality is a far more important goal to me in conferencing.

Lets compare who can do 48khz conferences with several 32k siren callers on a polycom 6000, several more using G722 at 16khz and another handful of people on g711 ulaw all at different rates and ptimes talking in near-real time with low delay and low echo. The fact that you can broadcast the conferences to icecast, control it from an external application and play files etc, and oh yeah, it can stream video.

Frankly, considering this is a free software project and so many people benefit, i would rather focus on quality than what numbers i can get from having robots call the conference in some way that probably does not match reality. I would love for someone to sponsor the effort to add features to the conference module, but of course, I do not hold my breath, instead I continue to improve it for free when I find time. This is one of many reasons I do not enjoy performance discussions unless I am talking to an engineer who understands the code or a banker ready to pay for improvements. That is not my way of saying pay me or forget it as you can clearly see the conference module has made it to where it is today with no financial support at all. Just the efforts of myself and several brave volunteers over the years who have contributed to it.

BTW,

We have a weekly call, there is one today in 30 minutes. Drop by sip:888@conference.freeswitch.org This is just an openVZ instance mind you running at 48khz waiting for anyone to call in and say hi.

On Fri, Dec 18, 2009 at 10:12 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: Hearing that Asterisk (1.4) scales 2x like FS is not common, sounds like a configuration error.

If not, I already see the title of the next Digium blog entry: "FreeSwitch scalability myth finally ends: The worst Asterisk version ever (1.4) beating the crap of the best and latest FS."

Anyway, you should compare FS trunk to Asterisk 1.6.2 to see who wins the final conference battle! :-)

François.

On Thu, 2009-12-17 at 16:41 -0500, Brian wrote: > I did a test with the trunk version for the one conference case, and > it is the same results as for 1.0.4. The audio failed at around 300 > listeners. Oddly though, it consumed less %CPU (240% instead of 300%), > and yet the audio still failed at the same number of listeners. > > > > Brian. > > > > From: Anthony Minessale [mailto:anthony.minessale@gmail.com] > Sent: Thursday, December 17, 2009 3:49 PM > To: freeswitch-users@lists.freeswitch.org > Subject: Re: [Freeswitch-users] mod_conference scalability > > > > > We didn't post it anywhere but we just get overwhelmed with them and > many of them are unfounded and take up a lot of time to track down. > That does not mean you have not found a real problem but the first > step is trying trunk. > > > > > On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> > wrote: > > I didn’t realize there was a policy about load testing questions. What > forum should I have used for this? > > > > I didn’t get the chance to test on FS trunk yet, but when I do I will > provide you with the feedback when I do. Just let me know what forum > to use for this topic from now on. > > > > Thanks, > > > > Brian. > > > > From: Anthony Minessale [mailto:anthony.minessale@gmail.com] > Sent: Thursday, December 17, 2009 2:42 PM > > > To: freeswitch-users@lists.freeswitch.org > Subject: Re: [Freeswitch-users] mod_conference scalability > > > > > One man's stable release is another man's 6 month old release with > hundreds of known fixed bugs. > If one of the core developers tells you to try it, you may as well > take the time to try it now that you have opened a forum questioning > the scalability. > > When you tested asterisk did you actually use 600 phones and verify > that each one can hear the audio perfectly and in time with what the > speaker was saying? Did you try same on FS? > > Did you optimize your dialplan on FS to deal with a load test or > follow any of the recommended performance tuning page. > > All of the answers to these questions are really moot because we have > a policy against entertaining load testing questions but if you like > asterisk, by all means, use it, and good luck to you if those numbers > you are testing at are what you plan to put in real > production......... > > On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> > wrote: > > Hi Mike, > > > > I didn’t get around to testing on the FreeSWITCH trunk yet. Are there > substantial fixes to mod_conference in the FreeSWITCH trunk that might > increase capacity for my scenario of one speaker and many listeners? > If I want to put this into a production environment, I would need a > stable version, which as far as I know is the 1.0.4 version. > > > > However, I did test on Asterisk 1.4 using app_conference, and doing > the same scenario was able to get 1 speaker and 600 listeners on a > single conference with no audio issues. The CPU at that point was just > over 300%, same as where the single conference scenario failed on > FreeSWITCH with 300 listeners. I was able to push it to over 700 > listeners before I reached 400% CPU usage (I guess maxing out my > quad-core processors), and asterisk finally crashed. But up until that > point, there were no audio problems. > > > > I’ve read a lot about how FreeSWITCH is supposed to be more scalable > than Asterisk, but unless there is something wrong with my FreeSWITCH > setup, Asterisk was clearly the winner in this test – more than > doubling FreeSWITCH capacity in this case. Again, maybe there is > something on the FreeSWITCH side that I’m doing wrong, but I don’t see > what it could be. > > > > Brian. > > > > > > From: Michael Jerris [mailto:mike@jerris.com] > Sent: Thursday, December 17, 2009 10:18 AM > To: freeswitch-users@lists.freeswitch.org > Subject: Re: [Freeswitch-users] mod_conference scalability > > > > > I would be curious what the same tests produce with svn trunk of > FreeSWITCH. > > > > > Mike > > > > > On Dec 16, 2009, at 4:49 PM, Brian wrote: > > > > > Hi, > > > > > > I’m new to FreeSWITCH and I’m testing the scalability of > mod_conference to see if it will scale better that other solutions. My > scenario is to have one speaker, and many listeners (mute). Since I > have only one speaker, I was expecting this to scale well because > there is no audio mixing required, just send each frame of the single > speaker to each listener. Unfortunately, my testing was disappointing, > and it didn’t scale nearly as well as I’d hoped (based on what I’ve > read on how FreeSWITCH is supposed to be generally very scalable). > > > > > > Here’s my server setup is this: > > > > > > FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig > of RAM. I’ve set file logging to “notice” level. My conference profile > is configured to suppress several events, hoping that it would improve > performance. > > > > > > Here are a few scenarios I tested, and roughly where I reached the > point of audio failure on the conferences: > > > > > > Scenario 1: > > > 1 conference, 1 speaker, audio failed at approx 300 listeners (mute) > > > > > > Scenario 2: > > > 4 conferences, 1 speaker per conference, audio failed approx 110 > listeners per conference (so just over 400 total channels on the > system). > > > > > > Scenario 3: > > > 16 conferences, 1 speaker per conference, audio failed at 32 listeners > per conference (so just over 500 total channels on the system). > > > > > > > > > Looking at the output from “top”, it seems that in all 3 scenarios, > the audio quality failed when the % CPU for the FreeSWITCH process > exceeded 300%. > > > > > > I was hoping maybe someone else might have done similar testing, or > maybe has suggestions on how to improve the performance. Or perhaps an > alternate solution to the one speaker, many listener case? > > > > > > Thanks, > > > > > > Brian. > > > > > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > Twitter: http://twitter.com/FreeSWITCH_wire > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > iax:guest@conference.freeswitch.org/888 > googletalk:conf+888@conference.freeswitch.org > pstn:+19193869900 > > > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > Twitter: http://twitter.com/FreeSWITCH_wire > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > iax:guest@conference.freeswitch.org/888 > googletalk:conf+888@conference.freeswitch.org > pstn:+19193869900 > > > _______________________________________________ > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

I was evaluating the technologies available, and I thought you would be interested in my results. However, almost every other reply I get from you to my posts, rather than being helpful, has been hostile and insulting. My scenario is not a hypothetical one of “having robots call the conference in a way that probably does not match reality”. In fact, this will very much reflect the reality of the application I’m building. Only instead of 300 listeners, I need to scale to over 2000 listeners minimum – per event, with possibly more than one concurrent event. I want to pack as many listeners on one server as I can. I’m trying to find a real solution to a real problem. I work with other open source projects and fund enhancements or fixes I need. FreeSWITCH would be no different. Brian. From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Friday, December 18, 2009 11:34 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability Conferencing is hardly the best place to judge performance. Quality is a far more important goal to me in conferencing. Lets compare who can do 48khz conferences with several 32k siren callers on a polycom 6000, several more using G722 at 16khz and another handful of people on g711 ulaw all at different rates and ptimes talking in near-real time with low delay and low echo. The fact that you can broadcast the conferences to icecast, control it from an external application and play files etc, and oh yeah, it can stream video. Frankly, considering this is a free software project and so many people benefit, i would rather focus on quality than what numbers i can get from having robots call the conference in some way that probably does not match reality. I would love for someone to sponsor the effort to add features to the conference module, but of course, I do not hold my breath, instead I continue to improve it for free when I find time. This is one of many reasons I do not enjoy performance discussions unless I am talking to an engineer who understands the code or a banker ready to pay for improvements. That is not my way of saying pay me or forget it as you can clearly see the conference module has made it to where it is today with no financial support at all. Just the efforts of myself and several brave volunteers over the years who have contributed to it. BTW, We have a weekly call, there is one today in 30 minutes. Drop by sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> This is just an openVZ instance mind you running at 48khz waiting for anyone to call in and say hi. On Fri, Dec 18, 2009 at 10:12 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: Hearing that Asterisk (1.4) scales 2x like FS is not common, sounds like a configuration error. If not, I already see the title of the next Digium blog entry: "FreeSwitch scalability myth finally ends: The worst Asterisk version ever (1.4) beating the crap of the best and latest FS." Anyway, you should compare FS trunk to Asterisk 1.6.2 to see who wins the final conference battle! :-) François. On Thu, 2009-12-17 at 16:41 -0500, Brian wrote:

I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners.

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 3:49 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

We didn't post it anywhere but we just get overwhelmed with them and many of them are unfounded and take up a lot of time to track down. That does not mean you have not found a real problem but the first step is trying trunk.

On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> wrote:

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM

To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

Brian, there was not one insulting word in anything I have said and as this is a community mailing list my replies are always voiced to address the public in general not you specifically, like I already mentioned in my last post. If you open a public forum on a FAQ be prepared to hear our policy. Indeed many people do unrealistic load testing and most people with strong will find it insulting when a group of people have a set of standard policy by which they try to deal with making a penny jar for all the 2 cents worth of input we get on a daily basis. I can't begin to iterate over all the cases we endure on a weekly basis. additionally 90% of bug reports are on older releases and we always make people reproduce their issues on SVN trunk because 3 core devs and a handful of helpers can't maintain 20 versions of the code. I gave you some really suggestions yesterday let me repaste it, I fail to see any insults: --------------------------------------------------------------------------------------------------------------------------------------------------- What exactly is your test process? you should try increasing the interval in the conference profile to a bigger time slice maybe 30 40 or 60ms you could also increase the ptime to match as well. like brian said you could use mod_shout to broadcast the single speaker to icecast and let people listen with itunes/winamp --------------------------------------------------------------------------------------------------------------------------------------------------- I have to get in these "fights" with people constantly so I guess that is part of my job and my biggest mistake is spending so much time trying to explain myself. - Show quoted text - On Fri, Dec 18, 2009 at 1:33 PM, Michael Collins <msc@freeswitch.org> wrote:

On Fri, Dec 18, 2009 at 11:14 AM, Brian <brian@proximosystems.com> wrote:

...

I was evaluating the technologies available, and I thought you would be interested in my results. However, almost every other reply I get from you to my posts, rather than being helpful, has been hostile and insulting.

Thanks for your input. Just so you know, Tony deals with people on a near daily basis who want to spend time doing crazy schemes under the guise of "load testing" or "researching a new solution" which are not grounded in reality. At first blush this scenario sounded like one of those schemes. However it definitely looks like you've built a test scenario that mimics reality better than most. I think we can give you a pass for not being able to get 500 people all at once to call in every time you need to test. :)

...

My scenario is not a hypothetical one of “having robots call the conference in a way that probably does not match reality”. In fact, this will very much reflect the reality of the application I’m building. Only instead of 300 listeners, I need to scale to over 2000 listeners minimum – per event, with possibly more than one concurrent event. I want to pack as many listeners on one server as I can. I’m trying to find a real solution to a real problem.

That kind of volume suggests that the icecast style solution would be best. It takes much less resources to send audio in one direction than it does to mix audio from multiple parties. I like bkw's initial suggestion of transferring a caller to the conference only when he/she needs to speak, such as to ask a question. Like Tony mentioned, his focus is on quality not quantity, so mod_conference probably isn't the best tool for this scenario.

...

I work with other open source projects and fund enhancements or fixes I need. FreeSWITCH would be no different.

Excellent! It looks like we don't already have a canned solution, obviously, but as bkw likes to say, all the Lego bricks are there to build the solution. Hop on IRC (#freeswitch in irc.freenode.net) or join the weekly conference which is going on right now and you might catch some of the devs and leading community members and you can chat in real-time about your challenges. (http://wiki.freeswitch.org/wiki/FS_weekly_2009_12_14)

-Michael

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

I am more than sure there is probably plenty of room for conference optimizations it's just a big task. We don't have a test labbed up and an urgency to work on it. If you really want us to pursue trying to improve the performance perhaps you can contact us at consulting@freeswitch.org and provide us with access your test environment and let us investigate the possibility of making improvements. On Fri, Dec 18, 2009 at 2:16 PM, Brian <brian@proximosystems.com> wrote:

Hi Michael,

Thanks for the invite, but I can’t make it on the call. Anyway, I’m not sure if discussing my specific case is meant for that type of call, is it?

After Brian’s suggestion to use shoutcast and local streams, I was looking at the code for those modules. I’m not familiar with shoutcast or icecast capabilities, so I don’t know if they can just pass though my audio stream unchanged (as uLaw packets). I want to avoid converting from uLaw to mp3 on the source server, and then back from mp3 to uLaw (or whatever phone codec) on the other server.

I was wondering if maybe there was a way to make a stream out of an existing channel, and have all the other channels just listen to that stream. It would be sort of halfway between conference and shoutcast. I would call in to the secondary server like I already do, but only instead of entering into a conference as a speaker, the channel would just start producing a local audio stream for the listener channels to tap into. It would avoid the need to have another piece of software to manage (shoutcast or icecast), and my support team would be happier...

However, I would still need to do tests for the streaming idea to see how that scales...

Brian.

*From:* Michael Collins [mailto:msc@freeswitch.org] *Sent:* Friday, December 18, 2009 2:33 PM

*To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_conference scalability

On Fri, Dec 18, 2009 at 11:14 AM, Brian <brian@proximosystems.com> wrote:

I was evaluating the technologies available, and I thought you would be interested in my results. However, almost every other reply I get from you to my posts, rather than being helpful, has been hostile and insulting.

Thanks for your input. Just so you know, Tony deals with people on a near daily basis who want to spend time doing crazy schemes under the guise of "load testing" or "researching a new solution" which are not grounded in reality. At first blush this scenario sounded like one of those schemes. However it definitely looks like you've built a test scenario that mimics reality better than most. I think we can give you a pass for not being able to get 500 people all at once to call in every time you need to test. :)

My scenario is not a hypothetical one of “having robots call the conference in a way that probably does not match reality”. In fact, this will very much reflect the reality of the application I’m building. Only instead of 300 listeners, I need to scale to over 2000 listeners minimum – per event, with possibly more than one concurrent event. I want to pack as many listeners on one server as I can. I’m trying to find a real solution to a real problem.

That kind of volume suggests that the icecast style solution would be best. It takes much less resources to send audio in one direction than it does to mix audio from multiple parties. I like bkw's initial suggestion of transferring a caller to the conference only when he/she needs to speak, such as to ask a question. Like Tony mentioned, his focus is on quality not quantity, so mod_conference probably isn't the best tool for this scenario.

I work with other open source projects and fund enhancements or fixes I need. FreeSWITCH would be no different.

Excellent! It looks like we don't already have a canned solution, obviously, but as bkw likes to say, all the Lego bricks are there to build the solution. Hop on IRC (#freeswitch in irc.freenode.net) or join the weekly conference which is going on right now and you might catch some of the devs and leading community members and you can chat in real-time about your challenges. (http://wiki.freeswitch.org/wiki/FS_weekly_2009_12_14)

-Michael

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900

Brian, Now that you know the scale freeswotch scales to in you scenario, and having designed a mult-server solution can you not add more server to scale further? As freeswitch continues to improve retest and revise your architecture design. Sent from my iPhone On Dec 18, 2009, at 11:14 AM, Brian <brian@proximosystems.com> wrote:

I was evaluating the technologies available, and I thought you would be interested in my results. However, almost every other reply I get from you to my posts, rather than being helpful, has been hostile and insulting.

My scenario is not a hypothetical one of “having robots call the con ference in a way that probably does not match reality”. In fact, thi s will very much reflect the reality of the application I’m building . Only instead of 300 listeners, I need to scale to over 2000 listen ers minimum – per event, with possibly more than one concurrent even t. I want to pack as many listeners on one server as I can. I’m tryi ng to find a real solution to a real problem.

I work with other open source projects and fund enhancements or fixes I need. FreeSWITCH would be no different.

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Friday, December 18, 2009 11:34 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

Conferencing is hardly the best place to judge performance. Quality is a far more important goal to me in conferencing.

Lets compare who can do 48khz conferences with several 32k siren callers on a polycom 6000, several more using G722 at 16khz and another handful of people on g711 ulaw all at different rates and ptimes talking in near-real time with low delay and low echo. The fact that you can broadcast the conferences to icecast, control it from an external application and play files etc, and oh yeah, it can stream video.

Frankly, considering this is a free software project and so many people benefit, i would rather focus on quality than what numbers i can get from having robots call the conference in some way that probably does not match reality. I would love for someone to sponsor the effort to add features to the conference module, but of course, I do not hold my breath, instead I continue to improve it for free when I find time. This is one of many reasons I do not enjoy performance discussions unless I am talking to an engineer who understands the code or a banker ready to pay for improvements. That is not my way of saying pay me or forget it as you can clearly see the conference module has made it to where it is today with no financial support at all. Just the efforts of myself and several brave volunteers over the years who have contributed to it.

BTW,

We have a weekly call, there is one today in 30 minutes. Drop by sip:888@conference.freeswitch.org This is just an openVZ instance mind you running at 48khz waiting for anyone to call in and say hi.

On Fri, Dec 18, 2009 at 10:12 AM, François Delawarde <fdelawarde@wirelessmundi.co m> wrote:

Hearing that Asterisk (1.4) scales 2x like FS is not common, sounds like a configuration error.

If not, I already see the title of the next Digium blog entry: "FreeSwitch scalability myth finally ends: The worst Asterisk version ever (1.4) beating the crap of the best and latest FS."

Anyway, you should compare FS trunk to Asterisk 1.6.2 to see who wins the final conference battle! :-)

François.

On Thu, 2009-12-17 at 16:41 -0500, Brian wrote:

...

I did a test with the trunk version for the one conference case, and it is the same results as for 1.0.4. The audio failed at around 300 listeners. Oddly though, it consumed less %CPU (240% instead of 300%), and yet the audio still failed at the same number of listeners.

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 3:49 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

We didn't post it anywhere but we just get overwhelmed with them and many of them are unfounded and take up a lot of time to track down. That does not mean you have not found a real problem but the first step is trying trunk.

On Thu, Dec 17, 2009 at 2:32 PM, Brian <brian@proximosystems.com> wrote:

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I w ill provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM

To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are th ere substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalab le than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org

Hi Brian, I imagine that one of the issues is that you're using a complex sledgehammer (mod_conference) to crack a simple nut - that of having multiple listeners listening to a single speaker. As far as I am aware, FreeSWITCH doesn't have anything built in which will allow this kind of simple audio path switching - maybe someone more knowledgeable than me will correct me if I'm wrong? I presented some stuff at ClueCon which would address this kind of simple application and ought to scale well beyond what you've seen with FS or Asterisk. It's still pretty basic [I'd do more with it if I wasn't so busy joshing with the other Brian on Facebook], and has never been deployed in anger but, if you're interested, drop me a note off-list. --Dave

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Hi, Came across this thread wile having the exact same issue, do you mind sharing some info about your solution? Thanks On Thu, Dec 17, 2009 at 4:06 PM, David Knell <dave@3c.co.uk> wrote:

Hi Brian,

I imagine that one of the issues is that you're using a complex sledgehammer (mod_conference) to crack a simple nut - that of having multiple listeners listening to a single speaker.

As far as I am aware, FreeSWITCH doesn't have anything built in which will allow this kind of simple audio path switching - maybe someone more knowledgeable than me will correct me if I'm wrong?

I presented some stuff at ClueCon which would address this kind of simple application and ought to scale well beyond what you've seen with FS or Asterisk. It's still pretty basic [I'd do more with it if I wasn't so busy joshing with the other Brian on Facebook], and has never been deployed in anger but, if you're interested, drop me a note off-list.

--Dave

...

I didn’t realize there was a policy about load testing questions. What forum should I have used for this?

I didn’t get the chance to test on FS trunk yet, but when I do I will provide you with the feedback when I do. Just let me know what forum to use for this topic from now on.

Thanks,

Brian.

From: Anthony Minessale [mailto:anthony.minessale@gmail.com] Sent: Thursday, December 17, 2009 2:42 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

One man's stable release is another man's 6 month old release with hundreds of known fixed bugs. If one of the core developers tells you to try it, you may as well take the time to try it now that you have opened a forum questioning the scalability.

When you tested asterisk did you actually use 600 phones and verify that each one can hear the audio perfectly and in time with what the speaker was saying? Did you try same on FS?

Did you optimize your dialplan on FS to deal with a load test or follow any of the recommended performance tuning page.

All of the answers to these questions are really moot because we have a policy against entertaining load testing questions but if you like asterisk, by all means, use it, and good luck to you if those numbers you are testing at are what you plan to put in real production.........

On Thu, Dec 17, 2009 at 1:29 PM, Brian <brian@proximosystems.com> wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

From: Michael Jerris [mailto:mike@jerris.com] Sent: Thursday, December 17, 2009 10:18 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

On 12/18/2009 03:29 AM, Brian wrote:

Hi Mike,

I didn’t get around to testing on the FreeSWITCH trunk yet. Are there substantial fixes to mod_conference in the FreeSWITCH trunk that might increase capacity for my scenario of one speaker and many listeners? If I want to put this into a production environment, I would need a stable version, which as far as I know is the 1.0.4 version.

However, I did test on Asterisk 1.4 using app_conference, and doing the same scenario was able to get 1 speaker and 600 listeners on a single conference with no audio issues. The CPU at that point was just over 300%, same as where the single conference scenario failed on FreeSWITCH with 300 listeners. I was able to push it to over 700 listeners before I reached 400% CPU usage (I guess maxing out my quad-core processors), and asterisk finally crashed. But up until that point, there were no audio problems.

I’ve read a lot about how FreeSWITCH is supposed to be more scalable than Asterisk, but unless there is something wrong with my FreeSWITCH setup, Asterisk was clearly the winner in this test – more than doubling FreeSWITCH capacity in this case. Again, maybe there is something on the FreeSWITCH side that I’m doing wrong, but I don’t see what it could be.

Brian.

I don't think you have mentioned which codecs are involved. This can have a profound effect. Steve

*From:* Michael Jerris [mailto:mike@jerris.com] *Sent:* Thursday, December 17, 2009 10:18 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mod_conference scalability

I would be curious what the same tests produce with svn trunk of FreeSWITCH.

Mike

On Dec 16, 2009, at 4:49 PM, Brian wrote:

Hi,

I’m new to FreeSWITCH and I’m testing the scalability of mod_conference to see if it will scale better that other solutions. My scenario is to have one speaker, and many listeners (mute). Since I have only one speaker, I was expecting this to scale well because there is no audio mixing required, just send each frame of the single speaker to each listener. Unfortunately, my testing was disappointing, and it didn’t scale nearly as well as I’d hoped (based on what I’ve read on how FreeSWITCH is supposed to be generally very scalable).

Here’s my server setup is this:

FreeSWITCH 1.0.4, 64 bit CentOS 5.3, on a quad-core Xeon server, 4 Gig of RAM. I’ve set file logging to “notice” level. My conference profile is configured to suppress several events, hoping that it would improve performance.

Here are a few scenarios I tested, and roughly where I reached the point of audio failure on the conferences:

Scenario 1:

1 conference, 1 speaker, audio failed at approx 300 listeners (mute)

Scenario 2:

4 conferences, 1 speaker per conference, audio failed approx 110 listeners per conference (so just over 400 total channels on the system).

Scenario 3:

16 conferences, 1 speaker per conference, audio failed at 32 listeners per conference (so just over 500 total channels on the system).

Looking at the output from “top”, it seems that in all 3 scenarios, the audio quality failed when the % CPU for the FreeSWITCH process exceeded 300%.

I was hoping maybe someone else might have done similar testing, or maybe has suggestions on how to improve the performance. Or perhaps an alternate solution to the one speaker, many listener case?

Thanks,

Brian.