FS beats Aculab Prosody S on subjective test on lay users for conference quality
Hello freeswitchers! I thought you're be pleased to know: FS beated Prosody S on subjective test on lay users. After 3 users testing the two conferences and reporting far better quality on FS, we decided to make a more rigorous test. Then we made a blind test by asking 4 lay users to dial two urls for 10 minutes from their x-lite and report the results. These users are non-techie and never heard about FS at all. The first url was Aculab Prosody S test app, then, a second call to FS on default conf (extension 3000). All users preferred FS. Some users were almost enthusiastic after entering FS conf. Some even started to blame ProsodyS after hearing FS. The average report told 50% more satisfaction on FS. Questions on report were simple ones, with results from 1(worse) to 5(best) : 1-Overall opinion(1-crap; 5-awesome); 2-Delay (1-more delay, 5-no delay at all); 3-Audio quality (1-noisy, choppy, etc; 5-cristal clear); Machines used: Prosody S: Windows Vista Business, Intel Core 2 Duo E7400 @ 2.8 GHz, 4Gb de RAM FreeSwitch: CentOS 5.4, Intel(R) Pentium(R) 4 CPU 2.40GHz, 1Gb RAM Congratulations! You made a great product! -- Fernando Gregianin Testa Voice Technology Ltda +55 11 35882166
This is great news. Before we start sending out commercials that say "4 out of 5 Brazilians prefer FreeSWITCH to Prosody" I would very much like know more about the tests: What kind of computers were the end users working? What codecs were being used by the x-lite clients when connecting to FS? What codecs were being used by the x-lite clients when connecting to Prosody? Were there any other factors that could possibly have affected the call quality? The reason I ask all of these is that I would love to put this account into the testimonials page on the wiki and make it available to other techie guys who are trying to make their business cases for using FreeSWITCH (or FOSS in general) to the corporate executives. Thanks for the great report! -Michael On Wed, Jul 29, 2009 at 7:53 AM, Fernando Testa < testa@voicetechnology.com.br> wrote:
Hello freeswitchers! I thought you're be pleased to know: FS beated Prosody S on subjective test on lay users. After 3 users testing the two conferences and reporting far better quality on FS, we decided to make a more rigorous test. Then we made a blind test by asking 4 lay users to dial two urls for 10 minutes from their x-lite and report the results. These users are non-techie and never heard about FS at all. The first url was Aculab Prosody S test app, then, a second call to FS on default conf (extension 3000). All users preferred FS. Some users were almost enthusiastic after entering FS conf. Some even started to blame ProsodyS after hearing FS. The average report told 50% more satisfaction on FS. Questions on report were simple ones, with results from 1(worse) to 5(best) : 1-Overall opinion(1-crap; 5-awesome); 2-Delay (1-more delay, 5-no delay at all); 3-Audio quality (1-noisy, choppy, etc; 5-cristal clear);
Machines used: Prosody S: Windows Vista Business, Intel Core 2 Duo E7400 @ 2.8 GHz, 4Gb de RAM FreeSwitch: CentOS 5.4, Intel(R) Pentium(R) 4 CPU 2.40GHz, 1Gb RAM
Congratulations! You made a great product! -- Fernando Gregianin Testa Voice Technology Ltda +55 11 35882166
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Fernando, Greetings from Rio..! It'd be interesting to understand more about these results - roughly speaking, two conferencing systems using the same codecs, etc., ought to perform pretty much identically, particularly with just a few callers. I'd be interested to see a network packet dump of a conference on each of the machines, if you're able to make one available. Cheers -- Dave
Hello freeswitchers!
I thought you're be pleased to know: FS beated Prosody S on subjective test on lay users. After 3 users testing the two conferences and reporting far better quality on FS, we decided to make a more rigorous test. Then we made a blind test by asking 4 lay users to dial two urls for 10 minutes from their x-lite and report the results. These users are non-techie and never heard about FS at all. The first url was Aculab Prosody S test app, then, a second call to FS on default conf (extension 3000). All users preferred FS. Some users were almost enthusiastic after entering FS conf. Some even started to blame ProsodyS after hearing FS. The average report told 50% more satisfaction on FS. Questions on report were simple ones, with results from 1(worse) to 5(best) : 1-Overall opinion(1-crap; 5-awesome); 2-Delay (1-more delay, 5-no delay at all); 3-Audio quality (1-noisy, choppy, etc; 5-cristal clear);
Machines used: Prosody S: Windows Vista Business, Intel Core 2 Duo E7400 @ 2.8 GHz, 4Gb de RAM FreeSwitch: CentOS 5.4, Intel(R) Pentium(R) 4 CPU 2.40GHz, 1Gb RAM
Congratulations! You made a great product! -- Fernando Gregianin Testa Voice Technology Ltda +55 11 35882166
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- David Knell, Director, 3C Limited T: +44 20 3298 2000 E: dave@3c.co.uk W: http://www.3c.co.uk
David Knell wrote:
Hi Fernando,
Greetings from Rio..!
It'd be interesting to understand more about these results - roughly speaking, two conferencing systems using the same codecs, etc., ought to perform pretty much identically, particularly with just a few callers.
I'd be interested to see a network packet dump of a conference on each of the machines, if you're able to make one available.
High quality conferencing is a difficult task, and still a research topic. No two conferencing systems perform alike. The interesting thing about this and other reports is that the conferencing in Freeswitch is not very clever right now, yet people are already saying it beats various other offerings, including long time commercial offerings. Steve
On Thu, 2009-07-30 at 09:21 +0800, Steve Underwood wrote:
High quality conferencing is a difficult task, and still a research topic. No two conferencing systems perform alike. The interesting thing about this and other reports is that the conferencing in Freeswitch is not very clever right now, yet people are already saying it beats various other offerings, including long time commercial offerings.
It may well be that a simplistic implementation (noise gate, add them all up) is all that's required for dealing with small groups or, more generally, groups of any size which have a small number of active speakers at any one time: it's predictable and unlikely to introduce unpleasant side effects. --Dave
David Knell wrote:
On Thu, 2009-07-30 at 09:21 +0800, Steve Underwood wrote:
High quality conferencing is a difficult task, and still a research topic. No two conferencing systems perform alike. The interesting thing about this and other reports is that the conferencing in Freeswitch is not very clever right now, yet people are already saying it beats various other offerings, including long time commercial offerings.
It may well be that a simplistic implementation (noise gate, add them all up) is all that's required for dealing with small groups or, more generally, groups of any size which have a small number of active speakers at any one time: it's predictable and unlikely to introduce unpleasant side effects.
This is one of those situations where when you've experienced something better you make that your baseline for acceptability. I would consider a noise gate horribly crude, and VAD as the minimum for acceptable performance. If you've only used a noise gate you get used to it. If you're not sufficiently versed in the art you may well think nothing better is even possible. The fact that even the simple scheme, with noise gating, in Freeswitch is getting high praise, is pretty damning of mature commercial products. Steve
Hi, a bit late on answering some of the questions, but, here we go.On Aculab, all codecs were G711u. The same codec we have on FS: freeswitch@conference> show channels API CALL [show(channels)] output: uuid,direction,created,created_epoch,name,state,cid_name,cid_num,ip_addr,dest,application,application_data,dialplan,context,read_codec,read_rate,write_codec,write_rate,secure c9cafeb8-803a-11de-8ceb-cb8648fd1ccf,inbound,2009-08-03 11:34:44,1249310084,sofia/internal/1000@192.168.0.40,CS_EXECUTE,Teste Testa,1000,192.168.0.165,3200,conference,3200-192.168.0.40@ultrawideband ,XML,default,L16,8000,PCMU,8000, cc5edb90-803a-11de-8ceb-cb8648fd1ccf,inbound,2009-08-03 11:34:49,1249310089,sofia/internal/1000@192.168.0.40 ,CS_EXECUTE,F.G.Testa,1000,192.168.0.249,3200,conference,3200-192.168.0.40@ultrawideband ,XML,default,L16,8000,PCMU,8000, 2 total. freeswitch@conference> conference list API CALL [conference(list)] output: Conference 3200-192.168.0.40 (2 members) 2;sofia/internal/1000@192.168.0.40 ;cc5edb90-803a-11de-8ceb-cb8648fd1ccf;F.G.Testa;1000;hear|speak;0;0;300 1;sofia/internal/1000@192.168.0.40;c9cafeb8-803a-11de-8ceb-cb8648fd1ccf;Teste Testa;1000;hear|speak|talking|floor;0;0;300 I think this answers some questions from Michael. A packet dump I don't have right now. Fernando G. Testa On Thu, Jul 30, 2009 at 11:13 AM, Steve Underwood <steveu@coppice.org>wrote:
David Knell wrote:
On Thu, 2009-07-30 at 09:21 +0800, Steve Underwood wrote:
High quality conferencing is a difficult task, and still a research topic. No two conferencing systems perform alike. The interesting thing about this and other reports is that the conferencing in Freeswitch is not very clever right now, yet people are already saying it beats various other offerings, including long time commercial offerings.
It may well be that a simplistic implementation (noise gate, add them all up) is all that's required for dealing with small groups or, more generally, groups of any size which have a small number of active speakers at any one time: it's predictable and unlikely to introduce unpleasant side effects.
This is one of those situations where when you've experienced something better you make that your baseline for acceptability. I would consider a noise gate horribly crude, and VAD as the minimum for acceptable performance. If you've only used a noise gate you get used to it. If you're not sufficiently versed in the art you may well think nothing better is even possible.
The fact that even the simple scheme, with noise gating, in Freeswitch is getting high praise, is pretty damning of mature commercial products.
Steve
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Fernando Gregianin Testa Voice Technology Ltda +55 11 35882166
participants (4)
-
David Knell
-
Fernando Testa
-
Michael Collins
-
Steve Underwood