[Freeswitch-users] Timing issues in AWS?
Michael Picher
mpicher at gmail.com
Sat Nov 26 18:06:41 MSK 2011
I managed to get my own CentOS 5.7 box built and loaded up as an AMI (for
those interested
http://sipxecs.blogspot.com/2011/11/building-centos-57-images-for-amazon.html
).
Kernel timing is 1000 Hz but still can't get acceptable performance on a
m1.large instance for conference bridge (running sipXecs 4.4.0 latest
patch). There is of course a lot of extra stuff going on with sipXecs
beyond FS... so it may be something else sucking down CPU. Maybe I'll try
moving conf bridge onto its own virtual in a cluster and see what happens...
I am able to get acceptable (unloaded) bridge performance with a c1.xlarge.
For the price of a c1.xlarge though you may as well purchase a 1/3 of a
rack and load up your own servers.
Mike
On Thu, Nov 24, 2011 at 7:10 PM, Fraser Redmond <fraserredmond at gmail.com>wrote:
> Yeah, I don't know how to interpret them either - I've seen the tests
> recommended a few times, but never with much explanation of what to look
> for.
>
> Our users noticed an improvement, be interesting to hear if yours do too.
>
> Cheers,
> Fraser
>
>
>
>
>
> On 24 November 2011 16:08, Stephen Dame <sdame at 207me.com> wrote:
>
>> Fraser, thanks for tips in compiling kernel.****
>>
>> ** **
>>
>> I was able to set up a 1000hz ec2 instance ... and run some comparative
>> tests using the same starting 10.04 LTS ami.****
>>
>> ** **
>>
>> Here are the results of timing tests.. Not sure how to interpret…
>> Having users test audio right now in voice conferences to see if they
>> notice any difference.****
>>
>> ** **
>>
>> Looked in wiki… I’m still not clear on what to look for.****
>>
>> ** **
>>
>> Regards,****
>>
>> Stephen****
>>
>> ** **
>>
>> *100hz kernel*
>>
>> ** **
>>
>> *time_test 600 10*
>>
>> test 1 sleep 600 8847****
>>
>> test 2 sleep 600 9969****
>>
>> test 3 sleep 600 10006****
>>
>> test 4 sleep 600 9986****
>>
>> test 5 sleep 600 10003****
>>
>> test 6 sleep 600 9977****
>>
>> test 7 sleep 600 10117****
>>
>> test 8 sleep 600 9872****
>>
>> test 9 sleep 600 10014****
>>
>> test 10 sleep 600 10124****
>>
>> ** **
>>
>> *timer_test 120 10*
>>
>> 2011-11-24 14:53:51.142568 [CONSOLE] mod_commands.c:329 Timer Test: 1
>> sleep 120 120289****
>>
>> 2011-11-24 14:53:51.263610 [CONSOLE] mod_commands.c:329 Timer Test: 2
>> sleep 120 120703****
>>
>> 2011-11-24 14:53:51.382587 [CONSOLE] mod_commands.c:329 Timer Test: 3
>> sleep 120 118952****
>>
>> 2011-11-24 14:53:51.502640 [CONSOLE] mod_commands.c:329 Timer Test: 4
>> sleep 120 120048****
>>
>> 2011-11-24 14:53:51.622851 [CONSOLE] mod_commands.c:329 Timer Test: 5
>> sleep 120 120213****
>>
>> 2011-11-24 14:53:51.742593 [CONSOLE] mod_commands.c:329 Timer Test: 6
>> sleep 120 119681****
>>
>> 2011-11-24 14:53:51.862583 [CONSOLE] mod_commands.c:329 Timer Test: 7
>> sleep 120 119963****
>>
>> 2011-11-24 14:53:51.982568 [CONSOLE] mod_commands.c:329 Timer Test: 8
>> sleep 120 119975****
>>
>> 2011-11-24 14:53:52.133931 [CONSOLE] mod_commands.c:329 Timer Test: 9
>> sleep 120 151338****
>>
>> 2011-11-24 14:53:52.232591 [CONSOLE] mod_commands.c:329 Timer Test: 10
>> sleep 120 98753****
>>
>> ** **
>>
>> ** **
>>
>> *1000hz kernel*
>>
>> ** **
>>
>> *time_test 600 10*
>>
>> test 1 sleep 600 1591****
>>
>> test 2 sleep 600 1009****
>>
>> test 3 sleep 600 971****
>>
>> test 4 sleep 600 1114****
>>
>> test 5 sleep 600 888****
>>
>> test 6 sleep 600 996****
>>
>> test 7 sleep 600 998****
>>
>> test 8 sleep 600 988****
>>
>> test 9 sleep 600 994****
>>
>> test 10 sleep 600 996****
>>
>> avg 1054****
>>
>> ** **
>>
>> *timer_test 120 10*****
>>
>> 2011-11-24 19:55:27.198564 [CONSOLE] mod_commands.c:562 Timer Test: 1
>> sleep 120 119956****
>>
>> 2011-11-24 19:55:27.318559 [CONSOLE] mod_commands.c:562 Timer Test: 2
>> sleep 120 119994****
>>
>> 2011-11-24 19:55:27.438559 [CONSOLE] mod_commands.c:562 Timer Test: 3
>> sleep 120 119939****
>>
>> 2011-11-24 19:55:27.558559 [CONSOLE] mod_commands.c:562 Timer Test: 4
>> sleep 120 119965****
>>
>> 2011-11-24 19:55:27.678559 [CONSOLE] mod_commands.c:562 Timer Test: 5
>> sleep 120 119967****
>>
>> 2011-11-24 19:55:27.798563 [CONSOLE] mod_commands.c:562 Timer Test: 6
>> sleep 120 119977****
>>
>> 2011-11-24 19:55:27.918559 [CONSOLE] mod_commands.c:562 Timer Test: 7
>> sleep 120 120000****
>>
>> 2011-11-24 19:55:28.038559 [CONSOLE] mod_commands.c:562 Timer Test: 8
>> sleep 120 119936****
>>
>> 2011-11-24 19:55:28.158559 [CONSOLE] mod_commands.c:562 Timer Test: 9
>> sleep 120 119973****
>>
>> 2011-11-24 19:55:28.278558 [CONSOLE] mod_commands.c:562 Timer Test: 10
>> sleep 120 119968****
>>
>> ** **
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Fraser
>> Redmond
>>
>> *Sent:* Saturday, November 19, 2011 7:43 PM
>> *To:* FreeSWITCH Users Help
>> *Subject:* Re: [Freeswitch-users] Timing issues in AWS?****
>>
>> ** **
>>
>> Hi Stephen,****
>>
>> ** **
>>
>> I spent most of last week working out how to upgrade the kernel timer
>> from 100HZ to 1000HZ on ubuntu on AWS. I documented the steps I took here:
>> ****
>>
>> ** **
>>
>> http://wiki.freeswitch.org/wiki/Amazon_EC2#Updating_Kernel_Timer_to_1000HZ
>> ****
>>
>> ** **
>>
>> Making that change gave us a noticeable jump in call quality.****
>>
>>
>> Cheers,
>> Fraser
>>
>>
>>
>> ****
>>
>> On 17 November 2011 08:49, Stephen Dame <sdame at 207me.com> wrote:****
>>
>> I’m running freeswitch on about 40 different m1.small c1.medium’s in AWS
>> regions, us-east, us-west, eu-west, and asia…. They are used in videoconf
>> component of BigBlueButton.org . For most part they work great… There are
>> occasional issues with voip quality but the app is 100% voip, with the BBB
>> client all browser based. So the conference is subject to every ones local
>> network connections and most issues “blamed” on the internet instead of
>> freeswitch J We also tie in skype and DID direct to improve latency for
>> some clients. But expectations are set so we meet them.****
>>
>> ****
>>
>> You probably have tougher business conferencing clients that want perfect
>> audio. But these are production deployed and generate revenue. All
>> these are on Ubuntu 10.04 official amis with no mods to kernels. ****
>>
>> ****
>>
>> I’m sure there are tweaks that can be made, and bare metal solutions that
>> would work a little better… Does anyone have any ideas how to optimize a
>> Ubuntu instance. I would engage in a few hours of consulting is so.****
>>
>> ****
>>
>> The tests below are interesting but above my paygrade to understand what
>> they mean? ****
>>
>> ****
>>
>> I’m running them but don’t have a clue how to interpret. If you want a
>> simple of real running data, would be glad to run sample tests on these
>> distributed servers and provide back for analysis.****
>>
>> ****
>>
>> Regards,****
>>
>> Stephen****
>>
>> ****
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Michael
>> Picher
>> *Sent:* Thursday, November 17, 2011 8:03 AM
>> *To:* FreeSWITCH Users Help
>> *Subject:* Re: [Freeswitch-users] Timing issues in AWS?****
>>
>> ****
>>
>> Not having anywhere near the same trouble in a Xen Server in my lab...
>>
>> Although test_time is skewed 180 degrees from what it was in AWS....
>>
>> freeswitch at internal> timer_test 120 10****
>>
>> Avg: 119.945ms Total Time: 1199.727ms
>>
>> 2011-11-17 07:57:22.721105 [CONSOLE] mod_commands.c:461 Timer Test:
>> samplecount after init: 1
>> 2011-11-17 07:57:22.751026 [CONSOLE] mod_commands.c:466 Timer Test:
>> samplecount after first step: 2
>> 2011-11-17 07:57:22.872047 [CONSOLE] mod_commands.c:475 Timer Test: 1
>> sleep 120 120993
>> 2011-11-17 07:57:22.990983 [CONSOLE] mod_commands.c:475 Timer Test: 2
>> sleep 120 118914
>> 2011-11-17 07:57:23.110953 [CONSOLE] mod_commands.c:475 Timer Test: 3
>> sleep 120 119919
>> 2011-11-17 07:57:23.230928 [CONSOLE] mod_commands.c:475 Timer Test: 4
>> sleep 120 120032
>> 2011-11-17 07:57:23.350831 [CONSOLE] mod_commands.c:475 Timer Test: 5
>> sleep 120 119802
>> 2011-11-17 07:57:23.470886 [CONSOLE] mod_commands.c:475 Timer Test: 6
>> sleep 120 119999
>> 2011-11-17 07:57:23.590873 [CONSOLE] mod_commands.c:475 Timer Test: 7
>> sleep 120 119975
>> 2011-11-17 07:57:23.710893 [CONSOLE] mod_commands.c:475 Timer Test: 8
>> sleep 120 120003
>> 2011-11-17 07:57:23.830774 [CONSOLE] mod_commands.c:475 Timer Test: 9
>> sleep 120 119849
>> 2011-11-17 07:57:23.950808 [CONSOLE] mod_commands.c:475 Timer Test: 10
>> sleep 120 119971****
>>
>>
>> freeswitch at internal> time_test 600 10****
>>
>> test 1 sleep 600 1
>> test 2 sleep 600 1
>> test 3 sleep 600 0
>> test 4 sleep 600 1
>> test 5 sleep 600 0
>> test 6 sleep 600 1
>> test 7 sleep 600 1
>> test 8 sleep 600 0
>> test 9 sleep 600 0
>> test 10 sleep 600 1
>> avg 0****
>>
>>
>> [root at openuc bin]# uname -r****
>>
>> 2.6.18-274.7.1.el5****
>>
>>
>> [root at openuc bin]# grep CONFIG_HZ /boot/config-*****
>>
>> /boot/config-2.6.18-238.12.1.el5:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-238.12.1.el5:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-238.12.1.el5:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-238.12.1.el5:CONFIG_HZ=1000
>> /boot/config-2.6.18-238.19.1.el5:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-238.19.1.el5:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-238.19.1.el5:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-238.19.1.el5:CONFIG_HZ=1000
>> /boot/config-2.6.18-238.9.1.el5:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-238.9.1.el5:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-238.9.1.el5:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-238.9.1.el5:CONFIG_HZ=1000
>> /boot/config-2.6.18-274.3.1.el5:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-274.3.1.el5:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-274.3.1.el5:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-274.3.1.el5:CONFIG_HZ=1000
>> /boot/config-2.6.18-274.7.1.el5:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-274.7.1.el5:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-274.7.1.el5:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-274.7.1.el5:CONFIG_HZ=1000****
>>
>>
>> Thoughts as to why AWS results are so different from XenServer? Other
>> than not knowing who else is on the AWS box?
>>
>> Thanks,
>> Mike
>>
>> ****
>>
>> On Wed, Nov 16, 2011 at 11:56 AM, Michael Picher <mpicher at gmail.com>
>> wrote:****
>>
>> m1.large
>>
>> I have a c1.xlarge queued up and ready to test...****
>>
>> ****
>>
>> On Wed, Nov 16, 2011 at 11:52 AM, Chris Chen <chris.chen2004 at gmail.com>
>> wrote:****
>>
>> Just a simple question, what kind of AWS instance are you running your
>> FreeSWITCH?****
>>
>> It makes huge difference.****
>>
>> Thanks,****
>>
>> Chris****
>>
>> On Wed, Nov 16, 2011 at 10:43 AM, Michael Picher <mpicher at gmail.com>
>> wrote:****
>>
>> Hi guys,
>>
>> Trying to get to the bottom of some conference bridge issues I'm having
>> with running the system in AWS.
>>
>> We're hearing a bunch of snap-crackle-pops in conference bridges and when
>> I tcpdum on the server itself I see them in the RTP and see RTP timestamp
>> problems.
>>
>> I've run the following:
>>
>>
>> timer_test****
>>
>> freeswitch at 127.0.0.1@internal> timer_test 120 10****
>>
>> Avg: 120.004ms Total Time: 1200.315ms****
>>
>> ****
>>
>> 2011-11-16 10:25:56.121131 [CONSOLE] mod_commands.c:461 Timer Test:
>> samplecount after init: 1
>> freeswitch at 127.0.0.1@internal> 2011-11-16 10:25:56.219163 [CONSOLE]
>> mod_commands.c:466 Timer Test: samplecount after first step: 2
>> 2011-11-16 10:25:56.339195 [CONSOLE] mod_commands.c:475 Timer Test: 1
>> sleep 120 120006
>> 2011-11-16 10:25:56.459259 [CONSOLE] mod_commands.c:475 Timer Test: 2
>> sleep 120 120040
>> 2011-11-16 10:25:56.579259 [CONSOLE] mod_commands.c:475 Timer Test: 3
>> sleep 120 119976
>> 2011-11-16 10:25:56.699291 [CONSOLE] mod_commands.c:475 Timer Test: 4
>> sleep 120 120007
>> 2011-11-16 10:25:56.819318 [CONSOLE] mod_commands.c:475 Timer Test: 5
>> sleep 120 120001
>> 2011-11-16 10:25:56.939375 [CONSOLE] mod_commands.c:475 Timer Test: 6
>> sleep 120 120031
>> 2011-11-16 10:25:57.059397 [CONSOLE] mod_commands.c:475 Timer Test: 7
>> sleep 120 119996
>> 2011-11-16 10:25:57.179422 [CONSOLE] mod_commands.c:475 Timer Test: 8
>> sleep 120 119994
>> 2011-11-16 10:25:57.299461 [CONSOLE] mod_commands.c:475 Timer Test: 9
>> sleep 120 120005
>> 2011-11-16 10:25:57.419478 [CONSOLE] mod_commands.c:475 Timer Test: 10
>> sleep 120 119991****
>>
>>
>> test_time****
>>
>> freeswitch at 127.0.0.1@internal> time_test 600 10****
>>
>>
>> test 1 sleep 600 1592
>> test 2 sleep 600 986
>> test 3 sleep 600 1018
>> test 4 sleep 600 980
>> test 5 sleep 600 1005
>> test 6 sleep 600 1000
>> test 7 sleep 600 972
>> test 8 sleep 600 990
>> test 9 sleep 600 1006
>> test 10 sleep 600 994
>> avg 1054****
>>
>>
>> For kernel:****
>>
>> [root at openuc bin]# uname -r
>>
>> 2.6.21.7-2.fc8xen****
>>
>>
>> CONFIG_HZ:****
>>
>> [root at openuc bin]# grep CONFIG_HZ /boot/config-*
>>
>> /boot/config-2.6.16.33-xenU-x86_64:CONFIG_HZ_100=y
>> /boot/config-2.6.16.33-xenU-x86_64:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.16.33-xenU-x86_64:# CONFIG_HZ_1000 is not set
>> /boot/config-2.6.16.33-xenU-x86_64:CONFIG_HZ=100
>> /boot/config-2.6.18-164.15.1.el5.centos.plus:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.18-164.15.1.el5.centos.plus:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.18-164.15.1.el5.centos.plus:CONFIG_HZ_1000=y
>> /boot/config-2.6.18-164.15.1.el5.centos.plus:CONFIG_HZ=1000
>> /boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_100 is not set
>> /boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_250 is not set
>> /boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_300 is not set
>> /boot/config-2.6.21-2952.fc8xen:CONFIG_HZ_1000=y
>> /boot/config-2.6.21-2952.fc8xen:CONFIG_HZ=1000****
>>
>> are the xenU kernel settings screwing me here?
>>
>> Thanks,
>> Mike
>>
>>
>> --
>> There are 10 kinds of people in this world, those who understand binary
>> and those who don't.
>>
>> mpicher at gmail.com
>> blog: http://www.sipxecs.info
>> call: sip:mpicher at sipxecs.info****
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org****
>>
>> ****
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org****
>>
>>
>>
>>
>> --
>> There are 10 kinds of people in this world, those who understand binary
>> and those who don't.
>>
>> mpicher at gmail.com
>> blog: http://www.sipxecs.info
>> call: sip:mpicher at sipxecs.info****
>>
>>
>>
>>
>> --
>> There are 10 kinds of people in this world, those who understand binary
>> and those who don't.
>>
>> mpicher at gmail.com
>> blog: http://www.sipxecs.info
>> call: sip:mpicher at sipxecs.info****
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org****
>>
>> ** **
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
--
There are 10 kinds of people in this world, those who understand binary and
those who don't.
mpicher at gmail.com
blog: http://www.sipxecs.info
call: sip:mpicher at sipxecs.info
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111126/2b1c928a/attachment-0001.html
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list