[Freeswitch-users] Timing issues in AWS?

Fri Nov 25 00:08:19 MSK 2011

Fraser,  thanks for tips in compiling kernel.

I was able to set up a 1000hz ec2 instance ... and run some comparative
tests using the same starting 10.04 LTS ami.

Here are the results of timing tests.. Not sure how to interpret.
Having users test audio right now in voice conferences to see if they notice
any difference.

Looked in wiki. I'm still not clear on what to look for.

Regards,

Stephen

100hz kernel

time_test 600 10

test 1 sleep 600 8847

test 2 sleep 600 9969

test 3 sleep 600 10006

test 4 sleep 600 9986

test 5 sleep 600 10003

test 6 sleep 600 9977

test 7 sleep 600 10117

test 8 sleep 600 9872

test 9 sleep 600 10014

test 10 sleep 600 10124

timer_test 120 10

2011-11-24 14:53:51.142568 [CONSOLE] mod_commands.c:329 Timer Test: 1 sleep
120 120289

2011-11-24 14:53:51.263610 [CONSOLE] mod_commands.c:329 Timer Test: 2 sleep
120 120703

2011-11-24 14:53:51.382587 [CONSOLE] mod_commands.c:329 Timer Test: 3 sleep
120 118952

2011-11-24 14:53:51.502640 [CONSOLE] mod_commands.c:329 Timer Test: 4 sleep
120 120048

2011-11-24 14:53:51.622851 [CONSOLE] mod_commands.c:329 Timer Test: 5 sleep
120 120213

2011-11-24 14:53:51.742593 [CONSOLE] mod_commands.c:329 Timer Test: 6 sleep
120 119681

2011-11-24 14:53:51.862583 [CONSOLE] mod_commands.c:329 Timer Test: 7 sleep
120 119963

2011-11-24 14:53:51.982568 [CONSOLE] mod_commands.c:329 Timer Test: 8 sleep
120 119975

2011-11-24 14:53:52.133931 [CONSOLE] mod_commands.c:329 Timer Test: 9 sleep
120 151338

2011-11-24 14:53:52.232591 [CONSOLE] mod_commands.c:329 Timer Test: 10 sleep
120 98753

1000hz kernel

time_test 600 10

test 1 sleep 600 1591

test 2 sleep 600 1009

test 3 sleep 600 971

test 4 sleep 600 1114

test 5 sleep 600 888

test 6 sleep 600 996

test 7 sleep 600 998

test 8 sleep 600 988

test 9 sleep 600 994

test 10 sleep 600 996

avg 1054

timer_test 120 10

2011-11-24 19:55:27.198564 [CONSOLE] mod_commands.c:562 Timer Test: 1 sleep
120 119956

2011-11-24 19:55:27.318559 [CONSOLE] mod_commands.c:562 Timer Test: 2 sleep
120 119994

2011-11-24 19:55:27.438559 [CONSOLE] mod_commands.c:562 Timer Test: 3 sleep
120 119939

2011-11-24 19:55:27.558559 [CONSOLE] mod_commands.c:562 Timer Test: 4 sleep
120 119965

2011-11-24 19:55:27.678559 [CONSOLE] mod_commands.c:562 Timer Test: 5 sleep
120 119967

2011-11-24 19:55:27.798563 [CONSOLE] mod_commands.c:562 Timer Test: 6 sleep
120 119977

2011-11-24 19:55:27.918559 [CONSOLE] mod_commands.c:562 Timer Test: 7 sleep
120 120000

2011-11-24 19:55:28.038559 [CONSOLE] mod_commands.c:562 Timer Test: 8 sleep
120 119936

2011-11-24 19:55:28.158559 [CONSOLE] mod_commands.c:562 Timer Test: 9 sleep
120 119973

2011-11-24 19:55:28.278558 [CONSOLE] mod_commands.c:562 Timer Test: 10 sleep
120 119968

From: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Fraser
Redmond
Sent: Saturday, November 19, 2011 7:43 PM
To: FreeSWITCH Users Help
Subject: Re: [Freeswitch-users] Timing issues in AWS?

Hi Stephen,

I spent most of last week working out how to upgrade the kernel timer from
100HZ to 1000HZ on ubuntu on AWS. I documented the steps I took here:

http://wiki.freeswitch.org/wiki/Amazon_EC2#Updating_Kernel_Timer_to_1000HZ

Making that change gave us a noticeable jump in call quality.

Cheers,
Fraser

On 17 November 2011 08:49, Stephen Dame <sdame at 207me.com> wrote:

I'm running freeswitch on about 40  different m1.small c1.medium's in AWS
regions, us-east, us-west, eu-west, and asia..  They are used in videoconf
component of BigBlueButton.org .   For most part they work great. There are
occasional issues with voip quality but the app is 100% voip, with the BBB
client all browser based.  So the conference is subject to every ones local
network connections and most issues "blamed" on the internet instead of
freeswitch J  We also tie in skype and DID direct to improve latency for
some clients.  But expectations are set so we meet them.

You probably have tougher business conferencing clients that want perfect
audio.  But these are production deployed and generate revenue.      All
these are on Ubuntu 10.04 official amis with no mods to kernels.  

I'm sure there are tweaks that can be made, and bare metal solutions that
would work a little better.  Does anyone have any ideas how to optimize a
Ubuntu instance.  I would engage in a few hours of consulting is so.

The tests below are interesting but above my paygrade to understand what
they mean? 

I'm running them but don't have a clue how to interpret.  If you want a
simple of real running data, would be glad to run sample tests on these
distributed servers and provide back for analysis.

Regards,

Stephen

From: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Michael
Picher
Sent: Thursday, November 17, 2011 8:03 AM
To: FreeSWITCH Users Help
Subject: Re: [Freeswitch-users] Timing issues in AWS?

Not having anywhere near the same trouble in a Xen Server in my lab...

Although test_time is skewed 180 degrees from what it was in AWS....

freeswitch at internal> timer_test 120 10

Avg: 119.945ms Total Time: 1199.727ms

2011-11-17 07:57:22.721105 [CONSOLE] mod_commands.c:461 Timer Test:
samplecount after init: 1
2011-11-17 07:57:22.751026 [CONSOLE] mod_commands.c:466 Timer Test:
samplecount after first step: 2
2011-11-17 07:57:22.872047 [CONSOLE] mod_commands.c:475 Timer Test: 1 sleep
120 120993
2011-11-17 07:57:22.990983 [CONSOLE] mod_commands.c:475 Timer Test: 2 sleep
120 118914
2011-11-17 07:57:23.110953 [CONSOLE] mod_commands.c:475 Timer Test: 3 sleep
120 119919
2011-11-17 07:57:23.230928 [CONSOLE] mod_commands.c:475 Timer Test: 4 sleep
120 120032
2011-11-17 07:57:23.350831 [CONSOLE] mod_commands.c:475 Timer Test: 5 sleep
120 119802
2011-11-17 07:57:23.470886 [CONSOLE] mod_commands.c:475 Timer Test: 6 sleep
120 119999
2011-11-17 07:57:23.590873 [CONSOLE] mod_commands.c:475 Timer Test: 7 sleep
120 119975
2011-11-17 07:57:23.710893 [CONSOLE] mod_commands.c:475 Timer Test: 8 sleep
120 120003
2011-11-17 07:57:23.830774 [CONSOLE] mod_commands.c:475 Timer Test: 9 sleep
120 119849
2011-11-17 07:57:23.950808 [CONSOLE] mod_commands.c:475 Timer Test: 10 sleep
120 119971

freeswitch at internal> time_test 600 10

test 1 sleep 600 1
test 2 sleep 600 1
test 3 sleep 600 0
test 4 sleep 600 1
test 5 sleep 600 0
test 6 sleep 600 1
test 7 sleep 600 1
test 8 sleep 600 0
test 9 sleep 600 0
test 10 sleep 600 1
avg 0

[root at openuc bin]# uname -r

2.6.18-274.7.1.el5

[root at openuc bin]# grep CONFIG_HZ /boot/config-*

/boot/config-2.6.18-238.12.1.el5:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-238.12.1.el5:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-238.12.1.el5:CONFIG_HZ_1000=y
/boot/config-2.6.18-238.12.1.el5:CONFIG_HZ=1000
/boot/config-2.6.18-238.19.1.el5:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-238.19.1.el5:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-238.19.1.el5:CONFIG_HZ_1000=y
/boot/config-2.6.18-238.19.1.el5:CONFIG_HZ=1000
/boot/config-2.6.18-238.9.1.el5:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-238.9.1.el5:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-238.9.1.el5:CONFIG_HZ_1000=y
/boot/config-2.6.18-238.9.1.el5:CONFIG_HZ=1000
/boot/config-2.6.18-274.3.1.el5:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-274.3.1.el5:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-274.3.1.el5:CONFIG_HZ_1000=y
/boot/config-2.6.18-274.3.1.el5:CONFIG_HZ=1000
/boot/config-2.6.18-274.7.1.el5:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-274.7.1.el5:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-274.7.1.el5:CONFIG_HZ_1000=y
/boot/config-2.6.18-274.7.1.el5:CONFIG_HZ=1000

Thoughts as to why AWS results are so different from XenServer?  Other than
not knowing who else is on the AWS box?

Thanks,
  Mike

On Wed, Nov 16, 2011 at 11:56 AM, Michael Picher <mpicher at gmail.com> wrote:

m1.large

I have a c1.xlarge queued up and ready to test...

On Wed, Nov 16, 2011 at 11:52 AM, Chris Chen <chris.chen2004 at gmail.com>
wrote:

Just a simple question, what kind of AWS instance are you running your
FreeSWITCH?

It makes huge difference.

Thanks,

Chris

On Wed, Nov 16, 2011 at 10:43 AM, Michael Picher <mpicher at gmail.com> wrote:

Hi guys,

Trying to get to the bottom of some conference bridge issues I'm having with
running the system in AWS.

We're hearing a bunch of snap-crackle-pops in conference bridges and when I
tcpdum on the server itself I see them in the RTP and see RTP timestamp
problems.

I've run the following:

timer_test

freeswitch at 127.0.0.1@internal> timer_test 120 10

Avg: 120.004ms Total Time: 1200.315ms

2011-11-16 10:25:56.121131 [CONSOLE] mod_commands.c:461 Timer Test:
samplecount after init: 1
freeswitch at 127.0.0.1@internal> 2011-11-16 10:25:56.219163 [CONSOLE]
mod_commands.c:466 Timer Test: samplecount after first step: 2
2011-11-16 10:25:56.339195 [CONSOLE] mod_commands.c:475 Timer Test: 1 sleep
120 120006
2011-11-16 10:25:56.459259 [CONSOLE] mod_commands.c:475 Timer Test: 2 sleep
120 120040
2011-11-16 10:25:56.579259 [CONSOLE] mod_commands.c:475 Timer Test: 3 sleep
120 119976
2011-11-16 10:25:56.699291 [CONSOLE] mod_commands.c:475 Timer Test: 4 sleep
120 120007
2011-11-16 10:25:56.819318 [CONSOLE] mod_commands.c:475 Timer Test: 5 sleep
120 120001
2011-11-16 10:25:56.939375 [CONSOLE] mod_commands.c:475 Timer Test: 6 sleep
120 120031
2011-11-16 10:25:57.059397 [CONSOLE] mod_commands.c:475 Timer Test: 7 sleep
120 119996
2011-11-16 10:25:57.179422 [CONSOLE] mod_commands.c:475 Timer Test: 8 sleep
120 119994
2011-11-16 10:25:57.299461 [CONSOLE] mod_commands.c:475 Timer Test: 9 sleep
120 120005
2011-11-16 10:25:57.419478 [CONSOLE] mod_commands.c:475 Timer Test: 10 sleep
120 119991

test_time

freeswitch at 127.0.0.1@internal> time_test 600 10

test 1 sleep 600 1592
test 2 sleep 600 986
test 3 sleep 600 1018
test 4 sleep 600 980
test 5 sleep 600 1005
test 6 sleep 600 1000
test 7 sleep 600 972
test 8 sleep 600 990
test 9 sleep 600 1006
test 10 sleep 600 994
avg 1054

For kernel:

[root at openuc bin]# uname -r

2.6.21.7-2.fc8xen

CONFIG_HZ:

[root at openuc bin]# grep CONFIG_HZ /boot/config-*

/boot/config-2.6.16.33-xenU-x86_64:CONFIG_HZ_100=y
/boot/config-2.6.16.33-xenU-x86_64:# CONFIG_HZ_250 is not set
/boot/config-2.6.16.33-xenU-x86_64:# CONFIG_HZ_1000 is not set
/boot/config-2.6.16.33-xenU-x86_64:CONFIG_HZ=100
/boot/config-2.6.18-164.15.1.el5.centos.plus:# CONFIG_HZ_100 is not set
/boot/config-2.6.18-164.15.1.el5.centos.plus:# CONFIG_HZ_250 is not set
/boot/config-2.6.18-164.15.1.el5.centos.plus:CONFIG_HZ_1000=y
/boot/config-2.6.18-164.15.1.el5.centos.plus:CONFIG_HZ=1000
/boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_100 is not set
/boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_250 is not set
/boot/config-2.6.21-2952.fc8xen:# CONFIG_HZ_300 is not set
/boot/config-2.6.21-2952.fc8xen:CONFIG_HZ_1000=y
/boot/config-2.6.21-2952.fc8xen:CONFIG_HZ=1000

are the xenU kernel settings screwing me here?

Thanks,
  Mike

-- 
There are 10 kinds of people in this world, those who understand binary and
those who don't.

mpicher at gmail.com
blog: http://www.sipxecs.info
call: sip:mpicher at sipxecs.info <mailto:sip%3Ampicher at sipxecs.info> 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

-- 
There are 10 kinds of people in this world, those who understand binary and
those who don't.

mpicher at gmail.com
blog: http://www.sipxecs.info
call: sip:mpicher at sipxecs.info <mailto:sip%3Ampicher at sipxecs.info> 

-- 
There are 10 kinds of people in this world, those who understand binary and
those who don't.

mpicher at gmail.com
blog: http://www.sipxecs.info
call: sip:mpicher at sipxecs.info <mailto:sip%3Ampicher at sipxecs.info> 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20111124/576c6308/attachment-0001.html 