I think that version wouldn’t have been using timerfd.  Using timerfd SHOULD be way more stable and less cpu, if its not, that would indicate an issue with the vm of some sort.


On Apr 18, 2017, at 1:15 PM, David Ponzone <david.ponzone@gmail.com> wrote:

Ah I feel better :)

To the OP, Bilal:

on a FS 1.5 running on KVM (running on a Dell R610), I have timer_test giving quite stable results:

2017-04-18 19:11:41.205316 [CONSOLE] mod_commands.c:846 Timer Test: 1 sleep 20 19994
2017-04-18 19:11:41.225341 [CONSOLE] mod_commands.c:846 Timer Test: 2 sleep 20 20001
2017-04-18 19:11:41.245338 [CONSOLE] mod_commands.c:846 Timer Test: 3 sleep 20 20020
2017-04-18 19:11:41.265341 [CONSOLE] mod_commands.c:846 Timer Test: 4 sleep 20 19981
2017-04-18 19:11:41.285341 [CONSOLE] mod_commands.c:846 Timer Test: 5 sleep 20 20023
2017-04-18 19:11:41.305354 [CONSOLE] mod_commands.c:846 Timer Test: 6 sleep 20 19990
2017-04-18 19:11:41.325343 [CONSOLE] mod_commands.c:846 Timer Test: 7 sleep 20 20011
2017-04-18 19:11:41.345343 [CONSOLE] mod_commands.c:846 Timer Test: 8 sleep 20 19978
2017-04-18 19:11:41.365339 [CONSOLE] mod_commands.c:846 Timer Test: 9 sleep 20 20020
2017-04-18 19:11:41.385347 [CONSOLE] mod_commands.c:846 Timer Test: 10 sleep 20 19984
2017-04-18 19:11:41.405335 [CONSOLE] mod_commands.c:846 Timer Test: 11 sleep 20 20010
2017-04-18 19:11:41.425347 [CONSOLE] mod_commands.c:846 Timer Test: 12 sleep 20 19990
2017-04-18 19:11:41.445336 [CONSOLE] mod_commands.c:846 Timer Test: 13 sleep 20 20014
2017-04-18 19:11:41.465337 [CONSOLE] mod_commands.c:846 Timer Test: 14 sleep 20 19983
2017-04-18 19:11:41.485344 [CONSOLE] mod_commands.c:846 Timer Test: 15 sleep 20 20028
2017-04-18 19:11:41.505372 [CONSOLE] mod_commands.c:846 Timer Test: 16 sleep 20 19988
2017-04-18 19:11:41.525370 [CONSOLE] mod_commands.c:846 Timer Test: 17 sleep 20 20038
2017-04-18 19:11:41.545372 [CONSOLE] mod_commands.c:846 Timer Test: 18 sleep 20 19968
2017-04-18 19:11:41.565369 [CONSOLE] mod_commands.c:846 Timer Test: 19 sleep 20 20024
2017-04-18 19:11:41.585372 [CONSOLE] mod_commands.c:846 Timer Test: 20 sleep 20 19982
2017-04-18 19:11:41.605312 [CONSOLE] mod_commands.c:846 Timer Test: 21 sleep 20 19966
2017-04-18 19:11:41.625345 [CONSOLE] mod_commands.c:846 Timer Test: 22 sleep 20 20007
2017-04-18 19:11:41.645357 [CONSOLE] mod_commands.c:846 Timer Test: 23 sleep 20 20035
2017-04-18 19:11:41.665372 [CONSOLE] mod_commands.c:846 Timer Test: 24 sleep 20 19992
2017-04-18 19:11:41.685369 [CONSOLE] mod_commands.c:846 Timer Test: 25 sleep 20 20020
2017-04-18 19:11:41.705376 [CONSOLE] mod_commands.c:846 Timer Test: 26 sleep 20 19984
2017-04-18 19:11:41.725375 [CONSOLE] mod_commands.c:846 Timer Test: 27 sleep 20 20022
2017-04-18 19:11:41.745370 [CONSOLE] mod_commands.c:846 Timer Test: 28 sleep 20 20093
2017-04-18 19:11:41.765370 [CONSOLE] mod_commands.c:846 Timer Test: 29 sleep 20 19902
2017-04-18 19:11:41.785367 [CONSOLE] mod_commands.c:846 Timer Test: 30 sleep 20 19974
2017-04-18 19:11:41.805341 [CONSOLE] mod_commands.c:846 Timer Test: 31 sleep 20 19996
2017-04-18 19:11:41.825329 [CONSOLE] mod_commands.c:846 Timer Test: 32 sleep 20 20081
2017-04-18 19:11:41.845368 [CONSOLE] mod_commands.c:846 Timer Test: 33 sleep 20 19953
2017-04-18 19:11:41.865376 [CONSOLE] mod_commands.c:846 Timer Test: 34 sleep 20 20105
2017-04-18 19:11:41.885376 [CONSOLE] mod_commands.c:846 Timer Test: 35 sleep 20 19967
2017-04-18 19:11:41.905350 [CONSOLE] mod_commands.c:846 Timer Test: 36 sleep 20 19882
2017-04-18 19:11:41.925347 [CONSOLE] mod_commands.c:846 Timer Test: 37 sleep 20 20021
2017-04-18 19:11:41.945378 [CONSOLE] mod_commands.c:846 Timer Test: 38 sleep 20 20007
2017-04-18 19:11:41.965332 [CONSOLE] mod_commands.c:846 Timer Test: 39 sleep 20 19983
2017-04-18 19:11:41.985333 [CONSOLE] mod_commands.c:846 Timer Test: 40 sleep 20 19967
2017-04-18 19:11:42.005327 [CONSOLE] mod_commands.c:846 Timer Test: 41 sleep 20 20001
2017-04-18 19:11:42.025337 [CONSOLE] mod_commands.c:846 Timer Test: 42 sleep 20 20008
2017-04-18 19:11:42.045334 [CONSOLE] mod_commands.c:846 Timer Test: 43 sleep 20 20005
2017-04-18 19:11:42.065360 [CONSOLE] mod_commands.c:846 Timer Test: 44 sleep 20 20018
2017-04-18 19:11:42.085372 [CONSOLE] mod_commands.c:846 Timer Test: 45 sleep 20 20039
2017-04-18 19:11:42.105377 [CONSOLE] mod_commands.c:846 Timer Test: 46 sleep 20 19978
2017-04-18 19:11:42.125370 [CONSOLE] mod_commands.c:846 Timer Test: 47 sleep 20 20015
2017-04-18 19:11:42.145372 [CONSOLE] mod_commands.c:846 Timer Test: 48 sleep 20 19980
2017-04-18 19:11:42.165376 [CONSOLE] mod_commands.c:846 Timer Test: 49 sleep 20 20027
2017-04-18 19:11:42.185368 [CONSOLE] mod_commands.c:846 Timer Test: 50 sleep 20 19969

Be sure to enable softtimer in switch.conf.xml
timer_test was far less table without it.

David Ponzone  Direction Technique
tel:      01 74 03 18 97
gsm:   06 66 98 76 34

Service Client IPeva
tel:      0811 46 26 26

Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.




Le 18 avr. 2017 à 18:56, Anthony Minessale <anthony.minessale@gmail.com> a écrit :

In general when a thread about performance or vm turns to lots of theories.  This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole.  To the contrary, asking about timer test was a good start!




On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com> wrote:
Superstition comment was in regards to tc malloc, not you.

On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote:

Anthony,

Could you elaborate a little bit more on why I am being superstitious ?
Is timer_test command obsolete ?
Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?

Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit :

I believe we may be stumbling into superstition at this point.

On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :

The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier  m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. 

Model   vCPU    Mem (GiB)       SSD Storage (GB)        
m3.large        2       7.5     1 x 32  
m3.2xlarge      8       30      2 x 80  



On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,

While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.

Bilal,

I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.

Best,
Colin

On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:

Michael

 

Just curious, why so many problems with VMs?

 

I would think most applications need real time clocks that provide consistent valid data?

 

Bilal,

 

you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.

 

Robert

 

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris
Sent: Friday, April 14, 2017 4:25 PM
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants

 

can you reproduce the same issue on real hardware?  We’ve seen all kinds of weird timing issues that could account for this running on aws.

 

On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:

 

Its an AWS m3.2xlarge instance.

 

On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:

Real hardware or VM?

> On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
>
> I have been struggling with an issue for almost 2 weeks.
>
> Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
>
> We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
>
> I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
>
> I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.

 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

ClueCon Weekly Development Call 
☎ sip:888@conference.freeswitch.org  ☎ +19193869900 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org