[Freeswitch-users] Choppy audio when conferencing 4+ participants

Tue Apr 18 21:15:46 MSD 2017

Ah I feel better :)

To the OP, Bilal:

on a FS 1.5 running on KVM (running on a Dell R610), I have timer_test giving quite stable results:

2017-04-18 19:11:41.205316 [CONSOLE] mod_commands.c:846 Timer Test: 1 sleep 20 19994
2017-04-18 19:11:41.225341 [CONSOLE] mod_commands.c:846 Timer Test: 2 sleep 20 20001
2017-04-18 19:11:41.245338 [CONSOLE] mod_commands.c:846 Timer Test: 3 sleep 20 20020
2017-04-18 19:11:41.265341 [CONSOLE] mod_commands.c:846 Timer Test: 4 sleep 20 19981
2017-04-18 19:11:41.285341 [CONSOLE] mod_commands.c:846 Timer Test: 5 sleep 20 20023
2017-04-18 19:11:41.305354 [CONSOLE] mod_commands.c:846 Timer Test: 6 sleep 20 19990
2017-04-18 19:11:41.325343 [CONSOLE] mod_commands.c:846 Timer Test: 7 sleep 20 20011
2017-04-18 19:11:41.345343 [CONSOLE] mod_commands.c:846 Timer Test: 8 sleep 20 19978
2017-04-18 19:11:41.365339 [CONSOLE] mod_commands.c:846 Timer Test: 9 sleep 20 20020
2017-04-18 19:11:41.385347 [CONSOLE] mod_commands.c:846 Timer Test: 10 sleep 20 19984
2017-04-18 19:11:41.405335 [CONSOLE] mod_commands.c:846 Timer Test: 11 sleep 20 20010
2017-04-18 19:11:41.425347 [CONSOLE] mod_commands.c:846 Timer Test: 12 sleep 20 19990
2017-04-18 19:11:41.445336 [CONSOLE] mod_commands.c:846 Timer Test: 13 sleep 20 20014
2017-04-18 19:11:41.465337 [CONSOLE] mod_commands.c:846 Timer Test: 14 sleep 20 19983
2017-04-18 19:11:41.485344 [CONSOLE] mod_commands.c:846 Timer Test: 15 sleep 20 20028
2017-04-18 19:11:41.505372 [CONSOLE] mod_commands.c:846 Timer Test: 16 sleep 20 19988
2017-04-18 19:11:41.525370 [CONSOLE] mod_commands.c:846 Timer Test: 17 sleep 20 20038
2017-04-18 19:11:41.545372 [CONSOLE] mod_commands.c:846 Timer Test: 18 sleep 20 19968
2017-04-18 19:11:41.565369 [CONSOLE] mod_commands.c:846 Timer Test: 19 sleep 20 20024
2017-04-18 19:11:41.585372 [CONSOLE] mod_commands.c:846 Timer Test: 20 sleep 20 19982
2017-04-18 19:11:41.605312 [CONSOLE] mod_commands.c:846 Timer Test: 21 sleep 20 19966
2017-04-18 19:11:41.625345 [CONSOLE] mod_commands.c:846 Timer Test: 22 sleep 20 20007
2017-04-18 19:11:41.645357 [CONSOLE] mod_commands.c:846 Timer Test: 23 sleep 20 20035
2017-04-18 19:11:41.665372 [CONSOLE] mod_commands.c:846 Timer Test: 24 sleep 20 19992
2017-04-18 19:11:41.685369 [CONSOLE] mod_commands.c:846 Timer Test: 25 sleep 20 20020
2017-04-18 19:11:41.705376 [CONSOLE] mod_commands.c:846 Timer Test: 26 sleep 20 19984
2017-04-18 19:11:41.725375 [CONSOLE] mod_commands.c:846 Timer Test: 27 sleep 20 20022
2017-04-18 19:11:41.745370 [CONSOLE] mod_commands.c:846 Timer Test: 28 sleep 20 20093
2017-04-18 19:11:41.765370 [CONSOLE] mod_commands.c:846 Timer Test: 29 sleep 20 19902
2017-04-18 19:11:41.785367 [CONSOLE] mod_commands.c:846 Timer Test: 30 sleep 20 19974
2017-04-18 19:11:41.805341 [CONSOLE] mod_commands.c:846 Timer Test: 31 sleep 20 19996
2017-04-18 19:11:41.825329 [CONSOLE] mod_commands.c:846 Timer Test: 32 sleep 20 20081
2017-04-18 19:11:41.845368 [CONSOLE] mod_commands.c:846 Timer Test: 33 sleep 20 19953
2017-04-18 19:11:41.865376 [CONSOLE] mod_commands.c:846 Timer Test: 34 sleep 20 20105
2017-04-18 19:11:41.885376 [CONSOLE] mod_commands.c:846 Timer Test: 35 sleep 20 19967
2017-04-18 19:11:41.905350 [CONSOLE] mod_commands.c:846 Timer Test: 36 sleep 20 19882
2017-04-18 19:11:41.925347 [CONSOLE] mod_commands.c:846 Timer Test: 37 sleep 20 20021
2017-04-18 19:11:41.945378 [CONSOLE] mod_commands.c:846 Timer Test: 38 sleep 20 20007
2017-04-18 19:11:41.965332 [CONSOLE] mod_commands.c:846 Timer Test: 39 sleep 20 19983
2017-04-18 19:11:41.985333 [CONSOLE] mod_commands.c:846 Timer Test: 40 sleep 20 19967
2017-04-18 19:11:42.005327 [CONSOLE] mod_commands.c:846 Timer Test: 41 sleep 20 20001
2017-04-18 19:11:42.025337 [CONSOLE] mod_commands.c:846 Timer Test: 42 sleep 20 20008
2017-04-18 19:11:42.045334 [CONSOLE] mod_commands.c:846 Timer Test: 43 sleep 20 20005
2017-04-18 19:11:42.065360 [CONSOLE] mod_commands.c:846 Timer Test: 44 sleep 20 20018
2017-04-18 19:11:42.085372 [CONSOLE] mod_commands.c:846 Timer Test: 45 sleep 20 20039
2017-04-18 19:11:42.105377 [CONSOLE] mod_commands.c:846 Timer Test: 46 sleep 20 19978
2017-04-18 19:11:42.125370 [CONSOLE] mod_commands.c:846 Timer Test: 47 sleep 20 20015
2017-04-18 19:11:42.145372 [CONSOLE] mod_commands.c:846 Timer Test: 48 sleep 20 19980
2017-04-18 19:11:42.165376 [CONSOLE] mod_commands.c:846 Timer Test: 49 sleep 20 20027
2017-04-18 19:11:42.185368 [CONSOLE] mod_commands.c:846 Timer Test: 50 sleep 20 19969

Be sure to enable softtimer in switch.conf.xml
timer_test was far less table without it.

David Ponzone  Direction Technique
email: david.ponzone at ipeva.fr <mailto:david.ponzone at ipeva.fr>
tel:      01 74 03 18 97
gsm:   06 66 98 76 34

Service Client IPeva
tel:      0811 46 26 26
www.ipeva.fr <blocked::http://www.ipeva.fr/>  -   www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/>

Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.

> Le 18 avr. 2017 à 18:56, Anthony Minessale <anthony.minessale at gmail.com> a écrit :
> 
> In general when a thread about performance or vm turns to lots of theories.  This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole.  To the contrary, asking about timer test was a good start!
> 
> 
> 
> 
> On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike at jerris.com <mailto:mike at jerris.com>> wrote:
> Superstition comment was in regards to tc malloc, not you.
> 
> 
>> On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone at gmail.com <mailto:david.ponzone at gmail.com>> wrote:
>> 
> 
>> Anthony,
>> 
>> Could you elaborate a little bit more on why I am being superstitious ?
>> Is timer_test command obsolete ?
>> Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
>> 
> 
>>> Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale at gmail.com <mailto:anthony.minessale at gmail.com>> a écrit :
>>> 
> 
>>> I believe we may be stumbling into superstition at this point.
>>> 
> 
>>> On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone at gmail.com <mailto:david.ponzone at gmail.com>> wrote:
> 
>>> Did you run the timer_test command ?
> 
>>>> Le 14 avr. 2017 à 23:04, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> a écrit :
>>>> 
>>>> The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier  m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. 
>>>> 
>>>> Model   vCPU    Mem (GiB)       SSD Storage (GB)        
>>>> m3.large        2       7.5     1 x 32  
>>>> m3.2xlarge      8       30      2 x 80  
>>>> 
>>>> 
>>>> 
>>>> On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>>> Robert,
>>>> 
>>>> While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
>>>> 
>>>> Bilal,
>>>> 
>>>> I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
>>>> 
>>>> Best,
>>>> Colin
>>>> 
>>>> On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky at ets.org <mailto:rmundkowsky at ets.org>> wrote:
>>>> Michael <>
>>>>  
>>>> 
>>>> Just curious, why so many problems with VMs?
>>>> 
>>>>  
>>>> 
>>>> I would think most applications need real time clocks that provide consistent valid data?
>>>> 
>>>>  
>>>> 
>>>> Bilal,
>>>> 
>>>>  
>>>> 
>>>> you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
>>>> 
>>>>  
>>>> 
>>>> Robert
>>>> 
>>>>  
>>>> 
>>>> From: freeswitch-users-bounces at lists.freeswitch.org <mailto:freeswitch-users-bounces at lists.freeswitch.org> [mailto:freeswitch-users-bounces at lists.freeswitch.org <mailto:freeswitch-users-bounces at lists.freeswitch.org>] On Behalf Of Michael Jerris
>>>> Sent: Friday, April 14, 2017 4:25 PM
>>>> To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org <mailto:freeswitch-users at lists.freeswitch.org>>
>>>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
>>>> 
>>>>  
>>>> 
>>>> can you reproduce the same issue on real hardware?  We’ve seen all kinds of weird timing issues that could account for this running on aws.
>>>> 
>>>>  
>>>> 
>>>> On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> wrote:
>>>> 
>>>>  
>>>> 
>>>> Its an AWS m3.2xlarge instance.
>>>> 
>>>>  
>>>> 
>>>> On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike at jerris.com <mailto:mike at jerris.com>> wrote:
>>>> 
>>>> Real hardware or VM?
>>>> 
>>>> > On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> wrote:
>>>> >
>>>> > I have been struggling with an issue for almost 2 weeks.
>>>> >
>>>> > Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
>>>> >
>>>> > We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
>>>> >
>>>> > I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
>>>> >
>>>> > I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
>>>> 
>>>>  
>>>> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org <http://www.freeswitch.org/>
> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
> http://www.cluecon.com <http://www.cluecon.com/>
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
> http://www.freeswitch.org <http://www.freeswitch.org/>
> -- 
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
> 
> ☞ http://freeswitch.org/ <http://freeswitch.org/>  ☞ http://cluecon.com/ <http://cluecon.com/>  ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH>
> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
> 
> ClueCon Weekly Development Call 
> ☎ sip:888 at conference.freeswitch.org <mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900 
> 
> https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s>
> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services: 
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170418/1de49386/attachment-0001.html 