[Freeswitch-users] Choppy audio when conferencing 4+ participants

Michael Jerris mike at jerris.com
Wed Apr 19 18:48:02 MSD 2017 

I think that version wouldn’t have been using timerfd.  Using timerfd SHOULD be way more stable and less cpu, if its not, that would indicate an issue with the vm of some sort.


> On Apr 18, 2017, at 1:15 PM, David Ponzone <david.ponzone at gmail.com> wrote:
> 
> Ah I feel better :)
> 
> To the OP, Bilal:
> 
> on a FS 1.5 running on KVM (running on a Dell R610), I have timer_test giving quite stable results:
> 
> 2017-04-18 19:11:41.205316 [CONSOLE] mod_commands.c:846 Timer Test: 1 sleep 20 19994
> 2017-04-18 19:11:41.225341 [CONSOLE] mod_commands.c:846 Timer Test: 2 sleep 20 20001
> 2017-04-18 19:11:41.245338 [CONSOLE] mod_commands.c:846 Timer Test: 3 sleep 20 20020
> 2017-04-18 19:11:41.265341 [CONSOLE] mod_commands.c:846 Timer Test: 4 sleep 20 19981
> 2017-04-18 19:11:41.285341 [CONSOLE] mod_commands.c:846 Timer Test: 5 sleep 20 20023
> 2017-04-18 19:11:41.305354 [CONSOLE] mod_commands.c:846 Timer Test: 6 sleep 20 19990
> 2017-04-18 19:11:41.325343 [CONSOLE] mod_commands.c:846 Timer Test: 7 sleep 20 20011
> 2017-04-18 19:11:41.345343 [CONSOLE] mod_commands.c:846 Timer Test: 8 sleep 20 19978
> 2017-04-18 19:11:41.365339 [CONSOLE] mod_commands.c:846 Timer Test: 9 sleep 20 20020
> 2017-04-18 19:11:41.385347 [CONSOLE] mod_commands.c:846 Timer Test: 10 sleep 20 19984
> 2017-04-18 19:11:41.405335 [CONSOLE] mod_commands.c:846 Timer Test: 11 sleep 20 20010
> 2017-04-18 19:11:41.425347 [CONSOLE] mod_commands.c:846 Timer Test: 12 sleep 20 19990
> 2017-04-18 19:11:41.445336 [CONSOLE] mod_commands.c:846 Timer Test: 13 sleep 20 20014
> 2017-04-18 19:11:41.465337 [CONSOLE] mod_commands.c:846 Timer Test: 14 sleep 20 19983
> 2017-04-18 19:11:41.485344 [CONSOLE] mod_commands.c:846 Timer Test: 15 sleep 20 20028
> 2017-04-18 19:11:41.505372 [CONSOLE] mod_commands.c:846 Timer Test: 16 sleep 20 19988
> 2017-04-18 19:11:41.525370 [CONSOLE] mod_commands.c:846 Timer Test: 17 sleep 20 20038
> 2017-04-18 19:11:41.545372 [CONSOLE] mod_commands.c:846 Timer Test: 18 sleep 20 19968
> 2017-04-18 19:11:41.565369 [CONSOLE] mod_commands.c:846 Timer Test: 19 sleep 20 20024
> 2017-04-18 19:11:41.585372 [CONSOLE] mod_commands.c:846 Timer Test: 20 sleep 20 19982
> 2017-04-18 19:11:41.605312 [CONSOLE] mod_commands.c:846 Timer Test: 21 sleep 20 19966
> 2017-04-18 19:11:41.625345 [CONSOLE] mod_commands.c:846 Timer Test: 22 sleep 20 20007
> 2017-04-18 19:11:41.645357 [CONSOLE] mod_commands.c:846 Timer Test: 23 sleep 20 20035
> 2017-04-18 19:11:41.665372 [CONSOLE] mod_commands.c:846 Timer Test: 24 sleep 20 19992
> 2017-04-18 19:11:41.685369 [CONSOLE] mod_commands.c:846 Timer Test: 25 sleep 20 20020
> 2017-04-18 19:11:41.705376 [CONSOLE] mod_commands.c:846 Timer Test: 26 sleep 20 19984
> 2017-04-18 19:11:41.725375 [CONSOLE] mod_commands.c:846 Timer Test: 27 sleep 20 20022
> 2017-04-18 19:11:41.745370 [CONSOLE] mod_commands.c:846 Timer Test: 28 sleep 20 20093
> 2017-04-18 19:11:41.765370 [CONSOLE] mod_commands.c:846 Timer Test: 29 sleep 20 19902
> 2017-04-18 19:11:41.785367 [CONSOLE] mod_commands.c:846 Timer Test: 30 sleep 20 19974
> 2017-04-18 19:11:41.805341 [CONSOLE] mod_commands.c:846 Timer Test: 31 sleep 20 19996
> 2017-04-18 19:11:41.825329 [CONSOLE] mod_commands.c:846 Timer Test: 32 sleep 20 20081
> 2017-04-18 19:11:41.845368 [CONSOLE] mod_commands.c:846 Timer Test: 33 sleep 20 19953
> 2017-04-18 19:11:41.865376 [CONSOLE] mod_commands.c:846 Timer Test: 34 sleep 20 20105
> 2017-04-18 19:11:41.885376 [CONSOLE] mod_commands.c:846 Timer Test: 35 sleep 20 19967
> 2017-04-18 19:11:41.905350 [CONSOLE] mod_commands.c:846 Timer Test: 36 sleep 20 19882
> 2017-04-18 19:11:41.925347 [CONSOLE] mod_commands.c:846 Timer Test: 37 sleep 20 20021
> 2017-04-18 19:11:41.945378 [CONSOLE] mod_commands.c:846 Timer Test: 38 sleep 20 20007
> 2017-04-18 19:11:41.965332 [CONSOLE] mod_commands.c:846 Timer Test: 39 sleep 20 19983
> 2017-04-18 19:11:41.985333 [CONSOLE] mod_commands.c:846 Timer Test: 40 sleep 20 19967
> 2017-04-18 19:11:42.005327 [CONSOLE] mod_commands.c:846 Timer Test: 41 sleep 20 20001
> 2017-04-18 19:11:42.025337 [CONSOLE] mod_commands.c:846 Timer Test: 42 sleep 20 20008
> 2017-04-18 19:11:42.045334 [CONSOLE] mod_commands.c:846 Timer Test: 43 sleep 20 20005
> 2017-04-18 19:11:42.065360 [CONSOLE] mod_commands.c:846 Timer Test: 44 sleep 20 20018
> 2017-04-18 19:11:42.085372 [CONSOLE] mod_commands.c:846 Timer Test: 45 sleep 20 20039
> 2017-04-18 19:11:42.105377 [CONSOLE] mod_commands.c:846 Timer Test: 46 sleep 20 19978
> 2017-04-18 19:11:42.125370 [CONSOLE] mod_commands.c:846 Timer Test: 47 sleep 20 20015
> 2017-04-18 19:11:42.145372 [CONSOLE] mod_commands.c:846 Timer Test: 48 sleep 20 19980
> 2017-04-18 19:11:42.165376 [CONSOLE] mod_commands.c:846 Timer Test: 49 sleep 20 20027
> 2017-04-18 19:11:42.185368 [CONSOLE] mod_commands.c:846 Timer Test: 50 sleep 20 19969
> 
> Be sure to enable softtimer in switch.conf.xml
> timer_test was far less table without it.
> 
> David Ponzone  Direction Technique
> email: david.ponzone at ipeva.fr <mailto:david.ponzone at ipeva.fr>
> tel:      01 74 03 18 97
> gsm:   06 66 98 76 34
> 
> Service Client IPeva
> tel:      0811 46 26 26
> www.ipeva.fr <blocked::http://www.ipeva.fr/>  -   www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/>
> 
> Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
> 
> 
> 
> 
>> Le 18 avr. 2017 à 18:56, Anthony Minessale <anthony.minessale at gmail.com <mailto:anthony.minessale at gmail.com>> a écrit :
>> 
>> In general when a thread about performance or vm turns to lots of theories.  This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole.  To the contrary, asking about timer test was a good start!
>> 
>> 
>> 
>> 
>> On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike at jerris.com <mailto:mike at jerris.com>> wrote:
>> Superstition comment was in regards to tc malloc, not you.
>> 
>> 
>>> On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone at gmail.com <mailto:david.ponzone at gmail.com>> wrote:
>>> 
>> 
>>> Anthony,
>>> 
>>> Could you elaborate a little bit more on why I am being superstitious ?
>>> Is timer_test command obsolete ?
>>> Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
>>> 
>> 
>>>> Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale at gmail.com <mailto:anthony.minessale at gmail.com>> a écrit :
>>>> 
>> 
>>>> I believe we may be stumbling into superstition at this point.
>>>> 
>> 
>>>> On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone at gmail.com <mailto:david.ponzone at gmail.com>> wrote:
>> 
>>>> Did you run the timer_test command ?
>> 
>>>>> Le 14 avr. 2017 à 23:04, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> a écrit :
>>>>> 
>>>>> The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier  m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. 
>>>>> 
>>>>> Model   vCPU    Mem (GiB)       SSD Storage (GB)        
>>>>> m3.large        2       7.5     1 x 32  
>>>>> m3.2xlarge      8       30      2 x 80  
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>>>> Robert,
>>>>> 
>>>>> While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
>>>>> 
>>>>> Bilal,
>>>>> 
>>>>> I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
>>>>> 
>>>>> Best,
>>>>> Colin
>>>>> 
>>>>> On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky at ets.org <mailto:rmundkowsky at ets.org>> wrote:
>>>>> Michael <>
>>>>>  
>>>>> 
>>>>> Just curious, why so many problems with VMs?
>>>>> 
>>>>>  
>>>>> 
>>>>> I would think most applications need real time clocks that provide consistent valid data?
>>>>> 
>>>>>  
>>>>> 
>>>>> Bilal,
>>>>> 
>>>>>  
>>>>> 
>>>>> you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
>>>>> 
>>>>>  
>>>>> 
>>>>> Robert
>>>>> 
>>>>>  
>>>>> 
>>>>> From: freeswitch-users-bounces at lists.freeswitch.org <mailto:freeswitch-users-bounces at lists.freeswitch.org> [mailto:freeswitch-users-bounces at lists.freeswitch.org <mailto:freeswitch-users-bounces at lists.freeswitch.org>] On Behalf Of Michael Jerris
>>>>> Sent: Friday, April 14, 2017 4:25 PM
>>>>> To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org <mailto:freeswitch-users at lists.freeswitch.org>>
>>>>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
>>>>> 
>>>>>  
>>>>> 
>>>>> can you reproduce the same issue on real hardware?  We’ve seen all kinds of weird timing issues that could account for this running on aws.
>>>>> 
>>>>>  
>>>>> 
>>>>> On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> wrote:
>>>>> 
>>>>>  
>>>>> 
>>>>> Its an AWS m3.2xlarge instance.
>>>>> 
>>>>>  
>>>>> 
>>>>> On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike at jerris.com <mailto:mike at jerris.com>> wrote:
>>>>> 
>>>>> Real hardware or VM?
>>>>> 
>>>>> > On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal at rgate-systems.com <mailto:bilal at rgate-systems.com>> wrote:
>>>>> >
>>>>> > I have been struggling with an issue for almost 2 weeks.
>>>>> >
>>>>> > Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
>>>>> >
>>>>> > We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
>>>>> >
>>>>> > I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
>>>>> >
>>>>> > I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
>>>>> 
>>>>>  
>>>>> 
>> 
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>> 
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>> http://www.cluecon.com <http://www.cluecon.com/>
>> 
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> -- 
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>> 
>> ☞ http://freeswitch.org/ <http://freeswitch.org/>  ☞ http://cluecon.com/ <http://cluecon.com/>  ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH>
>> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
>> 
>> ClueCon Weekly Development Call 
>> ☎ sip:888 at conference.freeswitch.org <mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900 
>> 
>> https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services: 
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>> 
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>> 
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services: 
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170419/b51f396c/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list