[Freeswitch-users] Choppy audio when conferencing 4+ participants

Tue Apr 25 18:38:02 MSD 2017

Umm, Not to do advertising here but we are a cloud vender that specializes in VoIP and other real time stuff J In fact we have been trying to work with the FS folks about having ready made FS templates, however it seems that we are all too busy to find a time that works for everyone. 

About running FS on VMs, in general running on a single server as a VM the only hypervisor that it seems to work great with is Hyper-V, for both either Linux or Windows deployments. This is because Hyper-V is a microkernel hypervisor, and not monolithic. However with a single server solution, it is still limited as to total workload on the same server.

On the other hand, if you’re talking about running FS on cloud infrastructure, not just a single server, then it runs great as a VM. We currently have like 3 customers that are doing major telco workloads as customers on our cloud.

If you need a highly available cloud based solution (cloud is not a place, but an architecture) then FS will run great in that environment, however you will need to shell out some bucks if you plan to set up your own private cloud for that.

If you want to do it in a hosted environment or at least give it a try, then check us out as just one possibility: http://www.xarix.net. 

Also, feel free to contact me directly to see if I can help you get your stuff running better.

--Dave

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Mundkowsky, Robert
Sent: Tuesday, April 18, 2017 4:19 PM
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants

Would be interesting if Cloud vendors had a RT Cloud offering and/or OS researchers/developers could work on possible changes to VM managers to handle RT better, because seems VMs are going to continue to be popular platform. Granted maybe this has already been done or offered by non-major vendors.

We are running on AWS. In general, it works pretty good. Sometimes it does not, but we have not spent time to try to figure out if issues are mainly due to network, client, and/or VMs.

Robert 

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Michael Jerris
Sent: Tuesday, April 18, 2017 4:00 PM
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants

It is certainly possible to run in a vm if done correctly, its just done incorrectly more than correctly.

On Apr 18, 2017, at 3:50 PM, Bipin Patel <bipin at xbipin.com> wrote:

Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM.
Correct me if I'm wrong
On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale at gmail.com> wrote:
In general when a thread about performance or vm turns to lots of theories.  This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole.  To the contrary, asking about timer test was a good start!

On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike at jerris.com> wrote:
Superstition comment was in regards to tc malloc, not you.

On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone at gmail.com> wrote:

Anthony,

Could you elaborate a little bit more on why I am being superstitious ?
Is timer_test command obsolete ?
Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?

Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale at gmail.com> a écrit :

I believe we may be stumbling into superstition at this point.

On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone at gmail.com> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal at rgate-systems.com> a écrit :

The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier  m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. 

Model   vCPU    Mem (GiB)       SSD Storage (GB)        
m3.large        2       7.5     1 x 32  
m3.2xlarge      8       30      2 x 80  

On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli at gmail.com> wrote:
Robert,

While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.

Bilal,

I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.

Best,
Colin

On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky at ets.org> wrote:
Michael

Just curious, why so many problems with VMs?

I would think most applications need real time clocks that provide consistent valid data?

Bilal,

you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.

Robert 

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Michael Jerris
Sent: Friday, April 14, 2017 4:25 PM
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants

can you reproduce the same issue on real hardware?  We’ve seen all kinds of weird timing issues that could account for this running on aws.

On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal at rgate-systems.com> wrote:

Its an AWS m3.2xlarge instance.

On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike at jerris.com> wrote:
Real hardware or VM?

> On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal at rgate-systems.com> wrote:
>
> I have been struggling with an issue for almost 2 weeks.
>
> Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
>
> We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
>
> I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
>
> I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.

This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.

Thank you for your compliance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170425/c1add7e6/attachment-0001.html 