Choppy audio when conferencing 4+ participants
I have been struggling with an issue for almost 2 weeks. Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences. We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation. I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue. I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions. Thanks
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
Its an AWS m3.2xlarge instance. On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote: Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
Michael Just curious, why so many problems with VMs? I would think most applications need real time clocks that provide consistent valid data? Bilal, you might try a larger AWS instance to make sure your are getting 100% of the box; might help some. Robert From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws. On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote: Its an AWS m3.2xlarge instance. On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com<mailto:mike@jerris.com>> wrote: Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
________________________________ This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance. ________________________________
On Fri, Apr 14, 2017 at 3:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Problems include: *) User error when configuring *) Resources available which can be unknown on a hosted vm platform. *) Countless other variables about the setup not known at this time. Everything discussed thus far is conjecture and supposition so there is not really any concrete subject to comment on. This is why we do not usually engage on performance and VM related subjects. We offer commercial assistance for matters such as this where it requires possible investigation and time spent to discover the source of an issue. consulting@freeswitch.org Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Robert, While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature. Bilal, I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment. Best, Colin On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80 On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Did you run the timer_test command ? David Ponzone Direction Technique email: david.ponzone@ipeva.fr <mailto:david.ponzone@ipeva.fr> tel: 01 74 03 18 97 gsm: 06 66 98 76 34 Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr <blocked::http://www.ipeva.fr/> - www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/> Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I believe we may be stumbling into superstition at this point. On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
David Ponzone Direction Technique email: david.ponzone@ipeva.fr tel: 01 74 03 18 97 gsm: 06 66 98 76 34
Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr - www.ipeva-studio.com
*Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. **IPeva** décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.*
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Il 17 apr 2017 23:28, "Anthony Minessale" <anthony.minessale@gmail.com> ha scritto: I believe we may be stumbling into superstition at this point. +1 -giovanni On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
David Ponzone Direction Technique email: david.ponzone@ipeva.fr tel: 01 74 03 18 97 gsm: 06 66 98 76 34
Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr - www.ipeva-studio.com
*Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. **IPeva** décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.*
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900> https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Anthony, Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ? David Ponzone Direction Technique email: david.ponzone@ipeva.fr <mailto:david.ponzone@ipeva.fr> tel: 01 74 03 18 97 gsm: 06 66 98 76 34 Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr <blocked::http://www.ipeva.fr/> - www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/> Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote: Did you run the timer_test command ?
David Ponzone Direction Technique email: david.ponzone@ipeva.fr <mailto:david.ponzone@ipeva.fr> tel: 01 74 03 18 97 gsm: 06 66 98 76 34
Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr <> - www.ipeva-studio.com <>
Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/>
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote: Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start! On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com> wrote:
Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org
wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Ah I feel better :) To the OP, Bilal: on a FS 1.5 running on KVM (running on a Dell R610), I have timer_test giving quite stable results: 2017-04-18 19:11:41.205316 [CONSOLE] mod_commands.c:846 Timer Test: 1 sleep 20 19994 2017-04-18 19:11:41.225341 [CONSOLE] mod_commands.c:846 Timer Test: 2 sleep 20 20001 2017-04-18 19:11:41.245338 [CONSOLE] mod_commands.c:846 Timer Test: 3 sleep 20 20020 2017-04-18 19:11:41.265341 [CONSOLE] mod_commands.c:846 Timer Test: 4 sleep 20 19981 2017-04-18 19:11:41.285341 [CONSOLE] mod_commands.c:846 Timer Test: 5 sleep 20 20023 2017-04-18 19:11:41.305354 [CONSOLE] mod_commands.c:846 Timer Test: 6 sleep 20 19990 2017-04-18 19:11:41.325343 [CONSOLE] mod_commands.c:846 Timer Test: 7 sleep 20 20011 2017-04-18 19:11:41.345343 [CONSOLE] mod_commands.c:846 Timer Test: 8 sleep 20 19978 2017-04-18 19:11:41.365339 [CONSOLE] mod_commands.c:846 Timer Test: 9 sleep 20 20020 2017-04-18 19:11:41.385347 [CONSOLE] mod_commands.c:846 Timer Test: 10 sleep 20 19984 2017-04-18 19:11:41.405335 [CONSOLE] mod_commands.c:846 Timer Test: 11 sleep 20 20010 2017-04-18 19:11:41.425347 [CONSOLE] mod_commands.c:846 Timer Test: 12 sleep 20 19990 2017-04-18 19:11:41.445336 [CONSOLE] mod_commands.c:846 Timer Test: 13 sleep 20 20014 2017-04-18 19:11:41.465337 [CONSOLE] mod_commands.c:846 Timer Test: 14 sleep 20 19983 2017-04-18 19:11:41.485344 [CONSOLE] mod_commands.c:846 Timer Test: 15 sleep 20 20028 2017-04-18 19:11:41.505372 [CONSOLE] mod_commands.c:846 Timer Test: 16 sleep 20 19988 2017-04-18 19:11:41.525370 [CONSOLE] mod_commands.c:846 Timer Test: 17 sleep 20 20038 2017-04-18 19:11:41.545372 [CONSOLE] mod_commands.c:846 Timer Test: 18 sleep 20 19968 2017-04-18 19:11:41.565369 [CONSOLE] mod_commands.c:846 Timer Test: 19 sleep 20 20024 2017-04-18 19:11:41.585372 [CONSOLE] mod_commands.c:846 Timer Test: 20 sleep 20 19982 2017-04-18 19:11:41.605312 [CONSOLE] mod_commands.c:846 Timer Test: 21 sleep 20 19966 2017-04-18 19:11:41.625345 [CONSOLE] mod_commands.c:846 Timer Test: 22 sleep 20 20007 2017-04-18 19:11:41.645357 [CONSOLE] mod_commands.c:846 Timer Test: 23 sleep 20 20035 2017-04-18 19:11:41.665372 [CONSOLE] mod_commands.c:846 Timer Test: 24 sleep 20 19992 2017-04-18 19:11:41.685369 [CONSOLE] mod_commands.c:846 Timer Test: 25 sleep 20 20020 2017-04-18 19:11:41.705376 [CONSOLE] mod_commands.c:846 Timer Test: 26 sleep 20 19984 2017-04-18 19:11:41.725375 [CONSOLE] mod_commands.c:846 Timer Test: 27 sleep 20 20022 2017-04-18 19:11:41.745370 [CONSOLE] mod_commands.c:846 Timer Test: 28 sleep 20 20093 2017-04-18 19:11:41.765370 [CONSOLE] mod_commands.c:846 Timer Test: 29 sleep 20 19902 2017-04-18 19:11:41.785367 [CONSOLE] mod_commands.c:846 Timer Test: 30 sleep 20 19974 2017-04-18 19:11:41.805341 [CONSOLE] mod_commands.c:846 Timer Test: 31 sleep 20 19996 2017-04-18 19:11:41.825329 [CONSOLE] mod_commands.c:846 Timer Test: 32 sleep 20 20081 2017-04-18 19:11:41.845368 [CONSOLE] mod_commands.c:846 Timer Test: 33 sleep 20 19953 2017-04-18 19:11:41.865376 [CONSOLE] mod_commands.c:846 Timer Test: 34 sleep 20 20105 2017-04-18 19:11:41.885376 [CONSOLE] mod_commands.c:846 Timer Test: 35 sleep 20 19967 2017-04-18 19:11:41.905350 [CONSOLE] mod_commands.c:846 Timer Test: 36 sleep 20 19882 2017-04-18 19:11:41.925347 [CONSOLE] mod_commands.c:846 Timer Test: 37 sleep 20 20021 2017-04-18 19:11:41.945378 [CONSOLE] mod_commands.c:846 Timer Test: 38 sleep 20 20007 2017-04-18 19:11:41.965332 [CONSOLE] mod_commands.c:846 Timer Test: 39 sleep 20 19983 2017-04-18 19:11:41.985333 [CONSOLE] mod_commands.c:846 Timer Test: 40 sleep 20 19967 2017-04-18 19:11:42.005327 [CONSOLE] mod_commands.c:846 Timer Test: 41 sleep 20 20001 2017-04-18 19:11:42.025337 [CONSOLE] mod_commands.c:846 Timer Test: 42 sleep 20 20008 2017-04-18 19:11:42.045334 [CONSOLE] mod_commands.c:846 Timer Test: 43 sleep 20 20005 2017-04-18 19:11:42.065360 [CONSOLE] mod_commands.c:846 Timer Test: 44 sleep 20 20018 2017-04-18 19:11:42.085372 [CONSOLE] mod_commands.c:846 Timer Test: 45 sleep 20 20039 2017-04-18 19:11:42.105377 [CONSOLE] mod_commands.c:846 Timer Test: 46 sleep 20 19978 2017-04-18 19:11:42.125370 [CONSOLE] mod_commands.c:846 Timer Test: 47 sleep 20 20015 2017-04-18 19:11:42.145372 [CONSOLE] mod_commands.c:846 Timer Test: 48 sleep 20 19980 2017-04-18 19:11:42.165376 [CONSOLE] mod_commands.c:846 Timer Test: 49 sleep 20 20027 2017-04-18 19:11:42.185368 [CONSOLE] mod_commands.c:846 Timer Test: 50 sleep 20 19969 Be sure to enable softtimer in switch.conf.xml timer_test was far less table without it. David Ponzone Direction Technique email: david.ponzone@ipeva.fr <mailto:david.ponzone@ipeva.fr> tel: 01 74 03 18 97 gsm: 06 66 98 76 34 Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr <blocked::http://www.ipeva.fr/> - www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/> Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
Le 18 avr. 2017 à 18:56, Anthony Minessale <anthony.minessale@gmail.com> a écrit :
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!
On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote: Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I think that version wouldn’t have been using timerfd. Using timerfd SHOULD be way more stable and less cpu, if its not, that would indicate an issue with the vm of some sort.
On Apr 18, 2017, at 1:15 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Ah I feel better :)
To the OP, Bilal:
on a FS 1.5 running on KVM (running on a Dell R610), I have timer_test giving quite stable results:
2017-04-18 19:11:41.205316 [CONSOLE] mod_commands.c:846 Timer Test: 1 sleep 20 19994 2017-04-18 19:11:41.225341 [CONSOLE] mod_commands.c:846 Timer Test: 2 sleep 20 20001 2017-04-18 19:11:41.245338 [CONSOLE] mod_commands.c:846 Timer Test: 3 sleep 20 20020 2017-04-18 19:11:41.265341 [CONSOLE] mod_commands.c:846 Timer Test: 4 sleep 20 19981 2017-04-18 19:11:41.285341 [CONSOLE] mod_commands.c:846 Timer Test: 5 sleep 20 20023 2017-04-18 19:11:41.305354 [CONSOLE] mod_commands.c:846 Timer Test: 6 sleep 20 19990 2017-04-18 19:11:41.325343 [CONSOLE] mod_commands.c:846 Timer Test: 7 sleep 20 20011 2017-04-18 19:11:41.345343 [CONSOLE] mod_commands.c:846 Timer Test: 8 sleep 20 19978 2017-04-18 19:11:41.365339 [CONSOLE] mod_commands.c:846 Timer Test: 9 sleep 20 20020 2017-04-18 19:11:41.385347 [CONSOLE] mod_commands.c:846 Timer Test: 10 sleep 20 19984 2017-04-18 19:11:41.405335 [CONSOLE] mod_commands.c:846 Timer Test: 11 sleep 20 20010 2017-04-18 19:11:41.425347 [CONSOLE] mod_commands.c:846 Timer Test: 12 sleep 20 19990 2017-04-18 19:11:41.445336 [CONSOLE] mod_commands.c:846 Timer Test: 13 sleep 20 20014 2017-04-18 19:11:41.465337 [CONSOLE] mod_commands.c:846 Timer Test: 14 sleep 20 19983 2017-04-18 19:11:41.485344 [CONSOLE] mod_commands.c:846 Timer Test: 15 sleep 20 20028 2017-04-18 19:11:41.505372 [CONSOLE] mod_commands.c:846 Timer Test: 16 sleep 20 19988 2017-04-18 19:11:41.525370 [CONSOLE] mod_commands.c:846 Timer Test: 17 sleep 20 20038 2017-04-18 19:11:41.545372 [CONSOLE] mod_commands.c:846 Timer Test: 18 sleep 20 19968 2017-04-18 19:11:41.565369 [CONSOLE] mod_commands.c:846 Timer Test: 19 sleep 20 20024 2017-04-18 19:11:41.585372 [CONSOLE] mod_commands.c:846 Timer Test: 20 sleep 20 19982 2017-04-18 19:11:41.605312 [CONSOLE] mod_commands.c:846 Timer Test: 21 sleep 20 19966 2017-04-18 19:11:41.625345 [CONSOLE] mod_commands.c:846 Timer Test: 22 sleep 20 20007 2017-04-18 19:11:41.645357 [CONSOLE] mod_commands.c:846 Timer Test: 23 sleep 20 20035 2017-04-18 19:11:41.665372 [CONSOLE] mod_commands.c:846 Timer Test: 24 sleep 20 19992 2017-04-18 19:11:41.685369 [CONSOLE] mod_commands.c:846 Timer Test: 25 sleep 20 20020 2017-04-18 19:11:41.705376 [CONSOLE] mod_commands.c:846 Timer Test: 26 sleep 20 19984 2017-04-18 19:11:41.725375 [CONSOLE] mod_commands.c:846 Timer Test: 27 sleep 20 20022 2017-04-18 19:11:41.745370 [CONSOLE] mod_commands.c:846 Timer Test: 28 sleep 20 20093 2017-04-18 19:11:41.765370 [CONSOLE] mod_commands.c:846 Timer Test: 29 sleep 20 19902 2017-04-18 19:11:41.785367 [CONSOLE] mod_commands.c:846 Timer Test: 30 sleep 20 19974 2017-04-18 19:11:41.805341 [CONSOLE] mod_commands.c:846 Timer Test: 31 sleep 20 19996 2017-04-18 19:11:41.825329 [CONSOLE] mod_commands.c:846 Timer Test: 32 sleep 20 20081 2017-04-18 19:11:41.845368 [CONSOLE] mod_commands.c:846 Timer Test: 33 sleep 20 19953 2017-04-18 19:11:41.865376 [CONSOLE] mod_commands.c:846 Timer Test: 34 sleep 20 20105 2017-04-18 19:11:41.885376 [CONSOLE] mod_commands.c:846 Timer Test: 35 sleep 20 19967 2017-04-18 19:11:41.905350 [CONSOLE] mod_commands.c:846 Timer Test: 36 sleep 20 19882 2017-04-18 19:11:41.925347 [CONSOLE] mod_commands.c:846 Timer Test: 37 sleep 20 20021 2017-04-18 19:11:41.945378 [CONSOLE] mod_commands.c:846 Timer Test: 38 sleep 20 20007 2017-04-18 19:11:41.965332 [CONSOLE] mod_commands.c:846 Timer Test: 39 sleep 20 19983 2017-04-18 19:11:41.985333 [CONSOLE] mod_commands.c:846 Timer Test: 40 sleep 20 19967 2017-04-18 19:11:42.005327 [CONSOLE] mod_commands.c:846 Timer Test: 41 sleep 20 20001 2017-04-18 19:11:42.025337 [CONSOLE] mod_commands.c:846 Timer Test: 42 sleep 20 20008 2017-04-18 19:11:42.045334 [CONSOLE] mod_commands.c:846 Timer Test: 43 sleep 20 20005 2017-04-18 19:11:42.065360 [CONSOLE] mod_commands.c:846 Timer Test: 44 sleep 20 20018 2017-04-18 19:11:42.085372 [CONSOLE] mod_commands.c:846 Timer Test: 45 sleep 20 20039 2017-04-18 19:11:42.105377 [CONSOLE] mod_commands.c:846 Timer Test: 46 sleep 20 19978 2017-04-18 19:11:42.125370 [CONSOLE] mod_commands.c:846 Timer Test: 47 sleep 20 20015 2017-04-18 19:11:42.145372 [CONSOLE] mod_commands.c:846 Timer Test: 48 sleep 20 19980 2017-04-18 19:11:42.165376 [CONSOLE] mod_commands.c:846 Timer Test: 49 sleep 20 20027 2017-04-18 19:11:42.185368 [CONSOLE] mod_commands.c:846 Timer Test: 50 sleep 20 19969
Be sure to enable softtimer in switch.conf.xml timer_test was far less table without it.
David Ponzone Direction Technique email: david.ponzone@ipeva.fr <mailto:david.ponzone@ipeva.fr> tel: 01 74 03 18 97 gsm: 06 66 98 76 34
Service Client IPeva tel: 0811 46 26 26 www.ipeva.fr <blocked::http://www.ipeva.fr/> - www.ipeva-studio.com <blocked::http://www.ipeva-studio.com/>
Ce message et toutes les pièces jointes sont confidentiels et établis à l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisée est interdite. Tout message électronique est susceptible d'altération. IPeva décline toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié. Si vous n'êtes pas destinataire de ce message, merci de le détruire immédiatement et d'avertir l'expéditeur.
Le 18 avr. 2017 à 18:56, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> a écrit :
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!
On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote: Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://confluence.freeswitch.org <http://confluence.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://www.freeswitch.org <http://www.freeswitch.org/> -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ <http://freeswitch.org/> ☞ http://cluecon.com/ <http://cluecon.com/> ☞ http://twitter.com/FreeSWITCH <http://twitter.com/FreeSWITCH> ☞ irc.freenode.net <http://irc.freenode.net/> #freeswitch ☞ http://freeswitch.org/g+ <http://freeswitch.org/g+>
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s <https://www.youtube.com/watch?v=9XXgW34t40s> https://www.youtube.com/watch?v=NLaDpGQuZDA <https://www.youtube.com/watch?v=NLaDpGQuZDA> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM. Correct me if I'm wrong On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale@gmail.com> wrote:
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!
On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com> wrote:
Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org
wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
---------- _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
It is certainly possible to run in a vm if done correctly, its just done incorrectly more than correctly.
On Apr 18, 2017, at 3:50 PM, Bipin Patel <bipin@xbipin.com> wrote:
Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM.
Correct me if I'm wrong
On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale@gmail.com> wrote:
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!
On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote: Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com <mailto:david.ponzone@gmail.com>> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com <mailto:colin.morelli@gmail.com>> wrote: Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org <mailto:rmundkowsky@ets.org>> wrote: Michael <>
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com <mailto:mike@jerris.com>> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com <mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
Would be interesting if Cloud vendors had a RT Cloud offering and/or OS researchers/developers could work on possible changes to VM managers to handle RT better, because seems VMs are going to continue to be popular platform. Granted maybe this has already been done or offered by non-major vendors. We are running on AWS. In general, it works pretty good. Sometimes it does not, but we have not spent time to try to figure out if issues are mainly due to network, client, and/or VMs. Robert From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Tuesday, April 18, 2017 4:00 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants It is certainly possible to run in a vm if done correctly, its just done incorrectly more than correctly. On Apr 18, 2017, at 3:50 PM, Bipin Patel <bipin@xbipin.com<mailto:bipin@xbipin.com>> wrote: Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM. Correct me if I'm wrong On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> wrote: In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start! On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com<mailto:mike@jerris.com>> wrote: Superstition comment was in regards to tc malloc, not you. On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com<mailto:david.ponzone@gmail.com>> wrote: Anthony, Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ? Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>> a écrit : I believe we may be stumbling into superstition at this point. On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com<mailto:david.ponzone@gmail.com>> wrote: Did you run the timer_test command ? Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> a écrit : The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80 On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com<mailto:colin.morelli@gmail.com>> wrote: Robert, While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature. Bilal, I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment. Best, Colin On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org<mailto:rmundkowsky@ets.org>> wrote: Michael Just curious, why so many problems with VMs? I would think most applications need real time clocks that provide consistent valid data? Bilal, you might try a larger AWS instance to make sure your are getting 100% of the box; might help some. Robert From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws. On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote: Its an AWS m3.2xlarge instance. On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com<mailto:mike@jerris.com>> wrote: Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
________________________________ This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance. ________________________________
Umm, Not to do advertising here but we are a cloud vender that specializes in VoIP and other real time stuff J In fact we have been trying to work with the FS folks about having ready made FS templates, however it seems that we are all too busy to find a time that works for everyone. About running FS on VMs, in general running on a single server as a VM the only hypervisor that it seems to work great with is Hyper-V, for both either Linux or Windows deployments. This is because Hyper-V is a microkernel hypervisor, and not monolithic. However with a single server solution, it is still limited as to total workload on the same server. On the other hand, if you’re talking about running FS on cloud infrastructure, not just a single server, then it runs great as a VM. We currently have like 3 customers that are doing major telco workloads as customers on our cloud. If you need a highly available cloud based solution (cloud is not a place, but an architecture) then FS will run great in that environment, however you will need to shell out some bucks if you plan to set up your own private cloud for that. If you want to do it in a hosted environment or at least give it a try, then check us out as just one possibility: http://www.xarix.net. Also, feel free to contact me directly to see if I can help you get your stuff running better. --Dave From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Mundkowsky, Robert Sent: Tuesday, April 18, 2017 4:19 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants Would be interesting if Cloud vendors had a RT Cloud offering and/or OS researchers/developers could work on possible changes to VM managers to handle RT better, because seems VMs are going to continue to be popular platform. Granted maybe this has already been done or offered by non-major vendors. We are running on AWS. In general, it works pretty good. Sometimes it does not, but we have not spent time to try to figure out if issues are mainly due to network, client, and/or VMs. Robert From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Tuesday, April 18, 2017 4:00 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants It is certainly possible to run in a vm if done correctly, its just done incorrectly more than correctly. On Apr 18, 2017, at 3:50 PM, Bipin Patel <bipin@xbipin.com> wrote: Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM. Correct me if I'm wrong On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale@gmail.com> wrote: In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start! On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com> wrote: Superstition comment was in regards to tc malloc, not you. On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote: Anthony, Could you elaborate a little bit more on why I am being superstitious ? Is timer_test command obsolete ? Or is it a way to emphase the fact that you won’t support FS on VM, anyway ? Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit : I believe we may be stumbling into superstition at this point. On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote: Did you run the timer_test command ? Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit : The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80 On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote: Robert, While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature. Bilal, I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment. Best, Colin On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote: Michael Just curious, why so many problems with VMs? I would think most applications need real time clocks that provide consistent valid data? Bilal, you might try a larger AWS instance to make sure your are getting 100% of the box; might help some. Robert From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws. On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote: Its an AWS m3.2xlarge instance. On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote: Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance.
do you give the real-time VM's priority, or some guaranteed amount of CPU power? On Tue, Apr 25, 2017 at 4:38 PM, Dave Kompel <drk@drkngs.net> wrote:
Umm, Not to do advertising here but we are a cloud vender that specializes in VoIP and other real time stuff J In fact we have been trying to work with the FS folks about having ready made FS templates, however it seems that we are all too busy to find a time that works for everyone.
About running FS on VMs, in general running on a single server as a VM the only hypervisor that it seems to work great with is Hyper-V, for both either Linux or Windows deployments. This is because Hyper-V is a microkernel hypervisor, and not monolithic. However with a single server solution, it is still limited as to total workload on the same server.
On the other hand, if you’re talking about running FS on cloud infrastructure, not just a single server, then it runs great as a VM. We currently have like 3 customers that are doing major telco workloads as customers on our cloud.
If you need a highly available cloud based solution (cloud is not a place, but an architecture) then FS will run great in that environment, however you will need to shell out some bucks if you plan to set up your own private cloud for that.
If you want to do it in a hosted environment or at least give it a try, then check us out as just one possibility: http://www.xarix.net.
Also, feel free to contact me directly to see if I can help you get your stuff running better.
--Dave
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Mundkowsky, Robert Sent: Tuesday, April 18, 2017 4:19 PM
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
Would be interesting if Cloud vendors had a RT Cloud offering and/or OS researchers/developers could work on possible changes to VM managers to handle RT better, because seems VMs are going to continue to be popular platform. Granted maybe this has already been done or offered by non-major vendors.
We are running on AWS. In general, it works pretty good. Sometimes it does not, but we have not spent time to try to figure out if issues are mainly due to network, client, and/or VMs.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Tuesday, April 18, 2017 4:00 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
It is certainly possible to run in a vm if done correctly, its just done incorrectly more than correctly.
On Apr 18, 2017, at 3:50 PM, Bipin Patel <bipin@xbipin.com> wrote:
Just to add my experience, I have a windows server running as VM on a dedicated server hosted with iweb, they call it smart server and I have been using freeswitch on it from a long time now and everything works perfect audio wise so I guess this has something to do with Linux and the kernel run by Amazon on their VM.
Correct me if I'm wrong
On April 18, 2017 8:58:00 PM Anthony Minessale <anthony.minessale@gmail.com> wrote:
In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!
On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <mike@jerris.com> wrote:
Superstition comment was in regards to tc malloc, not you.
On Apr 18, 2017, at 7:35 AM, David Ponzone <david.ponzone@gmail.com> wrote:
Anthony,
Could you elaborate a little bit more on why I am being superstitious ?
Is timer_test command obsolete ?
Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?
Le 17 avr. 2017 à 23:27, Anthony Minessale <anthony.minessale@gmail.com> a écrit :
I believe we may be stumbling into superstition at this point.
On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <david.ponzone@gmail.com> wrote:
Did you run the timer_test command ?
Le 14 avr. 2017 à 23:04, Bilal Dar <bilal@rgate-systems.com> a écrit :
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB)
m3.large 2 7.5 1 x 32
m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best,
Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
________________________________
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance.
________________________________
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
HI, We have already run Freeswitch (git master version) several month on china . With an real server have 2 physical E5 2403V2 CPU (4core per cpu ,no HT) + 32G Ram,no video, no transcode,just proxy the stream , we can provide about 2000 sessions ( rtp stream) to our customer, audio quality guaranteed. We also have server with FreeSwitch installed on Aliyun (an chinese cloud server provider ), which based on KVM technology. Conference function is also tested, audio quality is good when using PCM codec, participants <= 10, streams is mixed. Based on our experience , plz DO NOT GIVE REAL-TIME PRIORITY TO Freeswitch . not only cloud server, but also physical server ,if you want maximize throughput. Many Cloud Server share the CPU time and I/O resources . "Realtime" cann't be guaranteed. And many cloud server have very low Packet forwarding rate . Maybe only 200 k pps. In real world , we always run Freeswitch with normal priority ,just renice it a little higher than other process. in our situation ,it's the best. Raymond At 2017-04-26 05:16:12, "Stanislav Sinyagin" <ssinyagin@gmail.com> wrote:
do you give the real-time VM's priority, or some guaranteed amount of CPU power?
On Tue, Apr 25, 2017 at 4:38 PM, Dave Kompel <drk@drkngs.net> wrote:
Umm, Not to do advertising here but we are a cloud vender that specializes in VoIP and other real time stuff J In fact we have been trying to work with the FS folks about having ready made FS templates, however it seems that we are all too busy to find a time that works for everyone.
I am going to try low latency driver and change frequency https://wiki.freeswitch.org/wiki/Amazon_EC2#Updating_Kernel_Timer_to_1000HZ Any other great ideas for the weekend. On Fri, Apr 14, 2017 at 2:04 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The best advice is Amazon Linux. /b On Fri, Apr 14, 2017 at 6:11 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I am going to try low latency driver and change frequency
https://wiki.freeswitch.org/wiki/Amazon_EC2#Updating_ Kernel_Timer_to_1000HZ
Any other great ideas for the weekend.
On Fri, Apr 14, 2017 at 2:04 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org
wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/ options/freeswitch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com Book a phone call (CST) <https://freeswitch.com/appointment> Allison prompts for FreeSWITCH: *https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *Skype:*briankwest
I had a simillar issue, I fixed using tcmalloc Le 14 avr. 2017 7:15 PM, "Brian West" <brian@freeswitch.org> a écrit :
The best advice is Amazon Linux.
/b
On Fri, Apr 14, 2017 at 6:11 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I am going to try low latency driver and change frequency
https://wiki.freeswitch.org/wiki/Amazon_EC2#Updating_Kernel_ Timer_to_1000HZ
Any other great ideas for the weekend.
On Fri, Apr 14, 2017 at 2:04 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible.
Model vCPU Mem (GiB) SSD Storage (GB) m3.large 2 7.5 1 x 32 m3.2xlarge 8 30 2 x 80
On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Robert,
While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are *very* good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have *really bad* audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.
Bilal,
I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.
Best, Colin
On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert < rmundkowsky@ets.org> wrote:
Michael
Just curious, why so many problems with VMs?
I would think most applications need real time clocks that provide consistent valid data?
Bilal,
you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.
Robert
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, April 14, 2017 4:25 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants
can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.
On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
Its an AWS m3.2xlarge instance.
On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com> wrote:
Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
------------------------------
This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited.
Thank you for your compliance. ------------------------------
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
*Brian West* brian@freeswitch.org
*Twitter: @FreeSWITCH , @briankwest*
http://www.freeswitchbook.com http://www.freeswitchcookbook.com
Book a phone call (CST) <https://freeswitch.com/appointment>
Allison prompts for FreeSWITCH:
*https://www.gofundme.com/allison-prompts-for-freeswitch* <https://www.gofundme.com/allison-prompts-for-freeswitch>
Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch>
*T:*+19184209001 <(918)%20420-9001> | *F:*+19184209002 <(918)%20420-9002> | *M:*+1918424WEST (9378) *Skype:*briankwest
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks Colin. I figured that was the case, but it is surprising how many things work without a good precise clock. Although might be most non-RT protocols expect slews and such. Robert From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Colin Morelli Sent: Friday, April 14, 2017 4:53 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants Robert, While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are very good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have really bad audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature. Bilal, I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment. Best, Colin On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <rmundkowsky@ets.org<mailto:rmundkowsky@ets.org>> wrote: Michael Just curious, why so many problems with VMs? I would think most applications need real time clocks that provide consistent valid data? Bilal, you might try a larger AWS instance to make sure your are getting 100% of the box; might help some. Robert From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Friday, April 14, 2017 4:25 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> Subject: Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws. On Apr 14, 2017, at 4:16 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote: Its an AWS m3.2xlarge instance. On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <mike@jerris.com<mailto:mike@jerris.com>> wrote: Real hardware or VM?
On Apr 14, 2017, at 12:48 PM, Bilal Dar <bilal@rgate-systems.com<mailto:bilal@rgate-systems.com>> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
________________________________ This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance. ________________________________ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ________________________________ This e-mail and any files transmitted with it may contain privileged or confidential information. It is solely for use by the individual for whom it is intended, even if addressed incorrectly. If you received this e-mail in error, please notify the sender; do not disclose, copy, distribute, or take any action in reliance on the contents of this information; and delete it from your system. Any other use of this e-mail is prohibited. Thank you for your compliance. ________________________________
as discussed many times already, there are several factors that affect audio quality in conferences or playback of audio files (regular bridged calls are less sensitive to this). 1) Less often than #2, but sometimes you don't get the clock that is precise enough. Conferencing and audio playback require a good reference clock, and some virtualized environments don't provide it. As noted somewhere in the wiki, by default AWS provides a kernel with really bad clock frequency, so you ought to change the kernel. 2) Any public VM hosting is a best-effort service, and you are very likely to have noisy neighbors. They all produce spikes in CPU load, and your VM is by no means prioritized against the others. So, your FS needs to compose an RTP frame and send it on-time, but it simply doesn't get the CPU cycles when it needs them. This leads to rather high jitter, and it's totally unpredictable. It may also depend on time of day or day of week, and you get totally different quality measurements with 5 minutes interval. So, for any kind of IVR or conferencing server, you ought to have physical hardware, or a hypervisor under your control, so that you allocate a guaranteed CPU resource to your FreeSWITCH machine. There's actually a quite broad choice of physical hardware hosting, so you can place your media servers on physical boxes, while leaving your application logic at your beloved AWS. The physical hardware doesn't have to be big: without transcoding, an average Celeron server can handle dozens of conference legs without problems. It's also quite easy to estimate your CPU capacity in advance, before sending production traffic: you create a conference, one leg goes to your ear, then few dozens of legs go to a silent sink on another server, and one leg is playing back some reference audio. Then you increase the number of participants until you start hearing quality impairments. There are also commercial tools for measuring voice quality, if you intend to run quality checks periodically. cheers, stanislav On Fri, Apr 14, 2017 at 6:48 PM, Bilal Dar <bilal@rgate-systems.com> wrote:
I have been struggling with an issue for almost 2 weeks.
Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.
We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.
I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.
I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.
Thanks
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (13)
-
Anthony Minessale
-
Bilal Dar
-
Bipin Patel
-
Brian West
-
Colin Morelli
-
Dave Kompel
-
David Ponzone
-
Giovanni Maruzzelli
-
Luis Daniel Lucio Quiroz
-
Michael Jerris
-
Mundkowsky, Robert
-
Raymond
-
Stanislav Sinyagin