<div>In general when a thread about performance or vm turns to lots of theories. This is a science so we need facts to diagnose and its often not possible once we get this far down the rabbit hole. To the contrary, asking about timer test was a good start!</div><div><br></div><div><br></div><div><br></div><div><br><div class="gmail_quote"><div>On Tue, Apr 18, 2017 at 11:41 AM Michael Jerris <<a href="mailto:mike@jerris.com">mike@jerris.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Superstition comment was in regards to tc malloc, not you.<div><br><div><blockquote type="cite"></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div>On Apr 18, 2017, at 7:35 AM, David Ponzone <<a href="mailto:david.ponzone@gmail.com" target="_blank">david.ponzone@gmail.com</a>> wrote:</div><br class="m_-8579652609755085387Apple-interchange-newline"></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div>Anthony,</div><div><br></div>Could you elaborate a little bit more on why I am being superstitious ?<div>Is timer_test command obsolete ?</div><div>Or is it a way to emphase the fact that you won’t support FS on VM, anyway ?</div><div><br></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div>Le 17 avr. 2017 à 23:27, Anthony Minessale <<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>> a écrit :</div><br class="m_-8579652609755085387Apple-interchange-newline"></blockquote></div></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div>I believe we may be stumbling into superstition at this point.<div><br></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote">On Fri, Apr 14, 2017 at 6:03 PM, David Ponzone <span><<a href="mailto:david.ponzone@gmail.com" target="_blank">david.ponzone@gmail.com</a>></span> wrote:<br></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Did you run the timer_test command ?</div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><div class="m_-8579652609755085387h5"><div><blockquote type="cite"><div>Le 14 avr. 2017 à 23:04, Bilal Dar <<a href="mailto:bilal@rgate-systems.com" target="_blank">bilal@rgate-systems.com</a>> a écrit :</div><br class="m_-8579652609755085387m_7398230337600912797Apple-interchange-newline"><div><div>The issue triggered when I ran out of inodes on the server, even after freeing inodes things never went back to normal. I was running earlier m3.large and now moved to m3.2xlarge servers, CPU/memory utilization is negligible. <div><div><br></div><div><div>Model vCPU Mem (GiB) SSD Storage (GB) </div><div>m3.large 2 7.5 1 x 32 </div><div>m3.2xlarge 8 30 2 x 80 </div></div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 14, 2017 at 1:53 PM, Colin Morelli <span><<a href="mailto:colin.morelli@gmail.com" target="_blank">colin.morelli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div>Robert,<div><br></div><div>While I'd love to see VMs provide more stable ground for FS, it's simply not the best task for a VM. Virtual machines scale well specifically because you can overprovision them. It would not be nearly as cost-effective to run VMs if each instance had a guaranteed dedicated slice of hardware to operate on. While hypervisors are <i>very</i> good at task prioritization, they're not perfect. If the hypervisor can't schedule processor time when FS needs it because the CPU cores are momentarily taken on other tasks, there's not a whole lot FS can do. This is not an issue with just FS, but with all real-time applications. In most apps, even large clock skews and bad hypervisors schedulers can go completely unnoticed. If there's consistent 5-10ms every time you click to load a web page, you'd probably have no idea. If there's 5-10ms jitter every time you try to read 20ms of audio, you have <i>really bad</i> audio. Granted most skews are not that bad, but the effects are pronounced when you're dealing with data that's real-time in nature.</div><div><br></div><div>Bilal,</div><div><br></div><div>I have no idea what AMI you're running, but a very rough "ear test" has made me fairly confident that I can get better performance running AmazonLinux AMIs over Ubuntu (and probably many others). It wouldn't surprise me if AmazonLinux builds a custom kernel that has been tuned to run better on AWS hardware. I'd say it's at least worth a quick experiment.</div><div><br></div><div>Best,</div><div>Colin</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-8579652609755085387m_7398230337600912797gmail-h5">On Fri, Apr 14, 2017 at 4:34 PM, Mundkowsky, Robert <span><<a href="mailto:rmundkowsky@ets.org" target="_blank">rmundkowsky@ets.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div class="m_-8579652609755085387m_7398230337600912797gmail-h5">
<div lang="EN-US">
<div class="m_-8579652609755085387m_7398230337600912797gmail-m_-470338840461164024m_-5335504817902696112WordSection1"><p class="MsoNormal"><a name="m_-8579652609755085387_m_7398230337600912797_m_-470338840461164024_m_-5335504817902696112__MailEndCompose"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Michael<u></u><u></u></span></a></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p><p class="MsoNormal" style="text-indent:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Just curious, why so many problems with VMs?<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p><p class="MsoNormal" style="text-indent:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">I would think most applications need real time clocks that provide consistent valid data?<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Bilal,<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p><p class="MsoNormal" style="text-indent:0.5in"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">you might try a larger AWS instance to make sure your are getting 100% of the box; might help some.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Robert
<u></u><u></u></span></p>
</div><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(225,225,225);padding:3pt 0in 0in"><p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Michael Jerris<br>
<b>Sent:</b> Friday, April 14, 2017 4:25 PM<br>
<b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
<b>Subject:</b> Re: [Freeswitch-users] Choppy audio when conferencing 4+ participants<u></u><u></u></span></p>
</div>
</div><div><div class="m_-8579652609755085387m_7398230337600912797gmail-m_-470338840461164024h5"><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">can you reproduce the same issue on real hardware? We’ve seen all kinds of weird timing issues that could account for this running on aws.<u></u><u></u></p>
<div><p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div><p class="MsoNormal">On Apr 14, 2017, at 4:16 PM, Bilal Dar <<a href="mailto:bilal@rgate-systems.com" target="_blank">bilal@rgate-systems.com</a>> wrote:<u></u><u></u></p>
</div><p class="MsoNormal"><u></u> <u></u></p>
<div>
<div><p class="MsoNormal">Its an AWS <span style="font-size:10.5pt;font-family:arial,sans-serif;color:rgb(51,51,51)">m3.2xlarge instance.</span><u></u><u></u></p>
</div>
<div><p class="MsoNormal"><u></u> <u></u></p>
<div><p class="MsoNormal">On Fri, Apr 14, 2017 at 9:57 AM, Michael Jerris <<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><p class="MsoNormal" style="margin-bottom:12pt">Real hardware or VM?<br>
<br>
> On Apr 14, 2017, at 12:48 PM, Bilal Dar <<a href="mailto:bilal@rgate-systems.com" target="_blank">bilal@rgate-systems.com</a>> wrote:<br>
><br>
> I have been struggling with an issue for almost 2 weeks.<br>
><br>
> Our regular calls have no quality issue and looking RTCP statistics network conditions are perfect. We have normally on peak hr 60 calls and around 10 conferences.<br>
><br>
> We have noticed that when we have 2 conferences of 4 or 5 participants, audio starts breaking for the users who are on conference. Regular calls do not experience any quality degradation.<br>
><br>
> I upgraded the server to specs of 30Gig memory and 8 vCPU but still the issue exists. Common thing I have noticed even during off-peak hrs is that two 4+ participant call can cause the issue.<br>
><br>
> I have ruled out network & hardware. Last change I made was moved all users to G.711 from G.722. Now I am not sure what other steps I can take. Appreciate any suggestions.<u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"><u></u> </p></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div style="word-wrap:break-word"><div><div><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><div class="m_-8579652609755085387h5"></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a></blockquote></div></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ +19193869900 </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>