I have every intention of replacing our CentOS 6 servers with CentOS 5.8 once I get this load average issue properly understood. I would note that I can reproduce this issue in both 6 and 5.8.<div><br><div>Adam<br><div><div>
</div></div>
<br><br><div class="gmail_quote">On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <span dir="ltr"><<a href="mailto:mgg@giagnocavo.net" target="_blank">mgg@giagnocavo.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn’t a high concern. Loading a few hundred channels doesn’t seem to make the
CPU go up very much. We haven’t had any other issues with CentOS 6 in production for several months.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven’t had time to do it yet.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-Michael<u></u><u></u></span></p>
<p class="MsoNormal"><a name="1391208860839312__MailEndCompose"></a><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>Anthony Minessale<br>
<b>Sent:</b> Friday, August 10, 2012 1:25 PM<br>
<b>To:</b> FreeSWITCH Users Help<br>
<b>Subject:</b> Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p>Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet.<u></u><u></u></p>
<div>
<p class="MsoNormal">On Aug 10, 2012 11:42 AM, "Adam Johnston" <<a href="mailto:ajohnston@blimessaging.com" target="_blank">ajohnston@blimessaging.com</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and
the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Adam<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <<a href="mailto:steveu@coppice.org" target="_blank">steveu@coppice.org</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On 08/10/2012 04:55 AM, Adam Johnston wrote:<br>
> Hi all,<br>
><br>
> My FreeSWITCH setup works as follow: I have a service that launches<br>
> and monitors faxes (via event sockets) on one of a few FreeSWITCH<br>
> instances. These FreeSWITCH instances are running on dual-core, 2ghz<br>
> CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200<br>
> simultaneous faxes on any one FreeSWITCH VM the load average shoots up<br>
> (typically ~30, although I've seen it get much higher) and the VM<br>
> becomes sluggish and occasionally unusable.<br>
><br>
> top and sar output confirms that CPU utilization is pretty low, as are<br>
> IO wait and RAM usage.<br>
><br>
> I'm using a git head from 2012-02-06, although I see the same problem<br>
> if update to a more current head. On the most recent head I tried<br>
> (2012-08-06) the issue is worse, and crashes with a backtrace similar<br>
> to this Jira, <a href="http://jira.freeswitch.org/browse/FS-2893" target="_blank">
http://jira.freeswitch.org/browse/FS-2893</a> despite my<br>
> compiling Lua and including the FreeSWITCH bindings.<br>
><br>
> These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the<br>
> process of loading 5.8 onto a physical machine and testing there. It's<br>
> also worth nothing that this load issue does not occur when I launch<br>
> faxes from a script on one of the VMs.<br>
><br>
> Has anyone else seen similar high load issues before?<br>
><br>
> Many thanks,<br>
> Adam Johnston<br>
><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">What are these 200 FAXes doing? sending or receiving? Using audio or<br>
T.38? It makes quite a different to the load. Sending 200 faxes by audio<br>
at 14,400bps will keep a dual core machine busy. I'm not sure you can<br>
trust the CPU utilisation figures in a VM environment.<br>
<span style="color:#888888"><br>
Steve</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
Join Us At ClueCon - Aug 7-9, 2012<br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
Join Us At ClueCon - Aug 7-9, 2012<br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</div>
</div></div></div>
</div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
Join Us At ClueCon - Aug 7-9, 2012<br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div>