I have every intention of replacing our CentOS 6 servers with CentOS 5.8 once I get this load average issue properly understood. I would note that I can reproduce this issue in both 6 and 5.8. Adam On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <mgg@giagnocavo.net>wrote:
On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn’t a high concern. Loading a few hundred channels doesn’t seem to make the CPU go up very much. We haven’t had any other issues with CentOS 6 in production for several months.****
** **
And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven’t had time to do it yet.****
** **
-Michael****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Friday, August 10, 2012 1:25 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait****
** **
Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet.****
On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:****
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.****
** **
If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.****
** **
Thanks,****
Adam****
** **
** **
On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org> wrote:****
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston ****
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment.
Steve****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org