[Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait

[Michael Giagnocavo]

On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn't a high concern. Loading a few hundred channels doesn't seem to make the CPU go up very much. We haven't had any other issues with CentOS 6 in production for several months.

And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven't had time to do it yet.

-Michael

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Anthony Minessale
Sent: Friday, August 10, 2012 1:25 PM
To: FreeSWITCH Users Help
Subject: Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait


Do not touch cent 6 with a 20 foot pole.   Something is seriously wrong with it.  We have not figured it out yet.
On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston at blimessaging.com<mailto:ajohnston at blimessaging.com>> wrote:
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.

If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.

Thanks,
Adam


On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu at coppice.org<mailto:steveu at coppice.org>> wrote:
On 08/10/2012 04:55 AM, Adam Johnston wrote:
> Hi all,
>
> My FreeSWITCH setup works as follow: I have a service that launches
> and monitors faxes (via event sockets) on one of a few FreeSWITCH
> instances. These FreeSWITCH instances are running on dual-core, 2ghz
> CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200
> simultaneous faxes on any one FreeSWITCH VM the load average shoots up
> (typically ~30, although I've seen it get much higher) and the VM
> becomes sluggish and occasionally unusable.
>
> top and sar output confirms that CPU utilization is pretty low, as are
> IO wait and RAM usage.
>
> I'm using a git head from 2012-02-06, although I see the same problem
> if update to a more current head. On the most recent head I tried
> (2012-08-06) the issue is worse, and crashes with a backtrace similar
> to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my
> compiling Lua and including the FreeSWITCH bindings.
>
> These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the
> process of loading 5.8 onto a physical machine and testing there. It's
> also worth nothing that this load issue does not occur when I launch
> faxes from a script on one of the VMs.
>
> Has anyone else seen similar high load issues before?
>
> Many thanks,
> Adam Johnston
>
What are these 200 FAXes doing? sending or receiving? Using audio or
T.38? It makes quite a different to the load. Sending 200 faxes by audio
at 14,400bps will keep a dual core machine busy. I'm not sure you can
trust the CPU utilisation figures in a VM environment.

Steve


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com




Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

Join Us At ClueCon - Aug 7-9, 2012

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com




Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

Join Us At ClueCon - Aug 7-9, 2012

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20120810/9797ef95/attachment-0001.html