[Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait

[Adam Johnston]

I have every intention of replacing our CentOS 6 servers with CentOS 5.8
once I get this load average issue properly understood. I would note that I
can reproduce this issue in both 6 and 5.8.

Adam


On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <mgg at giagnocavo.net>wrote:

>  On CentOS 6.2 we see one thread that is runaway full CPU, but with 16
> threads it isn’t a high concern. Loading a few hundred channels doesn’t
> seem to make the CPU go up very much. We haven’t had any other issues with
> CentOS 6 in production for several months.****
>
> ** **
>
> And yes, we plan on isolating it on a text machine and dumping and
> figuring out what is busted on that particular thread, just haven’t had
> time to do it yet.****
>
> ** **
>
> -Michael****
>
> ** **
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Anthony
> Minessale
> *Sent:* Friday, August 10, 2012 1:25 PM
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high load with ~200 fax calls despite
> low CPU utilization and low IO wait****
>
> ** **
>
> Do not touch cent 6 with a 20 foot pole.   Something is seriously wrong
> with it.  We have not figured it out yet.****
>
> On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston at blimessaging.com>
> wrote:****
>
> The 200 faxes are sending, they attempt T.38 but fallback to audio so it's
> a mix of both.****
>
> ** **
>
> If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't
> see nearly the same level of CPU load, so I'm inclined to think it has
> something to do with the (many) event socket connections between my
> controller application and the VMs. I'm setting up a hardware FreeSWITCH
> server and writing a test harness to prove this out.****
>
> ** **
>
> Thanks,****
>
> Adam****
>
> ** **
>
> ** **
>
> On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu at coppice.org>
> wrote:****
>
> On 08/10/2012 04:55 AM, Adam Johnston wrote:
> > Hi all,
> >
> > My FreeSWITCH setup works as follow: I have a service that launches
> > and monitors faxes (via event sockets) on one of a few FreeSWITCH
> > instances. These FreeSWITCH instances are running on dual-core, 2ghz
> > CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200
> > simultaneous faxes on any one FreeSWITCH VM the load average shoots up
> > (typically ~30, although I've seen it get much higher) and the VM
> > becomes sluggish and occasionally unusable.
> >
> > top and sar output confirms that CPU utilization is pretty low, as are
> > IO wait and RAM usage.
> >
> > I'm using a git head from 2012-02-06, although I see the same problem
> > if update to a more current head. On the most recent head I tried
> > (2012-08-06) the issue is worse, and crashes with a backtrace similar
> > to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my
> > compiling Lua and including the FreeSWITCH bindings.
> >
> > These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the
> > process of loading 5.8 onto a physical machine and testing there. It's
> > also worth nothing that this load issue does not occur when I launch
> > faxes from a script on one of the VMs.
> >
> > Has anyone else seen similar high load issues before?
> >
> > Many thanks,
> > Adam Johnston
> >****
>
> What are these 200 FAXes doing? sending or receiving? Using audio or
> T.38? It makes quite a different to the load. Sending 200 faxes by audio
> at 14,400bps will keep a dual core machine busy. I'm not sure you can
> trust the CPU utilisation figures in a VM environment.
>
> Steve****
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> Join Us At ClueCon - Aug 7-9, 2012
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> ** **
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> Join Us At ClueCon - Aug 7-9, 2012
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> Join Us At ClueCon - Aug 7-9, 2012
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20120810/5013fd0d/attachment.html