[Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait

[Anthony Minessale]

If you consider this an issue.... Move it to Jira.
 On Aug 10, 2012 3:50 PM, "Adam Johnston" <ajohnston at blimessaging.com>
wrote:

> I have every intention of replacing our CentOS 6 servers with CentOS 5.8
> once I get this load average issue properly understood. I would note that I
> can reproduce this issue in both 6 and 5.8.
>
> Adam
>
>
> On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <mgg at giagnocavo.net>wrote:
>
>>  On CentOS 6.2 we see one thread that is runaway full CPU, but with 16
>> threads it isn’t a high concern. Loading a few hundred channels doesn’t
>> seem to make the CPU go up very much. We haven’t had any other issues with
>> CentOS 6 in production for several months.****
>>
>> ** **
>>
>> And yes, we plan on isolating it on a text machine and dumping and
>> figuring out what is busted on that particular thread, just haven’t had
>> time to do it yet.****
>>
>> ** **
>>
>> -Michael****
>>
>> ** **
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Anthony
>> Minessale
>> *Sent:* Friday, August 10, 2012 1:25 PM
>> *To:* FreeSWITCH Users Help
>> *Subject:* Re: [Freeswitch-users] high load with ~200 fax calls despite
>> low CPU utilization and low IO wait****
>>
>> ** **
>>
>> Do not touch cent 6 with a 20 foot pole.   Something is seriously wrong
>> with it.  We have not figured it out yet.****
>>
>> On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston at blimessaging.com>
>> wrote:****
>>
>> The 200 faxes are sending, they attempt T.38 but fallback to audio so
>> it's a mix of both.****
>>
>> ** **
>>
>> If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't
>> see nearly the same level of CPU load, so I'm inclined to think it has
>> something to do with the (many) event socket connections between my
>> controller application and the VMs. I'm setting up a hardware FreeSWITCH
>> server and writing a test harness to prove this out.****
>>
>> ** **
>>
>> Thanks,****
>>
>> Adam****
>>
>> ** **
>>
>> ** **
>>
>> On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu at coppice.org>
>> wrote:****
>>
>> On 08/10/2012 04:55 AM, Adam Johnston wrote:
>> > Hi all,
>> >
>> > My FreeSWITCH setup works as follow: I have a service that launches
>> > and monitors faxes (via event sockets) on one of a few FreeSWITCH
>> > instances. These FreeSWITCH instances are running on dual-core, 2ghz
>> > CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200
>> > simultaneous faxes on any one FreeSWITCH VM the load average shoots up
>> > (typically ~30, although I've seen it get much higher) and the VM
>> > becomes sluggish and occasionally unusable.
>> >
>> > top and sar output confirms that CPU utilization is pretty low, as are
>> > IO wait and RAM usage.
>> >
>> > I'm using a git head from 2012-02-06, although I see the same problem
>> > if update to a more current head. On the most recent head I tried
>> > (2012-08-06) the issue is worse, and crashes with a backtrace similar
>> > to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my
>> > compiling Lua and including the FreeSWITCH bindings.
>> >
>> > These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the
>> > process of loading 5.8 onto a physical machine and testing there. It's
>> > also worth nothing that this load issue does not occur when I launch
>> > faxes from a script on one of the VMs.
>> >
>> > Has anyone else seen similar high load issues before?
>> >
>> > Many thanks,
>> > Adam Johnston
>> >****
>>
>> What are these 200 FAXes doing? sending or receiving? Using audio or
>> T.38? It makes quite a different to the load. Sending 200 faxes by audio
>> at 14,400bps will keep a dual core machine busy. I'm not sure you can
>> trust the CPU utilisation figures in a VM environment.
>>
>> Steve****
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> Join Us At ClueCon - Aug 7-9, 2012
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org****
>>
>> ** **
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> Join Us At ClueCon - Aug 7-9, 2012
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org****
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> Join Us At ClueCon - Aug 7-9, 2012
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> Join Us At ClueCon - Aug 7-9, 2012
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20120810/7ccc0a6a/attachment.html