high load with ~200 fax calls despite low CPU utilization and low IO wait
Hi all, My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable. top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage. I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings. These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs. Has anyone else seen similar high load issues before? Many thanks, Adam Johnston
I'm running freeswitch also on a vm running centos 6.3 with fusionpbx. I can vouch for the sluggishness. What I did is expand the memory. Try expanding the memory to 4gb it should help but unfortunately if your on vm your always going to get sub par performance. On Aug 9, 2012 4:58 PM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
You should look to your VM solution provider for assistance(most have their own performance tools) traditional linux tools do not always provide proper load information inside of a VM. Also look at the load details on the host server to see where its straining generally when your VM is un responsive its due to he host. ~Mitch On Thu, Aug 9, 2012 at 2:33 PM, BookBag <asaad2@gmail.com> wrote:
I'm running freeswitch also on a vm running centos 6.3 with fusionpbx. I can vouch for the sluggishness. What I did is expand the memory. Try expanding the memory to 4gb it should help but unfortunately if your on vm your always going to get sub par performance.
On Aug 9, 2012 4:58 PM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi guys, i like to define a counter my script: nummer_pointer = argv[1]; api = freeswitch.API(); msg_pointerdb = api:execute("hash", "select/mailbox_pointer_msg/msg"); if msg_pointerdb == "nil" then msg_pointerdb_add = api:execute("hash", "insert/mailbox_pointer_msg/msg/1"); elseif msg_pointerdb > 200 then print(msg_pointerdb) else msg_pointerdb = api:execute("hash", "select/mailbox_pointer_msg/msg" ) pointer = msg_pointerdb + nummer_pointer msg_pointerdb_add = api:execute("hash", "insert/mailbox_pointer_msg/msg/" ..pointer ) end I become the error: 2012-08-10 01:41:35.499182 [ERR] mod_lua.cpp:198 /usr/local/freeswitch/scripts/t est-elseif.lua:6: attempt to compare number with string what is wrong please? thanks. m --------------- Du kannst mich jederzeit kostenlos per Festnetz erreichen unter: http://www.blindi.net/callback homepage: http://www.blindi.net blinde-misc mailingliste für blinde. anmeldung unter: http://www.blindi.net/mailman/listinfo/blinde-misc
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment. Steve
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both. If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out. Thanks, Adam On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org>wrote:
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment.
Steve
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet. On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.
If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.
Thanks, Adam
On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org>wrote:
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment.
Steve
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn't a high concern. Loading a few hundred channels doesn't seem to make the CPU go up very much. We haven't had any other issues with CentOS 6 in production for several months. And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven't had time to do it yet. -Michael From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Friday, August 10, 2012 1:25 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet. On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston@blimessaging.com<mailto:ajohnston@blimessaging.com>> wrote: The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both. If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out. Thanks, Adam On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org<mailto:steveu@coppice.org>> wrote: On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment. Steve _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com Join Us At ClueCon - Aug 7-9, 2012 FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com Join Us At ClueCon - Aug 7-9, 2012 FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I have every intention of replacing our CentOS 6 servers with CentOS 5.8 once I get this load average issue properly understood. I would note that I can reproduce this issue in both 6 and 5.8. Adam On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <mgg@giagnocavo.net>wrote:
On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn’t a high concern. Loading a few hundred channels doesn’t seem to make the CPU go up very much. We haven’t had any other issues with CentOS 6 in production for several months.****
** **
And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven’t had time to do it yet.****
** **
-Michael****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Friday, August 10, 2012 1:25 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait****
** **
Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet.****
On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:****
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.****
** **
If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.****
** **
Thanks,****
Adam****
** **
** **
On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org> wrote:****
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston ****
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment.
Steve****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If you consider this an issue.... Move it to Jira. On Aug 10, 2012 3:50 PM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:
I have every intention of replacing our CentOS 6 servers with CentOS 5.8 once I get this load average issue properly understood. I would note that I can reproduce this issue in both 6 and 5.8.
Adam
On Fri, Aug 10, 2012 at 3:33 PM, Michael Giagnocavo <mgg@giagnocavo.net>wrote:
On CentOS 6.2 we see one thread that is runaway full CPU, but with 16 threads it isn’t a high concern. Loading a few hundred channels doesn’t seem to make the CPU go up very much. We haven’t had any other issues with CentOS 6 in production for several months.****
** **
And yes, we plan on isolating it on a text machine and dumping and figuring out what is busted on that particular thread, just haven’t had time to do it yet.****
** **
-Michael****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Friday, August 10, 2012 1:25 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high load with ~200 fax calls despite low CPU utilization and low IO wait****
** **
Do not touch cent 6 with a 20 foot pole. Something is seriously wrong with it. We have not figured it out yet.****
On Aug 10, 2012 11:42 AM, "Adam Johnston" <ajohnston@blimessaging.com> wrote:****
The 200 faxes are sending, they attempt T.38 but fallback to audio so it's a mix of both.****
** **
If I log into a FreeSWITCH VM and launch 200 faxes from a script I don't see nearly the same level of CPU load, so I'm inclined to think it has something to do with the (many) event socket connections between my controller application and the VMs. I'm setting up a hardware FreeSWITCH server and writing a test harness to prove this out.****
** **
Thanks,****
Adam****
** **
** **
On Fri, Aug 10, 2012 at 11:53 AM, Steve Underwood <steveu@coppice.org> wrote:****
On 08/10/2012 04:55 AM, Adam Johnston wrote:
Hi all,
My FreeSWITCH setup works as follow: I have a service that launches and monitors faxes (via event sockets) on one of a few FreeSWITCH instances. These FreeSWITCH instances are running on dual-core, 2ghz CentOS 5.8 and CentOS 6.0 VMs. The issue is that once I get to ~200 simultaneous faxes on any one FreeSWITCH VM the load average shoots up (typically ~30, although I've seen it get much higher) and the VM becomes sluggish and occasionally unusable.
top and sar output confirms that CPU utilization is pretty low, as are IO wait and RAM usage.
I'm using a git head from 2012-02-06, although I see the same problem if update to a more current head. On the most recent head I tried (2012-08-06) the issue is worse, and crashes with a backtrace similar to this Jira, http://jira.freeswitch.org/browse/FS-2893 despite my compiling Lua and including the FreeSWITCH bindings.
These issues occur on both the CentOS 5.8 and 6.0 VMs. I'm in the process of loading 5.8 onto a physical machine and testing there. It's also worth nothing that this load issue does not occur when I launch faxes from a script on one of the VMs.
Has anyone else seen similar high load issues before?
Many thanks, Adam Johnston ****
What are these 200 FAXes doing? sending or receiving? Using audio or T.38? It makes quite a different to the load. Sending 200 faxes by audio at 14,400bps will keep a dual core machine busy. I'm not sure you can trust the CPU utilisation figures in a VM environment.
Steve****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Adam Johnston
-
Anthony Minessale
-
BookBag
-
Michael Giagnocavo
-
Mitch Capper
-
Steve Underwood
-
Thomas Hoellriegel