[Freeswitch-users] Thread failure with low CPU utilization?
Kevin Wormington
kworm at sofnet.com
Mon Dec 14 22:30:11 MSK 2015
This post
http://unix.stackexchange.com/questions/47595/linux-max-threads-count
has a simple program to check out how many threads can be created. You
can run the test program as root and then as FS user and see if it's a
ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
> Is there a reason to think that CentOS 6 would cause threads to fail to
> spawn? This is basically just calling libapr, which is code from like 8
> years ago.
>
> Vladislav: This means it failed to create a thread (switch_thread_create
> in switch_apr.c). FS generates a ton of threads. The only time I’ve seen
> this was a long time ago on 32-bit Windows, since default stack of 1MB
> means you hit address space limits pretty quickly. But on 64-bit, you
> should be able to go all out.
>
> I’d start by adding some if’d printf’s to check the various things that
> switch_thread_create does and try to pinpoint what is failing. If the
> libapr call is failing, then you should be able to create a minimal
> program that just spawns threads and see if that one is able to get
> 1000+. From there I’d guess troubleshooting is gonna be the same as
> figuring out why pthread_create fails and that should be generic to your
> system. (Please post if you figure out why.)
>
> If other apps are able to spawn and keep thousands of threads and just
> FS isn’t then I guess you’re unlucky?
>
> -Michael
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
> *Giovanni Maruzzelli
> *Sent:* Monday, 14 December, 2015 10:57
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
>
> centos 6 is not supported by FreeSWITCH 1.6 !
>
> -giovanni
>
> On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com
> <mailto:deforceczt at gmail.com>> wrote:
>
> Centos 6,latest update. Via init script
>
> On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org
> <mailto:krice at freeswitch.org>> wrote:
>
> A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
>
> b)How are you starting FreeSWITCH? (eg: in the foreground or
> background with an init script)
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org
> <mailto:freeswitch-users-bounces at lists.freeswitch.org>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org
> <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On
> Behalf Of *Vladislav Ivanov
> *Sent:* Monday, December 14, 2015 9:47 AM
> *To:* freeswitch-users at lists.freeswitch.org
> <mailto:freeswitch-users at lists.freeswitch.org>
> *Subject:* [Freeswitch-users] Thread failure with low CPU
> utilization?
>
> Hello,
>
> I have 1.6.5 Freeswitch and I'm trying to test how much
> lines/cps it can handle on my test machine:
>
> CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz)
> RAM: 32GB
> SSD: 500 GB
>
> I'm using call generator to check flood the freeswitch, but when
> FS gets 1k sessions it sends
> [CRIT] switch_core_session.c:1781 Thread Failure!
> and the Star Wars quote... But the CPU load is less than 10%
>
> Ulimit is as follows:
>
> core file size (blocks, -c) 0
>
> data seg size (kbytes, -d) unlimited
>
> scheduling priority (-e) 0
>
> file size (blocks, -f) unlimited
>
> pending signals (-i) 128330
>
> max locked memory (kbytes, -l) 64
>
> max memory size (kbytes, -m) unlimited
>
> open files (-n) 32768
>
> pipe size (512 bytes, -p) 8
>
> POSIX message queues (bytes, -q) 819200
>
> real-time priority (-r) 0
>
> stack size (kbytes, -s) 2048
>
> cpu time (seconds, -t) unlimited
>
> max user processes (-u) 32768
>
> virtual memory (kbytes, -v) unlimited
>
> file locks (-x) unlimited
>
>
> Server is physical, not virtual, but I dont have the same issue
> with virtual freeswitch.
>
> Here is the "status" before the session limit:
>
> freeswitch at internal> status
>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
> milliseconds, 603 microseconds
>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> 64bit) is ready
>
> 994 session(s) since startup
>
> 994 session(s) - peak 994, last 5min 994
>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41
>
> 10000 session(s) max
>
> min idle cpu 10.00/96.77
>
> Current Stack Size/Max 2048K/2048K
>
> freeswitch at internal> status
>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
> milliseconds, 877 microseconds
>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> 64bit) is ready
>
> 1003 session(s) since startup
>
> 1003 session(s) - peak 1003, last 5min 1003
>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41
>
> 993 session(s) max
>
> min idle cpu 10.00/96.77
>
> Current Stack Size/Max 2048K/2048K
>
> freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
> switch_core_session.c:1781 Thread Failure!
>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
> LUKE: I'm hit, but not bad.
>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
> LUKE'S VOICE: Artoo, see what you can do with it. Hang on back
> there....
>
> Green laserfire moves past the beeping little robot as his head
> turns. After a few beeps and a twist of his mechanical arm,
>
> Artoo reduces the max sessions to 993 thus, saving the switch
> from certain doom.
>
> Any help with that? Really strange error...
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
>
> --
>
> Sincerely,
>
> Giovanni Maruzzelli
> Cell : +39-347-2665618
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list