[Freeswitch-users] Thread failure with low CPU utilization?
Anthony Minessale
anthony.minessale at gmail.com
Tue Dec 15 00:06:00 MSK 2015
Are you running FS as root or a regular user, you need to check the ulimits
of concurrent threads and some boxes have a hard limit of threads, its not
all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm at sofnet.com> wrote:
> This post
> http://unix.stackexchange.com/questions/47595/linux-max-threads-count
> has a simple program to check out how many threads can be created. You
> can run the test program as root and then as FS user and see if it's a
> ulimit type of problem.
>
> On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
> > Is there a reason to think that CentOS 6 would cause threads to fail to
> > spawn? This is basically just calling libapr, which is code from like 8
> > years ago.
> >
> > Vladislav: This means it failed to create a thread (switch_thread_create
> > in switch_apr.c). FS generates a ton of threads. The only time I’ve seen
> > this was a long time ago on 32-bit Windows, since default stack of 1MB
> > means you hit address space limits pretty quickly. But on 64-bit, you
> > should be able to go all out.
> >
> > I’d start by adding some if’d printf’s to check the various things that
> > switch_thread_create does and try to pinpoint what is failing. If the
> > libapr call is failing, then you should be able to create a minimal
> > program that just spawns threads and see if that one is able to get
> > 1000+. From there I’d guess troubleshooting is gonna be the same as
> > figuring out why pthread_create fails and that should be generic to your
> > system. (Please post if you figure out why.)
> >
> > If other apps are able to spawn and keep thousands of threads and just
> > FS isn’t then I guess you’re unlucky?
> >
> > -Michael
> >
> > *From:*freeswitch-users-bounces at lists.freeswitch.org
> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
> > *Giovanni Maruzzelli
> > *Sent:* Monday, 14 December, 2015 10:57
> > *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> > *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
> utilization?
> >
> > centos 6 is not supported by FreeSWITCH 1.6 !
> >
> > -giovanni
> >
> > On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com
> > <mailto:deforceczt at gmail.com>> wrote:
> >
> > Centos 6,latest update. Via init script
> >
> > On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org
> > <mailto:krice at freeswitch.org>> wrote:
> >
> > A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
> >
> > b)How are you starting FreeSWITCH? (eg: in the foreground or
> > background with an init script)
> >
> > *From:*freeswitch-users-bounces at lists.freeswitch.org
> > <mailto:freeswitch-users-bounces at lists.freeswitch.org>
> > [mailto:freeswitch-users-bounces at lists.freeswitch.org
> > <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On
> > Behalf Of *Vladislav Ivanov
> > *Sent:* Monday, December 14, 2015 9:47 AM
> > *To:* freeswitch-users at lists.freeswitch.org
> > <mailto:freeswitch-users at lists.freeswitch.org>
> > *Subject:* [Freeswitch-users] Thread failure with low CPU
> > utilization?
> >
> > Hello,
> >
> > I have 1.6.5 Freeswitch and I'm trying to test how much
> > lines/cps it can handle on my test machine:
> >
> > CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz)
> > RAM: 32GB
> > SSD: 500 GB
> >
> > I'm using call generator to check flood the freeswitch, but when
> > FS gets 1k sessions it sends
> > [CRIT] switch_core_session.c:1781 Thread Failure!
> > and the Star Wars quote... But the CPU load is less than 10%
> >
> > Ulimit is as follows:
> >
> > core file size (blocks, -c) 0
> >
> > data seg size (kbytes, -d) unlimited
> >
> > scheduling priority (-e) 0
> >
> > file size (blocks, -f) unlimited
> >
> > pending signals (-i) 128330
> >
> > max locked memory (kbytes, -l) 64
> >
> > max memory size (kbytes, -m) unlimited
> >
> > open files (-n) 32768
> >
> > pipe size (512 bytes, -p) 8
> >
> > POSIX message queues (bytes, -q) 819200
> >
> > real-time priority (-r) 0
> >
> > stack size (kbytes, -s) 2048
> >
> > cpu time (seconds, -t) unlimited
> >
> > max user processes (-u) 32768
> >
> > virtual memory (kbytes, -v) unlimited
> >
> > file locks (-x) unlimited
> >
> >
> > Server is physical, not virtual, but I dont have the same issue
> > with virtual freeswitch.
> >
> > Here is the "status" before the session limit:
> >
> > freeswitch at internal> status
> >
> > UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
> > milliseconds, 603 microseconds
> >
> > FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> > 64bit) is ready
> >
> > 994 session(s) since startup
> >
> > 994 session(s) - peak 994, last 5min 994
> >
> > 40 session(s) per Sec out of max 200, peak 41, last 5min 41
> >
> > 10000 session(s) max
> >
> > min idle cpu 10.00/96.77
> >
> > Current Stack Size/Max 2048K/2048K
> >
> > freeswitch at internal> status
> >
> > UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
> > milliseconds, 877 microseconds
> >
> > FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> > 64bit) is ready
> >
> > 1003 session(s) since startup
> >
> > 1003 session(s) - peak 1003, last 5min 1003
> >
> > 40 session(s) per Sec out of max 200, peak 41, last 5min 41
> >
> > 993 session(s) max
> >
> > min idle cpu 10.00/96.77
> >
> > Current Stack Size/Max 2048K/2048K
> >
> > freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
> > switch_core_session.c:1781 Thread Failure!
> >
> > 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
> > LUKE: I'm hit, but not bad.
> >
> > 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
> > LUKE'S VOICE: Artoo, see what you can do with it. Hang on back
> > there....
> >
> > Green laserfire moves past the beeping little robot as his head
> > turns. After a few beeps and a twist of his mechanical arm,
> >
> > Artoo reduces the max sessions to 993 thus, saving the switch
> > from certain doom.
> >
> > Any help with that? Really strange error...
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> > --
> >
> > Sincerely,
> >
> > Giovanni Maruzzelli
> > Cell : +39-347-2665618
> >
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151214/85f4a033/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list