[Freeswitch-users] Thread failure with low CPU utilization?

Anthony Minessale anthony.minessale at gmail.com
Tue Dec 15 00:06:00 MSK 2015


Are you running FS as root or a regular user, you need to check the ulimits
of concurrent threads and some boxes have a hard limit of threads, its not
all tied to CPU usage.


On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm at sofnet.com> wrote:

> This post
> http://unix.stackexchange.com/questions/47595/linux-max-threads-count
> has a simple program to check out how many threads can be created.  You
> can run the test program as root and then as FS user and see if it's a
> ulimit type of problem.
>
> On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
> > Is there a reason to think that CentOS 6 would cause threads to fail to
> > spawn? This is basically just calling libapr, which is code from like 8
> > years ago.
> >
> > Vladislav: This means it failed to create a thread (switch_thread_create
> > in switch_apr.c). FS generates a ton of threads. The only time I’ve seen
> > this was a long time ago on 32-bit Windows, since default stack of 1MB
> > means you hit address space limits pretty quickly. But on 64-bit, you
> > should be able to go all out.
> >
> > I’d start by adding some if’d printf’s to check the various things that
> > switch_thread_create does and try to pinpoint what is failing. If the
> > libapr call is failing, then you should be able to create a minimal
> > program that just spawns threads and see if that one is able to get
> > 1000+. From there I’d guess troubleshooting is gonna be the same as
> > figuring out why pthread_create fails and that should be generic to your
> > system. (Please post if you figure out why.)
> >
> > If other apps are able to spawn and keep thousands of threads and just
> > FS isn’t then I guess you’re unlucky?
> >
> > -Michael
> >
> > *From:*freeswitch-users-bounces at lists.freeswitch.org
> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
> > *Giovanni Maruzzelli
> > *Sent:* Monday, 14 December, 2015 10:57
> > *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> > *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
> utilization?
> >
> > centos 6 is not supported by FreeSWITCH 1.6 !
> >
> > -giovanni
> >
> > On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com
> > <mailto:deforceczt at gmail.com>> wrote:
> >
> >     Centos 6,latest update. Via init script
> >
> >     On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org
> >     <mailto:krice at freeswitch.org>> wrote:
> >
> >         A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
> >
> >         b)How are you starting FreeSWITCH? (eg: in the foreground or
> >         background with an init script)
> >
> >         *From:*freeswitch-users-bounces at lists.freeswitch.org
> >         <mailto:freeswitch-users-bounces at lists.freeswitch.org>
> >         [mailto:freeswitch-users-bounces at lists.freeswitch.org
> >         <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On
> >         Behalf Of *Vladislav Ivanov
> >         *Sent:* Monday, December 14, 2015 9:47 AM
> >         *To:* freeswitch-users at lists.freeswitch.org
> >         <mailto:freeswitch-users at lists.freeswitch.org>
> >         *Subject:* [Freeswitch-users] Thread failure with low CPU
> >         utilization?
> >
> >         Hello,
> >
> >         I have 1.6.5 Freeswitch and I'm trying to test how much
> >         lines/cps it can handle on my test machine:
> >
> >         CPU: Intel(R) Xeon(R) CPU E5450  @ 3.00GHz x2 (8 cores, 3 GHz)
> >         RAM: 32GB
> >         SSD: 500 GB
> >
> >         I'm using call generator to check flood the freeswitch, but when
> >         FS gets 1k sessions it sends
> >         [CRIT] switch_core_session.c:1781 Thread Failure!
> >         and the Star Wars quote... But the CPU load is less than 10%
> >
> >         Ulimit is as follows:
> >
> >         core file size          (blocks, -c) 0
> >
> >         data seg size           (kbytes, -d) unlimited
> >
> >         scheduling priority             (-e) 0
> >
> >         file size               (blocks, -f) unlimited
> >
> >         pending signals                 (-i) 128330
> >
> >         max locked memory       (kbytes, -l) 64
> >
> >         max memory size         (kbytes, -m) unlimited
> >
> >         open files                      (-n) 32768
> >
> >         pipe size            (512 bytes, -p) 8
> >
> >         POSIX message queues     (bytes, -q) 819200
> >
> >         real-time priority              (-r) 0
> >
> >         stack size              (kbytes, -s) 2048
> >
> >         cpu time               (seconds, -t) unlimited
> >
> >         max user processes              (-u) 32768
> >
> >         virtual memory          (kbytes, -v) unlimited
> >
> >         file locks                      (-x) unlimited
> >
> >
> >         Server is physical, not virtual, but I dont have the same issue
> >         with virtual freeswitch.
> >
> >         Here is the "status" before the session limit:
> >
> >         freeswitch at internal> status
> >
> >         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
> >         milliseconds, 603 microseconds
> >
> >         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> >         64bit) is ready
> >
> >         994 session(s) since startup
> >
> >         994 session(s) - peak 994, last 5min 994
> >
> >         40 session(s) per Sec out of max 200, peak 41, last 5min 41
> >
> >         10000 session(s) max
> >
> >         min idle cpu 10.00/96.77
> >
> >         Current Stack Size/Max 2048K/2048K
> >
> >         freeswitch at internal> status
> >
> >         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
> >         milliseconds, 877 microseconds
> >
> >         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
> >         64bit) is ready
> >
> >         1003 session(s) since startup
> >
> >         1003 session(s) - peak 1003, last 5min 1003
> >
> >         40 session(s) per Sec out of max 200, peak 41, last 5min 41
> >
> >         993 session(s) max
> >
> >         min idle cpu 10.00/96.77
> >
> >         Current Stack Size/Max 2048K/2048K
> >
> >         freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
> >         switch_core_session.c:1781 Thread Failure!
> >
> >         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
> >         LUKE: I'm hit, but not bad.
> >
> >         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
> >         LUKE'S VOICE: Artoo, see what you can do with it. Hang on back
> >         there....
> >
> >         Green laserfire moves past the beeping little robot as his head
> >         turns.  After a few beeps and a twist of his mechanical arm,
> >
> >         Artoo reduces the max sessions to 993 thus, saving the switch
> >         from certain doom.
> >
> >         Any help with that? Really strange error...
> >
> >
>  _________________________________________________________________________
> >         Professional FreeSWITCH Consulting Services:
> >         consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >         http://www.freeswitchsolutions.com
> >
> >         Official FreeSWITCH Sites
> >         http://www.freeswitch.org
> >         http://confluence.freeswitch.org
> >         http://www.cluecon.com
> >
> >         FreeSWITCH-users mailing list
> >         FreeSWITCH-users at lists.freeswitch.org
> >         <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >         UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >         http://www.freeswitch.org
> >
> >
> >
>  _________________________________________________________________________
> >     Professional FreeSWITCH Consulting Services:
> >     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> >     http://www.freeswitchsolutions.com
> >
> >     Official FreeSWITCH Sites
> >     http://www.freeswitch.org
> >     http://confluence.freeswitch.org
> >     http://www.cluecon.com
> >
> >     FreeSWITCH-users mailing list
> >     FreeSWITCH-users at lists.freeswitch.org
> >     <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >     UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >     http://www.freeswitch.org
> >
> >
> >
> >
> > --
> >
> > Sincerely,
> >
> > Giovanni Maruzzelli
> > Cell : +39-347-2665618
> >
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/http://cluecon.com/http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151214/85f4a033/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list