[Freeswitch-users] Thread failure with low CPU utilization?

Tue Dec 15 14:16:23 MSK 2015

Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Vladislav Ivanov
Sent: Tuesday, 15 December, 2015 2:15
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Thread failure with low CPU utilization?

Hey guys,

Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
*          soft     nproc          32768
*          hard     nproc          32768

was not propagated on freeswitch user...

Added new line:
freeswitch          soft     nproc          32768
freeswitch          hard     nproc          32768

and it worked fine from that.
Thanks to you all!

2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale at gmail.com<mailto:anthony.minessale at gmail.com>>:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.

On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm at sofnet.com<mailto:kworm at sofnet.com>> wrote:
This post
http://unix.stackexchange.com/questions/47595/linux-max-threads-count
has a simple program to check out how many threads can be created.  You
can run the test program as root and then as FS user and see if it's a
ulimit type of problem.

On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
> Is there a reason to think that CentOS 6 would cause threads to fail to
> spawn? This is basically just calling libapr, which is code from like 8
> years ago.
>
> Vladislav: This means it failed to create a thread (switch_thread_create
> in switch_apr.c). FS generates a ton of threads. The only time I’ve seen
> this was a long time ago on 32-bit Windows, since default stack of 1MB
> means you hit address space limits pretty quickly. But on 64-bit, you
> should be able to go all out.
>
> I’d start by adding some if’d printf’s to check the various things that
> switch_thread_create does and try to pinpoint what is failing. If the
> libapr call is failing, then you should be able to create a minimal
> program that just spawns threads and see if that one is able to get
> 1000+. From there I’d guess troubleshooting is gonna be the same as
> figuring out why pthread_create fails and that should be generic to your
> system. (Please post if you figure out why.)
>
> If other apps are able to spawn and keep thousands of threads and just
> FS isn’t then I guess you’re unlucky?
>
> -Michael
>
> *From:*freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>
> [mailto:freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of
> *Giovanni Maruzzelli
> *Sent:* Monday, 14 December, 2015 10:57
> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org<mailto:freeswitch-users at lists.freeswitch.org>>
> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
>
> centos 6 is not supported by FreeSWITCH 1.6 !
>
> -giovanni
>
> On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com<mailto:deforceczt at gmail.com>
> <mailto:deforceczt at gmail.com<mailto:deforceczt at gmail.com>>> wrote:
>
>     Centos 6,latest update. Via init script
>
>     On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org<mailto:krice at freeswitch.org>
>     <mailto:krice at freeswitch.org<mailto:krice at freeswitch.org>>> wrote:
>
>         A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
>
>         b)How are you starting FreeSWITCH? (eg: in the foreground or
>         background with an init script)
>
>         *From:*freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>
>         <mailto:freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>>
>         [mailto:freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>
>         <mailto:freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>>] *On
>         Behalf Of *Vladislav Ivanov
>         *Sent:* Monday, December 14, 2015 9:47 AM
>         *To:* freeswitch-users at lists.freeswitch.org<mailto:freeswitch-users at lists.freeswitch.org>
>         <mailto:freeswitch-users at lists.freeswitch.org<mailto:freeswitch-users at lists.freeswitch.org>>
>         *Subject:* [Freeswitch-users] Thread failure with low CPU
>         utilization?
>
>         Hello,
>
>         I have 1.6.5 Freeswitch and I'm trying to test how much
>         lines/cps it can handle on my test machine:
>
>         CPU: Intel(R) Xeon(R) CPU E5450  @ 3.00GHz x2 (8 cores, 3 GHz)
>         RAM: 32GB
>         SSD: 500 GB
>
>         I'm using call generator to check flood the freeswitch, but when
>         FS gets 1k sessions it sends
>         [CRIT] switch_core_session.c:1781 Thread Failure!
>         and the Star Wars quote... But the CPU load is less than 10%
>
>         Ulimit is as follows:
>
>         core file size          (blocks, -c) 0
>
>         data seg size           (kbytes, -d) unlimited
>
>         scheduling priority             (-e) 0
>
>         file size               (blocks, -f) unlimited
>
>         pending signals                 (-i) 128330
>
>         max locked memory       (kbytes, -l) 64
>
>         max memory size         (kbytes, -m) unlimited
>
>         open files                      (-n) 32768
>
>         pipe size            (512 bytes, -p) 8
>
>         POSIX message queues     (bytes, -q) 819200
>
>         real-time priority              (-r) 0
>
>         stack size              (kbytes, -s) 2048
>
>         cpu time               (seconds, -t) unlimited
>
>         max user processes              (-u) 32768
>
>         virtual memory          (kbytes, -v) unlimited
>
>         file locks                      (-x) unlimited
>
>
>         Server is physical, not virtual, but I dont have the same issue
>         with virtual freeswitch.
>
>         Here is the "status" before the session limit:
>
>         freeswitch at internal> status
>
>         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
>         milliseconds, 603 microseconds
>
>         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>         64bit) is ready
>
>         994 session(s) since startup
>
>         994 session(s) - peak 994, last 5min 994
>
>         40 session(s) per Sec out of max 200, peak 41, last 5min 41
>
>         10000 session(s) max
>
>         min idle cpu 10.00/96.77
>
>         Current Stack Size/Max 2048K/2048K
>
>         freeswitch at internal> status
>
>         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
>         milliseconds, 877 microseconds
>
>         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>         64bit) is ready
>
>         1003 session(s) since startup
>
>         1003 session(s) - peak 1003, last 5min 1003
>
>         40 session(s) per Sec out of max 200, peak 41, last 5min 41
>
>         993 session(s) max
>
>         min idle cpu 10.00/96.77
>
>         Current Stack Size/Max 2048K/2048K
>
>         freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
>         switch_core_session.c:1781 Thread Failure!
>
>         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
>         LUKE: I'm hit, but not bad.
>
>         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
>         LUKE'S VOICE: Artoo, see what you can do with it. Hang on back
>         there....
>
>         Green laserfire moves past the beeping little robot as his head
>         turns.  After a few beeps and a twist of his mechanical arm,
>
>         Artoo reduces the max sessions to 993 thus, saving the switch
>         from certain doom.
>
>         Any help with that? Really strange error...
>
>         _________________________________________________________________________
>         Professional FreeSWITCH Consulting Services:
>         consulting at freeswitch.org<mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org<mailto:consulting at freeswitch.org>>
>         http://www.freeswitchsolutions.com
>
>         Official FreeSWITCH Sites
>         http://www.freeswitch.org
>         http://confluence.freeswitch.org
>         http://www.cluecon.com
>
>         FreeSWITCH-users mailing list
>         FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
>         <mailto:FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>>
>         http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>         UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>         http://www.freeswitch.org
>
>
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org<mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org<mailto:consulting at freeswitch.org>>
>     http://www.freeswitchsolutions.com
>
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://confluence.freeswitch.org
>     http://www.cluecon.com
>
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
>     <mailto:FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
>
>
>
>
> --
>
> Sincerely,
>
> Giovanni Maruzzelli
> Cell : +39-347-2665618<tel:%2B39-347-2665618>
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org<mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org<mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900<tel:%2B19193869900>

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151215/7d7d3005/attachment-0001.html 