[Freeswitch-users] Thread failure with low CPU utilization?

Mon Dec 14 22:08:23 MSK 2015

Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.

Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.

I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)

If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?

-Michael

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Giovanni Maruzzelli
Sent: Monday, 14 December, 2015 10:57
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] Thread failure with low CPU utilization?

centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni

On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com<mailto:deforceczt at gmail.com>> wrote:

Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org<mailto:krice at freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)

From: freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org> [mailto:freeswitch-users-bounces at lists.freeswitch.org<mailto:freeswitch-users-bounces at lists.freeswitch.org>] On Behalf Of Vladislav Ivanov
Sent: Monday, December 14, 2015 9:47 AM
To: freeswitch-users at lists.freeswitch.org<mailto:freeswitch-users at lists.freeswitch.org>
Subject: [Freeswitch-users] Thread failure with low CPU utilization?

Hello,

I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:

CPU: Intel(R) Xeon(R) CPU E5450  @ 3.00GHz x2 (8 cores, 3 GHz)
RAM: 32GB
SSD: 500 GB

I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends
[CRIT] switch_core_session.c:1781 Thread Failure!
and the Star Wars quote... But the CPU load is less than 10%

Ulimit is as follows:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128330
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 2048
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32768
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.

Here is the "status" before the session limit:
freeswitch at internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K

freeswitch at internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K

freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns.  After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.

Any help with that? Really strange error...

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Sincerely,

Giovanni Maruzzelli
Cell : +39-347-2665618
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151214/28b3adea/attachment-0001.html 