Hello, I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine: CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10% Ulimit is as follows: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128330 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 2048 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Server is physical, not virtual, but I dont have the same issue with virtual freeswitch. Here is the "status" before the session limit: freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 994 session(s) since startup 994 session(s) - peak 994, last 5min 994 40 session(s) per Sec out of max 200, peak 41, last 5min 41 10000 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 1003 session(s) since startup 1003 session(s) - peak 1003, last 5min 1003 40 session(s) per Sec out of max 200, peak 41, last 5min 41 993 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure! 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad. 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 993 thus, saving the switch from certain doom. Any help with that? Really strange error...
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc) b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script) From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Vladislav Ivanov Sent: Monday, December 14, 2015 9:47 AM To: freeswitch-users@lists.freeswitch.org Subject: [Freeswitch-users] Thread failure with low CPU utilization? Hello, I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine: CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10% Ulimit is as follows: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128330 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 2048 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Server is physical, not virtual, but I dont have the same issue with virtual freeswitch. Here is the "status" before the session limit: freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 994 session(s) since startup 994 session(s) - peak 994, last 5min 994 40 session(s) per Sec out of max 200, peak 41, last 5min 41 10000 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 1003 session(s) since startup 1003 session(s) - peak 1003, last 5min 1003 40 session(s) per Sec out of max 200, peak 41, last 5min 41 993 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure! 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad. 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 993 thus, saving the switch from certain doom. Any help with that? Really strange error...
Centos 6,latest update. Via init script On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
centos 6 is not supported by FreeSWITCH 1.6 ! -giovanni On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com> wrote:
Centos 6,latest update. Via init script On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely, Giovanni Maruzzelli Cell : +39-347-2665618
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago. Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out. I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.) If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky? -Michael From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Giovanni Maruzzelli Sent: Monday, 14 December, 2015 10:57 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Thread failure with low CPU utilization? centos 6 is not supported by FreeSWITCH 1.6 ! -giovanni On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com<mailto:deforceczt@gmail.com>> wrote: Centos 6,latest update. Via init script On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org<mailto:krice@freeswitch.org>> wrote: A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc) b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script) From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Vladislav Ivanov Sent: Monday, December 14, 2015 9:47 AM To: freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> Subject: [Freeswitch-users] Thread failure with low CPU utilization? Hello, I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine: CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10% Ulimit is as follows: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128330 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 2048 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Server is physical, not virtual, but I dont have the same issue with virtual freeswitch. Here is the "status" before the session limit: freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 994 session(s) since startup 994 session(s) - peak 994, last 5min 994 40 session(s) per Sec out of max 200, peak 41, last 5min 41 10000 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> status UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready 1003 session(s) since startup 1003 session(s) - peak 1003, last 5min 1003 40 session(s) per Sec out of max 200, peak 41, last 5min 41 993 session(s) max min idle cpu 10.00/96.77 Current Stack Size/Max 2048K/2048K freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure! 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad. 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 993 thus, saving the switch from certain doom. Any help with that? Really strange error... _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Sincerely, Giovanni Maruzzelli Cell : +39-347-2665618
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem. On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage. On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hey guys, Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line: * soft nproc 32768 * hard nproc 32768 was not propagated on freeswitch user... Added new line: freeswitch soft nproc 32768 freeswitch hard nproc 32768 and it worked fine from that. Thanks to you all! 2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com>:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Vladislav Ivanov Sent: Tuesday, 15 December, 2015 2:15 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Thread failure with low CPU utilization? Hey guys, Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line: * soft nproc 32768 * hard nproc 32768 was not propagated on freeswitch user... Added new line: freeswitch soft nproc 32768 freeswitch hard nproc 32768 and it worked fine from that. Thanks to you all! 2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com<mailto:anthony.minessale@gmail.com>>: Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage. On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com<mailto:kworm@sofnet.com>> wrote: This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem. On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com<mailto:deforceczt@gmail.com> <mailto:deforceczt@gmail.com<mailto:deforceczt@gmail.com>>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org<mailto:krice@freeswitch.org> <mailto:krice@freeswitch.org<mailto:krice@freeswitch.org>>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> <mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org> <mailto:freeswitch-users@lists.freeswitch.org<mailto:freeswitch-users@lists.freeswitch.org>> *Subject:* [Freeswitch-users] Thread failure with low CPU utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org<mailto:consulting@freeswitch.org>> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <mailto:FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org<mailto:consulting@freeswitch.org>> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> <mailto:FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618<tel:%2B39-347-2665618>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900<tel:%2B19193869900> https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
1000 threads is not really a show stopper =D That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth. On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg@giagnocavo.net> wrote:
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Tuesday, 15 December, 2015 2:15 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
Hey guys,
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
* soft nproc 32768
* hard nproc 32768
was not propagated on freeswitch user...
Added new line:
freeswitch soft nproc 32768
freeswitch hard nproc 32768
and it worked fine from that. Thanks to you all!
2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com
:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU
utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hey guys, I have a new problem that is happening with CPU load now. I have been testing freeswitch with sipp, and i'm encountering strange issue with same load. 1st time I test with 10 cps, 5 second duration calls, for just cps load. CPU idle is min idle cpu 10.00/84.67 Then I add CPS by 5, it should be enough power to be 15 cps on the machine when I have idle 84%... But my CPU goes heavily down, more and more down until it goes to 0 and spams: 2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session Anything I can do to debug or check why it is causing this? (Same is happening on Centos 6, Debian) OS: "Debian GNU/Linux 8 (jessie) FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) I'm kinda lost at this point... 2015-12-15 19:21 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com>:
1000 threads is not really a show stopper =D That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth.
On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg@giagnocavo.net> wrote:
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Tuesday, 15 December, 2015 2:15 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
Hey guys,
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
* soft nproc 32768
* hard nproc 32768
was not propagated on freeswitch user...
Added new line:
freeswitch soft nproc 32768
freeswitch hard nproc 32768
and it worked fine from that. Thanks to you all!
2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com
:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU
utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but when FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
More issues related to resource limits. Check your allowed max sessions and sps in your config and any other ulimits on the system that you may have too low. Its wise to have limits low enough that the calls start failing before your box does. On Tue, Dec 15, 2015 at 12:15 PM, Vladislav Ivanov <deforceczt@gmail.com> wrote:
Hey guys,
I have a new problem that is happening with CPU load now. I have been testing freeswitch with sipp, and i'm encountering strange issue with same load. 1st time I test with 10 cps, 5 second duration calls, for just cps load. CPU idle is min idle cpu 10.00/84.67 Then I add CPS by 5, it should be enough power to be 15 cps on the machine when I have idle 84%... But my CPU goes heavily down, more and more down until it goes to 0 and spams: 2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session
Anything I can do to debug or check why it is causing this? (Same is happening on Centos 6, Debian) OS: "Debian GNU/Linux 8 (jessie) FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit)
I'm kinda lost at this point...
2015-12-15 19:21 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com> :
1000 threads is not really a show stopper =D That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth.
On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg@giagnocavo.net> wrote:
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Tuesday, 15 December, 2015 2:15 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
Hey guys,
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
* soft nproc 32768
* hard nproc 32768
was not propagated on freeswitch user...
Added new line:
freeswitch soft nproc 32768
freeswitch hard nproc 32768
and it worked fine from that. Thanks to you all!
2015-12-14 23:06 GMT+02:00 Anthony Minessale < anthony.minessale@gmail.com>:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov < deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU
utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but
when
FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same issue with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
I'm using the systemd file from https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+1.6+Video Just calling calls with 10 cps, low duration, should it slowly use the cpu? It really slowly eats all CPU, then drops all calls, repeats again... Shouldn't it be 10% cpu, nothing more for the same cps, no logs writing. Just simple minimal. 2015-12-15 22:22 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com>:
More issues related to resource limits. Check your allowed max sessions and sps in your config and any other ulimits on the system that you may have too low.
Its wise to have limits low enough that the calls start failing before your box does.
On Tue, Dec 15, 2015 at 12:15 PM, Vladislav Ivanov <deforceczt@gmail.com> wrote:
Hey guys,
I have a new problem that is happening with CPU load now. I have been testing freeswitch with sipp, and i'm encountering strange issue with same load. 1st time I test with 10 cps, 5 second duration calls, for just cps load. CPU idle is min idle cpu 10.00/84.67 Then I add CPS by 5, it should be enough power to be 15 cps on the machine when I have idle 84%... But my CPU goes heavily down, more and more down until it goes to 0 and spams: 2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session
Anything I can do to debug or check why it is causing this? (Same is happening on Centos 6, Debian) OS: "Debian GNU/Linux 8 (jessie) FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit)
I'm kinda lost at this point...
2015-12-15 19:21 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com
:
1000 threads is not really a show stopper =D That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth.
On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg@giagnocavo.net> wrote:
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Tuesday, 15 December, 2015 2:15 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
Hey guys,
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
* soft nproc 32768
* hard nproc 32768
was not propagated on freeswitch user...
Added new line:
freeswitch soft nproc 32768
freeswitch hard nproc 32768
and it worked fine from that. Thanks to you all!
2015-12-14 23:06 GMT+02:00 Anthony Minessale < anthony.minessale@gmail.com>:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov < deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU
utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz) RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but
when
FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same
issue
with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there....
Green laserfire moves past the beeping little robot as his
head
turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Just to give more info about this, I had an issue with simultaneous calls and db connection. Seems that adding entries in limit was the bootle neck. ulimit command was used as well. Luis Daniel Lucio Quiroz CISSP, CISM, CISA Linux, VoIP and much more fun www.okay.com.mx Need LCR? Check out LCR for FusionPBX with FreeSWITCH Need Billing? Check out Billing for FusionPBX with FreeSWITCH 2015-12-15 16:38 GMT-05:00 Vladislav Ivanov <deforceczt@gmail.com>:
I'm using the systemd file from https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+1.6+Video
Just calling calls with 10 cps, low duration, should it slowly use the cpu? It really slowly eats all CPU, then drops all calls, repeats again... Shouldn't it be 10% cpu, nothing more for the same cps, no logs writing. Just simple minimal.
2015-12-15 22:22 GMT+02:00 Anthony Minessale <anthony.minessale@gmail.com> :
More issues related to resource limits. Check your allowed max sessions and sps in your config and any other ulimits on the system that you may have too low.
Its wise to have limits low enough that the calls start failing before your box does.
On Tue, Dec 15, 2015 at 12:15 PM, Vladislav Ivanov <deforceczt@gmail.com> wrote:
Hey guys,
I have a new problem that is happening with CPU load now. I have been testing freeswitch with sipp, and i'm encountering strange issue with same load. 1st time I test with 10 cps, 5 second duration calls, for just cps load. CPU idle is min idle cpu 10.00/84.67 Then I add CPS by 5, it should be enough power to be 15 cps on the machine when I have idle 84%... But my CPU goes heavily down, more and more down until it goes to 0 and spams: 2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session 2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session
Anything I can do to debug or check why it is causing this? (Same is happening on Centos 6, Debian) OS: "Debian GNU/Linux 8 (jessie) FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit)
I'm kinda lost at this point...
2015-12-15 19:21 GMT+02:00 Anthony Minessale < anthony.minessale@gmail.com>:
1000 threads is not really a show stopper =D That is what the artoo message, it tells you that the new current thread limit has been auto detected and that is the new max calls you can handle henceforth.
On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg@giagnocavo.net
wrote:
Perhaps write a patch to detect these limits and then CRIT on start if they conflict with settings.
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Vladislav Ivanov *Sent:* Tuesday, 15 December, 2015 2:15 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
Hey guys,
Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:
* soft nproc 32768
* hard nproc 32768
was not propagated on freeswitch user...
Added new line:
freeswitch soft nproc 32768
freeswitch hard nproc 32768
and it worked fine from that. Thanks to you all!
2015-12-14 23:06 GMT+02:00 Anthony Minessale < anthony.minessale@gmail.com>:
Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.
On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm@sofnet.com> wrote:
This post http://unix.stackexchange.com/questions/47595/linux-max-threads-count has a simple program to check out how many threads can be created. You can run the test program as root and then as FS user and see if it's a ulimit type of problem.
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
Is there a reason to think that CentOS 6 would cause threads to fail to spawn? This is basically just calling libapr, which is code from like 8 years ago.
Vladislav: This means it failed to create a thread (switch_thread_create in switch_apr.c). FS generates a ton of threads. The only time I’ve seen this was a long time ago on 32-bit Windows, since default stack of 1MB means you hit address space limits pretty quickly. But on 64-bit, you should be able to go all out.
I’d start by adding some if’d printf’s to check the various things that switch_thread_create does and try to pinpoint what is failing. If the libapr call is failing, then you should be able to create a minimal program that just spawns threads and see if that one is able to get 1000+. From there I’d guess troubleshooting is gonna be the same as figuring out why pthread_create fails and that should be generic to your system. (Please post if you figure out why.)
If other apps are able to spawn and keep thousands of threads and just FS isn’t then I guess you’re unlucky?
-Michael
*From:*freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Giovanni Maruzzelli *Sent:* Monday, 14 December, 2015 10:57 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?
centos 6 is not supported by FreeSWITCH 1.6 !
-giovanni
On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov < deforceczt@gmail.com <mailto:deforceczt@gmail.com>> wrote:
Centos 6,latest update. Via init script
On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice@freeswitch.org <mailto:krice@freeswitch.org>> wrote:
A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)
b)How are you starting FreeSWITCH? (eg: in the foreground or background with an init script)
*From:*freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Vladislav Ivanov *Sent:* Monday, December 14, 2015 9:47 AM *To:* freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Thread failure with low CPU
utilization?
Hello,
I have 1.6.5 Freeswitch and I'm trying to test how much lines/cps it can handle on my test machine:
CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3
GHz)
RAM: 32GB SSD: 500 GB
I'm using call generator to check flood the freeswitch, but
when
FS gets 1k sessions it sends [CRIT] switch_core_session.c:1781 Thread Failure! and the Star Wars quote... But the CPU load is less than 10%
Ulimit is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128330
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 2048
cpu time (seconds, -t) unlimited
max user processes (-u) 32768
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Server is physical, not virtual, but I dont have the same
issue
with virtual freeswitch.
Here is the "status" before the session limit:
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137 milliseconds, 603 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
994 session(s) since startup
994 session(s) - peak 994, last 5min 994
40 session(s) per Sec out of max 200, peak 41, last 5min 41
10000 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> status
UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370 milliseconds, 877 microseconds
FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit) is ready
1003 session(s) since startup
1003 session(s) - peak 1003, last 5min 1003
40 session(s) per Sec out of max 200, peak 41, last 5min 41
993 session(s) max
min idle cpu 10.00/96.77
Current Stack Size/Max 2048K/2048K
freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1781 Thread Failure!
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737 LUKE: I'm hit, but not bad.
2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738 LUKE'S VOICE: Artoo, see what you can do with it. Hang on
back
there....
Green laserfire moves past the beeping little robot as his
head
turns. After a few beeps and a twist of his mechanical arm,
Artoo reduces the max sessions to 993 thus, saving the switch from certain doom.
Any help with that? Really strange error...
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
--
Sincerely,
Giovanni Maruzzelli Cell : +39-347-2665618
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Anthony Minessale
-
Giovanni Maruzzelli
-
Ken Rice
-
Kevin Wormington
-
Luis Daniel Lucio Quiroz
-
Michael Giagnocavo
-
Vladislav Ivanov