[Freeswitch-users] Thread failure with low CPU utilization?
Vladislav Ivanov
deforceczt at gmail.com
Tue Dec 15 21:15:59 MSK 2015
Hey guys,
I have a new problem that is happening with CPU load now.
I have been testing freeswitch with sipp, and i'm encountering strange
issue with same load.
1st time I test with 10 cps, 5 second duration calls, for just cps load.
CPU idle is min idle cpu 10.00/84.67
Then I add CPS by 5, it should be enough power to be 15 cps on the machine
when I have idle 84%...
But my CPU goes heavily down, more and more down until it goes to 0 and
spams:
2015-12-15 20:13:43.199451 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.259702 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.279578 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.299562 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.399496 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:13:43.419667 [CRIT] mod_sofia.c:4406 Error Creating Session
2015-12-15 20:14:32.819449 [CRIT] mod_sofia.c:4406 Error Creating Session
Anything I can do to debug or check why it is causing this?
(Same is happening on Centos 6, Debian)
OS: "Debian GNU/Linux 8 (jessie)
FS: FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z 64bit)
I'm kinda lost at this point...
2015-12-15 19:21 GMT+02:00 Anthony Minessale <anthony.minessale at gmail.com>:
> 1000 threads is not really a show stopper =D
> That is what the artoo message, it tells you that the new current thread
> limit has been auto detected and that is the new max calls you can handle
> henceforth.
>
>
> On Tue, Dec 15, 2015 at 5:16 AM, Michael Giagnocavo <mgg at giagnocavo.net>
> wrote:
>
>> Perhaps write a patch to detect these limits and then CRIT on start if
>> they conflict with settings.
>>
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Vladislav
>> Ivanov
>> *Sent:* Tuesday, 15 December, 2015 2:15
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
>> utilization?
>>
>>
>>
>> Hey guys,
>>
>> Anthony was correct in this case, I ran freeswitch not from root, for
>> some reason my /etc/security/limits.conf file line:
>>
>> * soft nproc 32768
>>
>> * hard nproc 32768
>>
>> was not propagated on freeswitch user...
>>
>> Added new line:
>>
>> freeswitch soft nproc 32768
>>
>> freeswitch hard nproc 32768
>>
>> and it worked fine from that.
>> Thanks to you all!
>>
>>
>>
>> 2015-12-14 23:06 GMT+02:00 Anthony Minessale <anthony.minessale at gmail.com
>> >:
>>
>> Are you running FS as root or a regular user, you need to check the
>> ulimits of concurrent threads and some boxes have a hard limit of threads,
>> its not all tied to CPU usage.
>>
>>
>>
>>
>>
>> On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <kworm at sofnet.com>
>> wrote:
>>
>> This post
>> http://unix.stackexchange.com/questions/47595/linux-max-threads-count
>> has a simple program to check out how many threads can be created. You
>> can run the test program as root and then as FS user and see if it's a
>> ulimit type of problem.
>>
>> On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:
>> > Is there a reason to think that CentOS 6 would cause threads to fail to
>> > spawn? This is basically just calling libapr, which is code from like 8
>> > years ago.
>> >
>> > Vladislav: This means it failed to create a thread (switch_thread_create
>> > in switch_apr.c). FS generates a ton of threads. The only time I’ve seen
>> > this was a long time ago on 32-bit Windows, since default stack of 1MB
>> > means you hit address space limits pretty quickly. But on 64-bit, you
>> > should be able to go all out.
>> >
>> > I’d start by adding some if’d printf’s to check the various things that
>> > switch_thread_create does and try to pinpoint what is failing. If the
>> > libapr call is failing, then you should be able to create a minimal
>> > program that just spawns threads and see if that one is able to get
>> > 1000+. From there I’d guess troubleshooting is gonna be the same as
>> > figuring out why pthread_create fails and that should be generic to your
>> > system. (Please post if you figure out why.)
>> >
>> > If other apps are able to spawn and keep thousands of threads and just
>> > FS isn’t then I guess you’re unlucky?
>> >
>> > -Michael
>> >
>> > *From:*freeswitch-users-bounces at lists.freeswitch.org
>> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of
>> > *Giovanni Maruzzelli
>> > *Sent:* Monday, 14 December, 2015 10:57
>> > *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> > *Subject:* Re: [Freeswitch-users] Thread failure with low CPU
>> utilization?
>> >
>> > centos 6 is not supported by FreeSWITCH 1.6 !
>> >
>> > -giovanni
>> >
>> > On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <deforceczt at gmail.com
>> > <mailto:deforceczt at gmail.com>> wrote:
>> >
>> > Centos 6,latest update. Via init script
>> >
>> > On 14 Dec 2015 6:25 p.m., "Ken Rice" <krice at freeswitch.org
>> > <mailto:krice at freeswitch.org>> wrote:
>> >
>> > A)What platform are you running FreeSWITCH on? (eg: Debian 8
>> etc)
>> >
>> > b)How are you starting FreeSWITCH? (eg: in the foreground or
>> > background with an init script)
>> >
>> > *From:*freeswitch-users-bounces at lists.freeswitch.org
>> > <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>> > [mailto:freeswitch-users-bounces at lists.freeswitch.org
>> > <mailto:freeswitch-users-bounces at lists.freeswitch.org>] *On
>> > Behalf Of *Vladislav Ivanov
>> > *Sent:* Monday, December 14, 2015 9:47 AM
>> > *To:* freeswitch-users at lists.freeswitch.org
>> > <mailto:freeswitch-users at lists.freeswitch.org>
>> > *Subject:* [Freeswitch-users] Thread failure with low CPU
>>
>> > utilization?
>> >
>> > Hello,
>> >
>> > I have 1.6.5 Freeswitch and I'm trying to test how much
>> > lines/cps it can handle on my test machine:
>> >
>> > CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz)
>> > RAM: 32GB
>> > SSD: 500 GB
>> >
>> > I'm using call generator to check flood the freeswitch, but when
>> > FS gets 1k sessions it sends
>> > [CRIT] switch_core_session.c:1781 Thread Failure!
>> > and the Star Wars quote... But the CPU load is less than 10%
>> >
>> > Ulimit is as follows:
>> >
>> > core file size (blocks, -c) 0
>> >
>> > data seg size (kbytes, -d) unlimited
>> >
>> > scheduling priority (-e) 0
>> >
>> > file size (blocks, -f) unlimited
>> >
>> > pending signals (-i) 128330
>> >
>> > max locked memory (kbytes, -l) 64
>> >
>> > max memory size (kbytes, -m) unlimited
>> >
>> > open files (-n) 32768
>> >
>> > pipe size (512 bytes, -p) 8
>> >
>> > POSIX message queues (bytes, -q) 819200
>> >
>> > real-time priority (-r) 0
>> >
>> > stack size (kbytes, -s) 2048
>> >
>> > cpu time (seconds, -t) unlimited
>> >
>> > max user processes (-u) 32768
>> >
>> > virtual memory (kbytes, -v) unlimited
>> >
>> > file locks (-x) unlimited
>> >
>> >
>> > Server is physical, not virtual, but I dont have the same issue
>> > with virtual freeswitch.
>> >
>> > Here is the "status" before the session limit:
>> >
>> > freeswitch at internal> status
>> >
>> > UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137
>> > milliseconds, 603 microseconds
>> >
>> > FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>> > 64bit) is ready
>> >
>> > 994 session(s) since startup
>> >
>> > 994 session(s) - peak 994, last 5min 994
>> >
>> > 40 session(s) per Sec out of max 200, peak 41, last 5min 41
>> >
>> > 10000 session(s) max
>> >
>> > min idle cpu 10.00/96.77
>> >
>> > Current Stack Size/Max 2048K/2048K
>> >
>> > freeswitch at internal> status
>> >
>> > UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370
>> > milliseconds, 877 microseconds
>> >
>> > FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z
>> > 64bit) is ready
>> >
>> > 1003 session(s) since startup
>> >
>> > 1003 session(s) - peak 1003, last 5min 1003
>> >
>> > 40 session(s) per Sec out of max 200, peak 41, last 5min 41
>> >
>> > 993 session(s) max
>> >
>> > min idle cpu 10.00/96.77
>> >
>> > Current Stack Size/Max 2048K/2048K
>> >
>> > freeswitch at internal> 2015-12-14 15:37:37.436968 [CRIT]
>> > switch_core_session.c:1781 Thread Failure!
>> >
>> > 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737
>> > LUKE: I'm hit, but not bad.
>> >
>> > 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738
>> > LUKE'S VOICE: Artoo, see what you can do with it. Hang on back
>> > there....
>> >
>> > Green laserfire moves past the beeping little robot as his head
>> > turns. After a few beeps and a twist of his mechanical arm,
>> >
>> > Artoo reduces the max sessions to 993 thus, saving the switch
>> > from certain doom.
>> >
>> > Any help with that? Really strange error...
>> >
>> >
>> _________________________________________________________________________
>> > Professional FreeSWITCH Consulting Services:
>>
>> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> > http://www.freeswitchsolutions.com
>> >
>> > Official FreeSWITCH Sites
>> > http://www.freeswitch.org
>> > http://confluence.freeswitch.org
>> > http://www.cluecon.com
>> >
>> > FreeSWITCH-users mailing list
>> > FreeSWITCH-users at lists.freeswitch.org
>> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> > UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> > http://www.freeswitch.org
>> >
>> >
>> >
>> _________________________________________________________________________
>> > Professional FreeSWITCH Consulting Services:
>> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> > http://www.freeswitchsolutions.com
>> >
>> > Official FreeSWITCH Sites
>> > http://www.freeswitch.org
>> > http://confluence.freeswitch.org
>> > http://www.cluecon.com
>> >
>> > FreeSWITCH-users mailing list
>> > FreeSWITCH-users at lists.freeswitch.org
>> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>
>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> > UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> > http://www.freeswitch.org
>> >
>> >
>> >
>> >
>> > --
>> >
>> > Sincerely,
>> >
>> > Giovanni Maruzzelli
>> > Cell : +39-347-2665618
>> >
>> >
>> >
>> >
>> _________________________________________________________________________
>> > Professional FreeSWITCH Consulting Services:
>> > consulting at freeswitch.org
>> > http://www.freeswitchsolutions.com
>> >
>> > Official FreeSWITCH Sites
>> > http://www.freeswitch.org
>> > http://confluence.freeswitch.org
>> > http://www.cluecon.com
>> >
>> > FreeSWITCH-users mailing list
>> > FreeSWITCH-users at lists.freeswitch.org
>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> > UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> > http://www.freeswitch.org
>> >
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>>
>> --
>>
>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>
>>
>>
>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>> http://twitter.com/FreeSWITCH
>>
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>>
>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>
>>
>>
>> https://www.youtube.com/watch?v=9XXgW34t40s
>>
>> https://www.youtube.com/watch?v=NLaDpGQuZDA
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>
> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>
> https://www.youtube.com/watch?v=9XXgW34t40s
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20151215/4bc1126a/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list