<div dir="ltr">Hey guys,<br><br>Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:<br><br><div>* soft nproc 32768</div><div>* hard nproc 32768<br><br>was not propagated on freeswitch user... <br><br>Added new line:<br><br><div>freeswitch soft nproc 32768</div><div>freeswitch hard nproc 32768<br><br>and it worked fine from that.<br>Thanks to you all!</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-12-14 23:06 GMT+02:00 Anthony Minessale <span dir="ltr"><<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.<div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <span dir="ltr"><<a href="mailto:kworm@sofnet.com" target="_blank">kworm@sofnet.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This post<br>
<a href="http://unix.stackexchange.com/questions/47595/linux-max-threads-count" rel="noreferrer" target="_blank">http://unix.stackexchange.com/questions/47595/linux-max-threads-count</a><br>
has a simple program to check out how many threads can be created. You<br>
can run the test program as root and then as FS user and see if it's a<br>
ulimit type of problem.<br>
<span><br>
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:<br>
> Is there a reason to think that CentOS 6 would cause threads to fail to<br>
> spawn? This is basically just calling libapr, which is code from like 8<br>
> years ago.<br>
><br>
> Vladislav: This means it failed to create a thread (switch_thread_create<br>
> in switch_apr.c). FS generates a ton of threads. The only time I’ve seen<br>
> this was a long time ago on 32-bit Windows, since default stack of 1MB<br>
> means you hit address space limits pretty quickly. But on 64-bit, you<br>
> should be able to go all out.<br>
><br>
> I’d start by adding some if’d printf’s to check the various things that<br>
> switch_thread_create does and try to pinpoint what is failing. If the<br>
> libapr call is failing, then you should be able to create a minimal<br>
> program that just spawns threads and see if that one is able to get<br>
> 1000+. From there I’d guess troubleshooting is gonna be the same as<br>
> figuring out why pthread_create fails and that should be generic to your<br>
> system. (Please post if you figure out why.)<br>
><br>
> If other apps are able to spawn and keep thousands of threads and just<br>
> FS isn’t then I guess you’re unlucky?<br>
><br>
> -Michael<br>
><br>
</span>> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] *On Behalf Of<br>
> *Giovanni Maruzzelli<br>
> *Sent:* Monday, 14 December, 2015 10:57<br>
> *To:* FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
> *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?<br>
<span>><br>
> centos 6 is not supported by FreeSWITCH 1.6 !<br>
><br>
> -giovanni<br>
><br>
> On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov <<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a><br>
</span><span>> <mailto:<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a>>> wrote:<br>
><br>
> Centos 6,latest update. Via init script<br>
><br>
> On 14 Dec 2015 6:25 p.m., "Ken Rice" <<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a><br>
</span><span>> <mailto:<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>>> wrote:<br>
><br>
> A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)<br>
><br>
> b)How are you starting FreeSWITCH? (eg: in the foreground or<br>
> background with an init script)<br>
><br>
</span>> *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>><br>
> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>>] *On<br>
> Behalf Of *Vladislav Ivanov<br>
> *Sent:* Monday, December 14, 2015 9:47 AM<br>
> *To:* <a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a><br>
> <mailto:<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
> *Subject:* [Freeswitch-users] Thread failure with low CPU<br>
<div><div>> utilization?<br>
><br>
> Hello,<br>
><br>
> I have 1.6.5 Freeswitch and I'm trying to test how much<br>
> lines/cps it can handle on my test machine:<br>
><br>
> CPU: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz x2 (8 cores, 3 GHz)<br>
> RAM: 32GB<br>
> SSD: 500 GB<br>
><br>
> I'm using call generator to check flood the freeswitch, but when<br>
> FS gets 1k sessions it sends<br>
> [CRIT] switch_core_session.c:1781 Thread Failure!<br>
> and the Star Wars quote... But the CPU load is less than 10%<br>
><br>
> Ulimit is as follows:<br>
><br>
> core file size (blocks, -c) 0<br>
><br>
> data seg size (kbytes, -d) unlimited<br>
><br>
> scheduling priority (-e) 0<br>
><br>
> file size (blocks, -f) unlimited<br>
><br>
> pending signals (-i) 128330<br>
><br>
> max locked memory (kbytes, -l) 64<br>
><br>
> max memory size (kbytes, -m) unlimited<br>
><br>
> open files (-n) 32768<br>
><br>
> pipe size (512 bytes, -p) 8<br>
><br>
> POSIX message queues (bytes, -q) 819200<br>
><br>
> real-time priority (-r) 0<br>
><br>
> stack size (kbytes, -s) 2048<br>
><br>
> cpu time (seconds, -t) unlimited<br>
><br>
> max user processes (-u) 32768<br>
><br>
> virtual memory (kbytes, -v) unlimited<br>
><br>
> file locks (-x) unlimited<br>
><br>
><br>
> Server is physical, not virtual, but I dont have the same issue<br>
> with virtual freeswitch.<br>
><br>
> Here is the "status" before the session limit:<br>
><br>
> freeswitch@internal> status<br>
><br>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137<br>
> milliseconds, 603 microseconds<br>
><br>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
> 64bit) is ready<br>
><br>
> 994 session(s) since startup<br>
><br>
> 994 session(s) - peak 994, last 5min 994<br>
><br>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
><br>
> 10000 session(s) max<br>
><br>
> min idle cpu 10.00/96.77<br>
><br>
> Current Stack Size/Max 2048K/2048K<br>
><br>
> freeswitch@internal> status<br>
><br>
> UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370<br>
> milliseconds, 877 microseconds<br>
><br>
> FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
> 64bit) is ready<br>
><br>
> 1003 session(s) since startup<br>
><br>
> 1003 session(s) - peak 1003, last 5min 1003<br>
><br>
> 40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
><br>
> 993 session(s) max<br>
><br>
> min idle cpu 10.00/96.77<br>
><br>
> Current Stack Size/Max 2048K/2048K<br>
><br>
> freeswitch@internal> 2015-12-14 15:37:37.436968 [CRIT]<br>
> switch_core_session.c:1781 Thread Failure!<br>
><br>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737<br>
> LUKE: I'm hit, but not bad.<br>
><br>
> 2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738<br>
> LUKE'S VOICE: Artoo, see what you can do with it. Hang on back<br>
> there....<br>
><br>
> Green laserfire moves past the beeping little robot as his head<br>
> turns. After a few beeps and a twist of his mechanical arm,<br>
><br>
> Artoo reduces the max sessions to 993 thus, saving the switch<br>
> from certain doom.<br>
><br>
> Any help with that? Really strange error...<br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
</div></div>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>><br>
<span>> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
</span>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<span>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
</span>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>><br>
<span>> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
</span>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<div><div>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> Sincerely,<br>
><br>
> Giovanni Maruzzelli<br>
> Cell : <a href="tel:%2B39-347-2665618" value="+393472665618" target="_blank">+39-347-2665618</a><br>
><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ <a href="tel:%2B19193869900" value="+19193869900" target="_blank">+19193869900</a> </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</font></span></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div>