<div dir="ltr">Hey guys,<br><br>Anthony was correct in this case, I ran freeswitch not from root, for some reason my /etc/security/limits.conf file line:<br><br><div>*          soft     nproc          32768</div><div>*          hard     nproc          32768<br><br>was not propagated on freeswitch user... <br><br>Added new line:<br><br><div>freeswitch          soft     nproc          32768</div><div>freeswitch          hard     nproc          32768<br><br>and it worked fine from that.<br>Thanks to you all!</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-12-14 23:06 GMT+02:00 Anthony Minessale <span dir="ltr">&lt;<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>&gt;</span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Are you running FS as root or a regular user, you need to check the ulimits of concurrent threads and some boxes have a hard limit of threads, its not all tied to CPU usage.<div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 1:30 PM, Kevin Wormington <span dir="ltr">&lt;<a href="mailto:kworm@sofnet.com" target="_blank">kworm@sofnet.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This post<br>
<a href="http://unix.stackexchange.com/questions/47595/linux-max-threads-count" rel="noreferrer" target="_blank">http://unix.stackexchange.com/questions/47595/linux-max-threads-count</a><br>
has a simple program to check out how many threads can be created.  You<br>
can run the test program as root and then as FS user and see if it&#39;s a<br>
ulimit type of problem.<br>
<span><br>
On 12/14/2015 01:08 PM, Michael Giagnocavo wrote:<br>
&gt; Is there a reason to think that CentOS 6 would cause threads to fail to<br>
&gt; spawn? This is basically just calling libapr, which is code from like 8<br>
&gt; years ago.<br>
&gt;<br>
&gt; Vladislav: This means it failed to create a thread (switch_thread_create<br>
&gt; in switch_apr.c). FS generates a ton of threads. The only time I’ve seen<br>
&gt; this was a long time ago on 32-bit Windows, since default stack of 1MB<br>
&gt; means you hit address space limits pretty quickly. But on 64-bit, you<br>
&gt; should be able to go all out.<br>
&gt;<br>
&gt; I’d start by adding some if’d printf’s to check the various things that<br>
&gt; switch_thread_create does and try to pinpoint what is failing. If the<br>
&gt; libapr call is failing, then you should be able to create a minimal<br>
&gt; program that just spawns threads and see if that one is able to get<br>
&gt; 1000+. From there I’d guess troubleshooting is gonna be the same as<br>
&gt; figuring out why pthread_create fails and that should be generic to your<br>
&gt; system. (Please post if you figure out why.)<br>
&gt;<br>
&gt; If other apps are able to spawn and keep thousands of threads and just<br>
&gt; FS isn’t then I guess you’re unlucky?<br>
&gt;<br>
&gt; -Michael<br>
&gt;<br>
</span>&gt; *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt; [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] *On Behalf Of<br>
&gt; *Giovanni Maruzzelli<br>
&gt; *Sent:* Monday, 14 December, 2015 10:57<br>
&gt; *To:* FreeSWITCH Users Help &lt;<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>&gt;<br>
&gt; *Subject:* Re: [Freeswitch-users] Thread failure with low CPU utilization?<br>
<span>&gt;<br>
&gt; centos 6 is not supported by FreeSWITCH 1.6 !<br>
&gt;<br>
&gt; -giovanni<br>
&gt;<br>
&gt; On Mon, Dec 14, 2015 at 5:47 PM, Vladislav Ivanov &lt;<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a><br>
</span><span>&gt; &lt;mailto:<a href="mailto:deforceczt@gmail.com" target="_blank">deforceczt@gmail.com</a>&gt;&gt; wrote:<br>
&gt;<br>
&gt;     Centos 6,latest update. Via init script<br>
&gt;<br>
&gt;     On 14 Dec 2015 6:25 p.m., &quot;Ken Rice&quot; &lt;<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a><br>
</span><span>&gt;     &lt;mailto:<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>&gt;&gt; wrote:<br>
&gt;<br>
&gt;         A)What platform are you running FreeSWITCH on? (eg: Debian 8 etc)<br>
&gt;<br>
&gt;         b)How are you starting FreeSWITCH? (eg: in the foreground or<br>
&gt;         background with an init script)<br>
&gt;<br>
</span>&gt;         *From:*<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;         &lt;mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>&gt;<br>
&gt;         [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a><br>
&gt;         &lt;mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>&gt;] *On<br>
&gt;         Behalf Of *Vladislav Ivanov<br>
&gt;         *Sent:* Monday, December 14, 2015 9:47 AM<br>
&gt;         *To:* <a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a><br>
&gt;         &lt;mailto:<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>&gt;<br>
&gt;         *Subject:* [Freeswitch-users] Thread failure with low CPU<br>
<div><div>&gt;         utilization?<br>
&gt;<br>
&gt;         Hello,<br>
&gt;<br>
&gt;         I have 1.6.5 Freeswitch and I&#39;m trying to test how much<br>
&gt;         lines/cps it can handle on my test machine:<br>
&gt;<br>
&gt;         CPU: Intel(R) Xeon(R) CPU E5450  @ 3.00GHz x2 (8 cores, 3 GHz)<br>
&gt;         RAM: 32GB<br>
&gt;         SSD: 500 GB<br>
&gt;<br>
&gt;         I&#39;m using call generator to check flood the freeswitch, but when<br>
&gt;         FS gets 1k sessions it sends<br>
&gt;         [CRIT] switch_core_session.c:1781 Thread Failure!<br>
&gt;         and the Star Wars quote... But the CPU load is less than 10%<br>
&gt;<br>
&gt;         Ulimit is as follows:<br>
&gt;<br>
&gt;         core file size          (blocks, -c) 0<br>
&gt;<br>
&gt;         data seg size           (kbytes, -d) unlimited<br>
&gt;<br>
&gt;         scheduling priority             (-e) 0<br>
&gt;<br>
&gt;         file size               (blocks, -f) unlimited<br>
&gt;<br>
&gt;         pending signals                 (-i) 128330<br>
&gt;<br>
&gt;         max locked memory       (kbytes, -l) 64<br>
&gt;<br>
&gt;         max memory size         (kbytes, -m) unlimited<br>
&gt;<br>
&gt;         open files                      (-n) 32768<br>
&gt;<br>
&gt;         pipe size            (512 bytes, -p) 8<br>
&gt;<br>
&gt;         POSIX message queues     (bytes, -q) 819200<br>
&gt;<br>
&gt;         real-time priority              (-r) 0<br>
&gt;<br>
&gt;         stack size              (kbytes, -s) 2048<br>
&gt;<br>
&gt;         cpu time               (seconds, -t) unlimited<br>
&gt;<br>
&gt;         max user processes              (-u) 32768<br>
&gt;<br>
&gt;         virtual memory          (kbytes, -v) unlimited<br>
&gt;<br>
&gt;         file locks                      (-x) unlimited<br>
&gt;<br>
&gt;<br>
&gt;         Server is physical, not virtual, but I dont have the same issue<br>
&gt;         with virtual freeswitch.<br>
&gt;<br>
&gt;         Here is the &quot;status&quot; before the session limit:<br>
&gt;<br>
&gt;         freeswitch@internal&gt; status<br>
&gt;<br>
&gt;         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 137<br>
&gt;         milliseconds, 603 microseconds<br>
&gt;<br>
&gt;         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
&gt;         64bit) is ready<br>
&gt;<br>
&gt;         994 session(s) since startup<br>
&gt;<br>
&gt;         994 session(s) - peak 994, last 5min 994<br>
&gt;<br>
&gt;         40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
&gt;<br>
&gt;         10000 session(s) max<br>
&gt;<br>
&gt;         min idle cpu 10.00/96.77<br>
&gt;<br>
&gt;         Current Stack Size/Max 2048K/2048K<br>
&gt;<br>
&gt;         freeswitch@internal&gt; status<br>
&gt;<br>
&gt;         UP 0 years, 0 days, 0 hours, 0 minutes, 56 seconds, 370<br>
&gt;         milliseconds, 877 microseconds<br>
&gt;<br>
&gt;         FreeSWITCH (Version 1.6.5 git 70b8c17 2015-11-20 20:57:50Z<br>
&gt;         64bit) is ready<br>
&gt;<br>
&gt;         1003 session(s) since startup<br>
&gt;<br>
&gt;         1003 session(s) - peak 1003, last 5min 1003<br>
&gt;<br>
&gt;         40 session(s) per Sec out of max 200, peak 41, last 5min 41<br>
&gt;<br>
&gt;         993 session(s) max<br>
&gt;<br>
&gt;         min idle cpu 10.00/96.77<br>
&gt;<br>
&gt;         Current Stack Size/Max 2048K/2048K<br>
&gt;<br>
&gt;         freeswitch@internal&gt; 2015-12-14 15:37:37.436968 [CRIT]<br>
&gt;         switch_core_session.c:1781 Thread Failure!<br>
&gt;<br>
&gt;         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1737<br>
&gt;         LUKE: I&#39;m hit, but not bad.<br>
&gt;<br>
&gt;         2015-12-14 15:37:37.436968 [CRIT] switch_core_session.c:1738<br>
&gt;         LUKE&#39;S VOICE: Artoo, see what you can do with it. Hang on back<br>
&gt;         there....<br>
&gt;<br>
&gt;         Green laserfire moves past the beeping little robot as his head<br>
&gt;         turns.  After a few beeps and a twist of his mechanical arm,<br>
&gt;<br>
&gt;         Artoo reduces the max sessions to 993 thus, saving the switch<br>
&gt;         from certain doom.<br>
&gt;<br>
&gt;         Any help with that? Really strange error...<br>
&gt;<br>
&gt;         _________________________________________________________________________<br>
&gt;         Professional FreeSWITCH Consulting Services:<br>
</div></div>&gt;         <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> &lt;mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>&gt;<br>
<span>&gt;         <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;<br>
&gt;         Official FreeSWITCH Sites<br>
&gt;         <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;         <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt;         <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;<br>
&gt;         FreeSWITCH-users mailing list<br>
&gt;         <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
</span>&gt;         &lt;mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>&gt;<br>
<span>&gt;         <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt;         UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt;         <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;<br>
&gt;<br>
&gt;     _________________________________________________________________________<br>
&gt;     Professional FreeSWITCH Consulting Services:<br>
</span>&gt;     <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a> &lt;mailto:<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>&gt;<br>
<span>&gt;     <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;<br>
&gt;     Official FreeSWITCH Sites<br>
&gt;     <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;     <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt;     <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;<br>
&gt;     FreeSWITCH-users mailing list<br>
&gt;     <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
</span>&gt;     &lt;mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>&gt;<br>
<div><div>&gt;     <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt;     UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt;     <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; --<br>
&gt;<br>
&gt; Sincerely,<br>
&gt;<br>
&gt; Giovanni Maruzzelli<br>
&gt; Cell : <a href="tel:%2B39-347-2665618" value="+393472665618" target="_blank">+39-347-2665618</a><br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; _________________________________________________________________________<br>
&gt; Professional FreeSWITCH Consulting Services:<br>
&gt; <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
&gt; <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;<br>
&gt; Official FreeSWITCH Sites<br>
&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt; <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt; <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;<br>
&gt; FreeSWITCH-users mailing list<br>
&gt; <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
&gt; <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt; UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a>  ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a>  ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a>  ☎ <a href="tel:%2B19193869900" value="+19193869900" target="_blank">+19193869900</a> </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</font></span></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div>