<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Dec 15, 2013 at 2:05 AM, Tim St. Pierre <span dir="ltr"><<a href="mailto:fs-list@communicatefreely.net" target="_blank">fs-list@communicatefreely.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hello everyone,<br>
<br>
Thanks for the help on the conference call!<br>
<br>
I have tracked down the source of the problem, and it's not at all what<br>
I expected.<br>
<br>
One of our customers has some sort of issue with their phone, and it<br>
seems that every time the phone connects to SIP/TCP, it opens a new<br>
socket, and never closes the old one. After two days, there are now<br>
3000 established TCP sockets from that customer (to the sofia profile).<br>
<br>
I will be paying them a visit on Monday morning. Now that I have<br>
changed the ulimit settings, this doesn't bring the system down at least.<br>
<br>
Is there any way I can limit the number of sockets a specific endpoint<br>
(or at least a specific source IP) can open?<br></blockquote><div><br></div><div>Yup, this can be easily done with iptables;</div><div><br></div><div><a href="http://www.cyberciti.biz/faq/iptables-connection-limits-howto/">http://www.cyberciti.biz/faq/iptables-connection-limits-howto/</a></div>
<div><a href="http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/">http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/</a><br></div><div><a href="https://sites.google.com/site/admin4life/singlehostiptables">https://sites.google.com/site/admin4life/singlehostiptables</a><br>
</div><div><br></div><div>You can also do it with most firewall appliances, i.e. if you have a Cisco ASA in front of your boxes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm hoping to upgrade to a current version, perhaps in the next two<br>
weeks when everything is quiet. I had 1.2.8 running for nearly 90 days,<br>
and that's a record for us. Needless to say, I was in no hurry to touch<br>
anything.<br>
<br>
Thanks!<br>
<div class="im"><br>
<br>
<br>
On 13-12-13 04:50 PM, Anthony Minessale wrote:<br>
> You still should not be on 1.2.8 =D<br>
><br>
><br>
><br>
> On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd]<br>
> <<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a><br>
</div><div class="im">> <mailto:<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a>>> wrote:<br>
><br>
> As per our discussion on the conference call, this was caused by too<br>
> many sessions and not enough file descriptors.<br>
><br>
> This can be fixed by changing the ulimit as explained here;<br>
> <a href="http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recommended_ULIMIT_settings" target="_blank">http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recommended_ULIMIT_settings</a><br>
><br>
> Cal<br>
><br>
><br>
> On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale<br>
</div>> <<a href="mailto:anthony.minessale@gmail.com">anthony.minessale@gmail.com</a> <mailto:<a href="mailto:anthony.minessale@gmail.com">anthony.minessale@gmail.com</a>>><br>
<div class="im">> wrote:<br>
><br>
> You may want to start by updating to a more recent version to<br>
> rule out anything already improved.<br>
> It may be environmental since it started happening after it was<br>
> once working better but its hard to tell and hard to debug older<br>
> versions.<br>
><br>
><br>
><br>
> On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media<br>
> Ltd] <<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a><br>
</div><div class="im">> <mailto:<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a>>> wrote:<br>
><br>
> I'll be on the FS conference call for the next 30 mins if<br>
> you want to ask any questions about this btw.<br>
><br>
> Cal<br>
><br>
><br>
> On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity<br>
> Media Ltd] <<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a><br>
</div><div class="im">> <mailto:<a href="mailto:cal.leeming@simplicitymedialtd.co.uk">cal.leeming@simplicitymedialtd.co.uk</a>>> wrote:<br>
><br>
><br>
><br>
><br>
> On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre<br>
> <<a href="mailto:fs-list@communicatefreely.net">fs-list@communicatefreely.net</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:fs-list@communicatefreely.net">fs-list@communicatefreely.net</a>>> wrote:<br>
><br>
> Hello,<br>
><br>
> I'm having a problem where FS seems to exhaust it's<br>
> open file limit<br>
> after about 7 hours of operation. Once this<br>
> happens, the database and<br>
> curl connections no longer function, and I usually<br>
> lose the ability to<br>
> get a CLI connection. Existing calls usually stay<br>
> up, until I kill -9<br>
> and launch again.<br>
><br>
> I'm using FreeSWITCH Version<br>
> 1.2.8+git~20130403T221701Z~79be96aa8e<br>
> on Centos 2.6.18-348.el5<br>
><br>
> This machine had been running solid for at least 90<br>
> days without a hitch<br>
> - I had almost hit 500K sessions, when this started<br>
> happening two days<br>
> ago. I have changed very little on the machine -<br>
> the odd dialplan<br>
> route, and nothing that seems to coincide with that<br>
> time.<br>
><br>
><br>
> It looks like CURL is not closing sockets properly,<br>
> as netstat shows<br>
> between 2000 - 4000 http connections in TIME_WAIT<br>
> state. The number<br>
> slowly creeps up, then down a little, then up some<br>
> more until it all<br>
> comes crashing down.<br>
><br>
><br>
> Can you confirm if your web server is responding to the<br>
> requests?<br>
><br>
> One possible explanation is this spike in traffic caused<br>
> excessive requests which your web server could not<br>
> handle, and thus you hit the ulimit.<br>
><br>
> You could increase the ulimit (or set it to unlimited)<br>
> as per;<br>
> <a href="http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/" target="_blank">http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/</a><br>
><br>
> However, if your web server is struggling to keep up<br>
> with the work load then increasing the max open files<br>
> will just make the problem worse. Increasing is fine, as<br>
> long as your web server can keep up with the load,<br>
> otherwise it will just continue to snowball.<br>
><br>
> Try installing some profiling tools on your web server<br>
> to monitor the stats, you can use something like New<br>
> Relic or AppNeta for this, or even just change your<br>
> access log to output the loading/response times into<br>
> your access log, then look for high response times.<br>
><br>
> You can also tweak kernel TCP stack settings as per;<br>
> <a href="http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/" target="_blank">http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/</a><br>
><br>
> Hope this helps<br>
><br>
><br>
><br>
> ls /proc/15742/fd|wc -l currently returns about 400.<br>
> It was 300 an hour<br>
> ago. When it gets to 1024, I'm going to have to<br>
> restart again.<br>
><br>
> Any suggestions as to where I should look?<br>
><br>
> -Tim<br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
</div></div>> <mailto:<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>><br>
<div class="im">> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication<br>
> Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<div class="im">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
</div>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>><br>
<div class="im">> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<div class="im">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<br>
><br>
> ☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞<br>
> <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><br>
</div>> ☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> <<a href="http://irc.freenode.net" target="_blank">http://irc.freenode.net</a>> #freeswitch ☞<br>
> _<a href="http://freeswitch.org/g+_" target="_blank">http://freeswitch.org/g+_</a><br>
<div class="im">><br>
> ClueCon Weekly Development Call<br>
> ☎ <a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:sip%253A888@conference.freeswitch.org">sip%3A888@conference.freeswitch.org</a>> ☎ <a href="tel:%2B19193869900" value="+19193869900">+19193869900</a><br>
> <tel:%2B19193869900><br>
<div class="im">><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
</div>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>><br>
<div class="im">> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<div class="im">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
</div>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>><br>
<div class="im">> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>><br>
<div class="im">> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<br>
><br>
> ☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞<br>
> <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><br>
</div>> ☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> <<a href="http://irc.freenode.net" target="_blank">http://irc.freenode.net</a>> #freeswitch ☞<br>
> _<a href="http://freeswitch.org/g+_" target="_blank">http://freeswitch.org/g+_</a><br>
<div class="im">><br>
> ClueCon Weekly Development Call<br>
> ☎ <a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br>
</div>> <mailto:<a href="mailto:sip%253A888@conference.freeswitch.org">sip%3A888@conference.freeswitch.org</a>> ☎ <a href="tel:%2B19193869900" value="+19193869900">+19193869900</a><br>
<div class=""><div class="h5">><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br></div></div>