[Freeswitch-users] mod_xml_curl creating too many open files

Tim St. Pierre fs-list at communicatefreely.net
Sat Dec 14 00:25:23 MSK 2013


Yes, it seems to be holding up okay.  It will quit at 5am, when there is
nobody on the phone.  It seems to be very predictable.  We have about
500 phones registering every 20 minutes.  The webserver gets 2-3
requests per second, occasionally with bursts of 20/s, but it seems to
be more than capable of handling it.

If the load drops off, should the sockets not eventually time out and
close?  That's an easy test to do.

I just hopped into the conference.

-Tim

On 13-12-13 04:10 PM, Cal Leeming [Simplicity Media Ltd] wrote:
> 
> 
> 
> On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre
> <fs-list at communicatefreely.net <mailto:fs-list at communicatefreely.net>>
> wrote:
> 
>     Hello,
> 
>     I'm having a problem where FS seems to exhaust it's open file limit
>     after about 7 hours of operation.  Once this happens, the database and
>     curl connections no longer function, and I usually lose the ability to
>     get a CLI connection.  Existing calls usually stay up, until I kill -9
>     and launch again.
> 
>     I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e
>     on Centos 2.6.18-348.el5
> 
>     This machine had been running solid for at least 90 days without a hitch
>     - I had almost hit 500K sessions, when this started happening two days
>     ago.  I have changed very little on the machine - the odd dialplan
>     route, and nothing that seems to coincide with that time.
> 
> 
>     It looks like CURL is not closing sockets properly, as netstat shows
>     between 2000 - 4000 http connections in TIME_WAIT state.  The number
>     slowly creeps up, then down a little, then up some more until it all
>     comes crashing down.
> 
> 
> Can you confirm if your web server is responding to the requests?
> 
> One possible explanation is this spike in traffic caused excessive
> requests which your web server could not handle, and thus you hit the
> ulimit.
> 
> You could increase the ulimit (or set it to unlimited) as per;
> http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
> 
> However, if your web server is struggling to keep up with the work load
> then increasing the max open files will just make the problem worse.
> Increasing is fine, as long as your web server can keep up with the
> load, otherwise it will just continue to snowball.
> 
> Try installing some profiling tools on your web server to monitor the
> stats, you can use something like New Relic or AppNeta for this, or even
> just change your access log to output the loading/response times into
> your access log, then look for high response times.
> 
> You can also tweak kernel TCP stack settings as per;
> http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/
> 
> Hope this helps
>  
> 
> 
>     ls /proc/15742/fd|wc -l currently returns about 400.  It was 300 an hour
>     ago.  When it gets to 1024, I'm going to have to restart again.
> 
>     Any suggestions as to where I should look?
> 
>     -Tim
> 
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
> 
>     
>     
> 
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
> 
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
> 
> 
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
> 



Join us at ClueCon 2013 Aug 6-8, 2013
More information about the FreeSWITCH-users mailing list