I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.

Cal

On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote:

On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net> wrote:

Hello,

I'm having a problem where FS seems to exhaust it's open file limit
after about 7 hours of operation.  Once this happens, the database and
curl connections no longer function, and I usually lose the ability to
get a CLI connection.  Existing calls usually stay up, until I kill -9
and launch again.

I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e
on Centos 2.6.18-348.el5

This machine had been running solid for at least 90 days without a hitch
- I had almost hit 500K sessions, when this started happening two days
ago.  I have changed very little on the machine - the odd dialplan
route, and nothing that seems to coincide with that time.

It looks like CURL is not closing sockets properly, as netstat shows
between 2000 - 4000 http connections in TIME_WAIT state.  The number
slowly creeps up, then down a little, then up some more until it all
comes crashing down.

Can you confirm if your web server is responding to the requests?

One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.

You could increase the ulimit (or set it to unlimited) as per;

http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.

Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.

You can also tweak kernel TCP stack settings as per;

http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/

Hope this helps

 

ls /proc/15742/fd|wc -l currently returns about 400.  It was 300 an hour
ago.  When it gets to 1024, I'm going to have to restart again.

Any suggestions as to where I should look?

-Tim

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org