[Freeswitch-users] mod_xml_curl creating too many open files

[Tim St. Pierre]

I thought that too, but I ran a ping test against the machine (to that
interface) when it happened.  It didn't lose a single packet.

-Tim

On 13-12-13 04:19 PM, Lloyd Aloysius wrote:
> Tim
> 
> I would first check the connectivity
> 
> Switch ports?
> 
> Lloyd
> 
> On Dec 13, 2013 4:14 PM, "Cal Leeming [Simplicity Media Ltd]"
> <cal.leeming at simplicitymedialtd.co.uk
> <mailto:cal.leeming at simplicitymedialtd.co.uk>> wrote:
> 
>     I'll be on the FS conference call for the next 30 mins if you want
>     to ask any questions about this btw.
> 
>     Cal
> 
> 
>     On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd]
>     <cal.leeming at simplicitymedialtd.co.uk
>     <mailto:cal.leeming at simplicitymedialtd.co.uk>> wrote:
> 
> 
> 
> 
>         On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre
>         <fs-list at communicatefreely.net
>         <mailto:fs-list at communicatefreely.net>> wrote:
> 
>             Hello,
> 
>             I'm having a problem where FS seems to exhaust it's open
>             file limit
>             after about 7 hours of operation.  Once this happens, the
>             database and
>             curl connections no longer function, and I usually lose the
>             ability to
>             get a CLI connection.  Existing calls usually stay up, until
>             I kill -9
>             and launch again.
> 
>             I'm using FreeSWITCH Version
>             1.2.8+git~20130403T221701Z~79be96aa8e
>             on Centos 2.6.18-348.el5
> 
>             This machine had been running solid for at least 90 days
>             without a hitch
>             - I had almost hit 500K sessions, when this started
>             happening two days
>             ago.  I have changed very little on the machine - the odd
>             dialplan
>             route, and nothing that seems to coincide with that time.
> 
> 
>             It looks like CURL is not closing sockets properly, as
>             netstat shows
>             between 2000 - 4000 http connections in TIME_WAIT state.
>              The number
>             slowly creeps up, then down a little, then up some more
>             until it all
>             comes crashing down.
> 
> 
>         Can you confirm if your web server is responding to the requests?
> 
>         One possible explanation is this spike in traffic caused
>         excessive requests which your web server could not handle, and
>         thus you hit the ulimit.
> 
>         You could increase the ulimit (or set it to unlimited) as per;
>         http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
> 
>         However, if your web server is struggling to keep up with the
>         work load then increasing the max open files will just make the
>         problem worse. Increasing is fine, as long as your web server
>         can keep up with the load, otherwise it will just continue to
>         snowball.
> 
>         Try installing some profiling tools on your web server to
>         monitor the stats, you can use something like New Relic or
>         AppNeta for this, or even just change your access log to output
>         the loading/response times into your access log, then look for
>         high response times.
> 
>         You can also tweak kernel TCP stack settings as per;
>         http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/
> 
>         Hope this helps
>          
> 
> 
>             ls /proc/15742/fd|wc -l currently returns about 400.  It was
>             300 an hour
>             ago.  When it gets to 1024, I'm going to have to restart again.
> 
>             Any suggestions as to where I should look?
> 
>             -Tim
> 
>             _________________________________________________________________________
>             Professional FreeSWITCH Consulting Services:
>             consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>             http://www.freeswitchsolutions.com
> 
>             
>             
> 
>             Official FreeSWITCH Sites
>             http://www.freeswitch.org
>             http://wiki.freeswitch.org
>             http://www.cluecon.com
> 
>             FreeSWITCH-users mailing list
>             FreeSWITCH-users at lists.freeswitch.org
>             <mailto:FreeSWITCH-users at lists.freeswitch.org>
>             http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>             UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>             http://www.freeswitch.org
> 
> 
> 
> 
>     _________________________________________________________________________
>     Professional FreeSWITCH Consulting Services:
>     consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>     http://www.freeswitchsolutions.com
> 
>     
>     
> 
>     Official FreeSWITCH Sites
>     http://www.freeswitch.org
>     http://wiki.freeswitch.org
>     http://www.cluecon.com
> 
>     FreeSWITCH-users mailing list
>     FreeSWITCH-users at lists.freeswitch.org
>     <mailto:FreeSWITCH-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
> 
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>