[Freeswitch-users] mod_xml_curl creating too many open files

[Lloyd Aloysius]

Tim

I would first check the connectivity

Switch ports?

Lloyd
On Dec 13, 2013 4:14 PM, "Cal Leeming [Simplicity Media Ltd]" <
cal.leeming at simplicitymedialtd.co.uk> wrote:

> I'll be on the FS conference call for the next 30 mins if you want to ask
> any questions about this btw.
>
> Cal
>
>
> On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:
>
>>
>>
>>
>> On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <
>> fs-list at communicatefreely.net> wrote:
>>
>>> Hello,
>>>
>>> I'm having a problem where FS seems to exhaust it's open file limit
>>> after about 7 hours of operation.  Once this happens, the database and
>>> curl connections no longer function, and I usually lose the ability to
>>> get a CLI connection.  Existing calls usually stay up, until I kill -9
>>> and launch again.
>>>
>>> I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e
>>> on Centos 2.6.18-348.el5
>>>
>>> This machine had been running solid for at least 90 days without a hitch
>>> - I had almost hit 500K sessions, when this started happening two days
>>> ago.  I have changed very little on the machine - the odd dialplan
>>> route, and nothing that seems to coincide with that time.
>>
>>
>>> It looks like CURL is not closing sockets properly, as netstat shows
>>> between 2000 - 4000 http connections in TIME_WAIT state.  The number
>>> slowly creeps up, then down a little, then up some more until it all
>>> comes crashing down.
>>>
>>
>> Can you confirm if your web server is responding to the requests?
>>
>> One possible explanation is this spike in traffic caused excessive
>> requests which your web server could not handle, and thus you hit the
>> ulimit.
>>
>> You could increase the ulimit (or set it to unlimited) as per;
>>
>> http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
>>
>> However, if your web server is struggling to keep up with the work load
>> then increasing the max open files will just make the problem worse.
>> Increasing is fine, as long as your web server can keep up with the load,
>> otherwise it will just continue to snowball.
>>
>> Try installing some profiling tools on your web server to monitor the
>> stats, you can use something like New Relic or AppNeta for this, or even
>> just change your access log to output the loading/response times into your
>> access log, then look for high response times.
>>
>> You can also tweak kernel TCP stack settings as per;
>>
>> http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/
>>
>> Hope this helps
>>
>>
>>>
>>> ls /proc/15742/fd|wc -l currently returns about 400.  It was 300 an hour
>>> ago.  When it gets to 1024, I'm going to have to restart again.
>>>
>>> Any suggestions as to where I should look?
>>>
>>> -Tim
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20131213/05f316d5/attachment.html