[Freeswitch-users] mod_xml_curl creating too many open files

[Anthony Minessale]

You still should not be on 1.2.8 =D



On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] <
cal.leeming at simplicitymedialtd.co.uk> wrote:

> As per our discussion on the conference call, this was caused by too many
> sessions and not enough file descriptors.
>
> This can be fixed by changing the ulimit as explained here;
>
> http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recommended_ULIMIT_settings
>
> Cal
>
>
> On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale <
> anthony.minessale at gmail.com> wrote:
>
>> You may want to start by updating to a more recent version to rule out
>> anything already improved.
>> It may be environmental since it started happening after it was once
>> working better but its hard to tell and hard to debug older versions.
>>
>>
>>
>> On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] <
>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>
>>> I'll be on the FS conference call for the next 30 mins if you want to
>>> ask any questions about this btw.
>>>
>>> Cal
>>>
>>>
>>> On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <
>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <
>>>> fs-list at communicatefreely.net> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm having a problem where FS seems to exhaust it's open file limit
>>>>> after about 7 hours of operation.  Once this happens, the database and
>>>>> curl connections no longer function, and I usually lose the ability to
>>>>> get a CLI connection.  Existing calls usually stay up, until I kill -9
>>>>> and launch again.
>>>>>
>>>>> I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e
>>>>> on Centos 2.6.18-348.el5
>>>>>
>>>>> This machine had been running solid for at least 90 days without a
>>>>> hitch
>>>>> - I had almost hit 500K sessions, when this started happening two days
>>>>> ago.  I have changed very little on the machine - the odd dialplan
>>>>> route, and nothing that seems to coincide with that time.
>>>>
>>>>
>>>>> It looks like CURL is not closing sockets properly, as netstat shows
>>>>> between 2000 - 4000 http connections in TIME_WAIT state.  The number
>>>>> slowly creeps up, then down a little, then up some more until it all
>>>>> comes crashing down.
>>>>>
>>>>
>>>> Can you confirm if your web server is responding to the requests?
>>>>
>>>> One possible explanation is this spike in traffic caused excessive
>>>> requests which your web server could not handle, and thus you hit the
>>>> ulimit.
>>>>
>>>> You could increase the ulimit (or set it to unlimited) as per;
>>>>
>>>> http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
>>>>
>>>> However, if your web server is struggling to keep up with the work load
>>>> then increasing the max open files will just make the problem worse.
>>>> Increasing is fine, as long as your web server can keep up with the load,
>>>> otherwise it will just continue to snowball.
>>>>
>>>> Try installing some profiling tools on your web server to monitor the
>>>> stats, you can use something like New Relic or AppNeta for this, or even
>>>> just change your access log to output the loading/response times into your
>>>> access log, then look for high response times.
>>>>
>>>> You can also tweak kernel TCP stack settings as per;
>>>>
>>>> http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/
>>>>
>>>> Hope this helps
>>>>
>>>>
>>>>>
>>>>> ls /proc/15742/fd|wc -l currently returns about 400.  It was 300 an
>>>>> hour
>>>>> ago.  When it gets to 1024, I'm going to have to restart again.
>>>>>
>>>>> Any suggestions as to where I should look?
>>>>>
>>>>> -Tim
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>
>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>> http://twitter.com/FreeSWITCH
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>


-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20131213/5b8184b0/attachment.html