[Freeswitch-users] mod_xml_curl creating too many open files
Cal Leeming [Simplicity Media Ltd]
cal.leeming at simplicitymedialtd.co.uk
Sun Dec 15 06:18:35 MSK 2013
On Sun, Dec 15, 2013 at 2:05 AM, Tim St. Pierre <
fs-list at communicatefreely.net> wrote:
> Hello everyone,
>
> Thanks for the help on the conference call!
>
> I have tracked down the source of the problem, and it's not at all what
> I expected.
>
> One of our customers has some sort of issue with their phone, and it
> seems that every time the phone connects to SIP/TCP, it opens a new
> socket, and never closes the old one. After two days, there are now
> 3000 established TCP sockets from that customer (to the sofia profile).
>
> I will be paying them a visit on Monday morning. Now that I have
> changed the ulimit settings, this doesn't bring the system down at least.
>
> Is there any way I can limit the number of sockets a specific endpoint
> (or at least a specific source IP) can open?
>
Yup, this can be easily done with iptables;
http://www.cyberciti.biz/faq/iptables-connection-limits-howto/
http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/
https://sites.google.com/site/admin4life/singlehostiptables
You can also do it with most firewall appliances, i.e. if you have a Cisco
ASA in front of your boxes.
>
> I'm hoping to upgrade to a current version, perhaps in the next two
> weeks when everything is quiet. I had 1.2.8 running for nearly 90 days,
> and that's a record for us. Needless to say, I was in no hurry to touch
> anything.
>
> Thanks!
>
>
>
> On 13-12-13 04:50 PM, Anthony Minessale wrote:
> > You still should not be on 1.2.8 =D
> >
> >
> >
> > On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd]
> > <cal.leeming at simplicitymedialtd.co.uk
> > <mailto:cal.leeming at simplicitymedialtd.co.uk>> wrote:
> >
> > As per our discussion on the conference call, this was caused by too
> > many sessions and not enough file descriptors.
> >
> > This can be fixed by changing the ulimit as explained here;
> >
> http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recommended_ULIMIT_settings
> >
> > Cal
> >
> >
> > On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale
> > <anthony.minessale at gmail.com <mailto:anthony.minessale at gmail.com>>
> > wrote:
> >
> > You may want to start by updating to a more recent version to
> > rule out anything already improved.
> > It may be environmental since it started happening after it was
> > once working better but its hard to tell and hard to debug older
> > versions.
> >
> >
> >
> > On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media
> > Ltd] <cal.leeming at simplicitymedialtd.co.uk
> > <mailto:cal.leeming at simplicitymedialtd.co.uk>> wrote:
> >
> > I'll be on the FS conference call for the next 30 mins if
> > you want to ask any questions about this btw.
> >
> > Cal
> >
> >
> > On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity
> > Media Ltd] <cal.leeming at simplicitymedialtd.co.uk
> > <mailto:cal.leeming at simplicitymedialtd.co.uk>> wrote:
> >
> >
> >
> >
> > On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre
> > <fs-list at communicatefreely.net
> > <mailto:fs-list at communicatefreely.net>> wrote:
> >
> > Hello,
> >
> > I'm having a problem where FS seems to exhaust it's
> > open file limit
> > after about 7 hours of operation. Once this
> > happens, the database and
> > curl connections no longer function, and I usually
> > lose the ability to
> > get a CLI connection. Existing calls usually stay
> > up, until I kill -9
> > and launch again.
> >
> > I'm using FreeSWITCH Version
> > 1.2.8+git~20130403T221701Z~79be96aa8e
> > on Centos 2.6.18-348.el5
> >
> > This machine had been running solid for at least 90
> > days without a hitch
> > - I had almost hit 500K sessions, when this started
> > happening two days
> > ago. I have changed very little on the machine -
> > the odd dialplan
> > route, and nothing that seems to coincide with that
> > time.
> >
> >
> > It looks like CURL is not closing sockets properly,
> > as netstat shows
> > between 2000 - 4000 http connections in TIME_WAIT
> > state. The number
> > slowly creeps up, then down a little, then up some
> > more until it all
> > comes crashing down.
> >
> >
> > Can you confirm if your web server is responding to the
> > requests?
> >
> > One possible explanation is this spike in traffic caused
> > excessive requests which your web server could not
> > handle, and thus you hit the ulimit.
> >
> > You could increase the ulimit (or set it to unlimited)
> > as per;
> >
> http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
> >
> > However, if your web server is struggling to keep up
> > with the work load then increasing the max open files
> > will just make the problem worse. Increasing is fine, as
> > long as your web server can keep up with the load,
> > otherwise it will just continue to snowball.
> >
> > Try installing some profiling tools on your web server
> > to monitor the stats, you can use something like New
> > Relic or AppNeta for this, or even just change your
> > access log to output the loading/response times into
> > your access log, then look for high response times.
> >
> > You can also tweak kernel TCP stack settings as per;
> >
> http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-time_wait-state/
> >
> > Hope this helps
> >
> >
> >
> > ls /proc/15742/fd|wc -l currently returns about 400.
> > It was 300 an hour
> > ago. When it gets to 1024, I'm going to have to
> > restart again.
> >
> > Any suggestions as to where I should look?
> >
> > -Tim
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> > FreeSWITCH-powered IP PBX: The CudaTel Communication
> > Server
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> >
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> > --
> > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
> >
> > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
> > http://twitter.com/FreeSWITCH
> > ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞
> > _http://freeswitch.org/g+_
> >
> > ClueCon Weekly Development Call
> > ☎ sip:888 at conference.freeswitch.org
> > <mailto:sip%3A888 at conference.freeswitch.org> ☎ +19193869900
> > <tel:%2B19193869900>
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > <mailto:FreeSWITCH-users at lists.freeswitch.org>
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
> >
> > --
> > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
> >
> > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
> > http://twitter.com/FreeSWITCH
> > ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞
> > _http://freeswitch.org/g+_
> >
> > ClueCon Weekly Development Call
> > ☎ sip:888 at conference.freeswitch.org
> > <mailto:sip%3A888 at conference.freeswitch.org> ☎ +19193869900
> >
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> >
> >
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20131215/756c9375/attachment-0001.html
Join us at ClueCon 2013 Aug 6-8, 2013
More information about the FreeSWITCH-users
mailing list