Thanks for the link! I set some limits so this can't happen again. I also blocked tcp to the internal profile from that client until I can go and fix their phones. Fortunately, all their phones except two are using udp (that ought to narrow it down). I have a Juniper J Series in front of everything, but I seem to recall that there was some reason I couldn't set connection limits there. I use it for all the other firewall stuff though. Thanks for saving my bacon! -Tim On 13-12-14 10:18 PM, Cal Leeming [Simplicity Media Ltd] wrote:
On Sun, Dec 15, 2013 at 2:05 AM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello everyone,
Thanks for the help on the conference call!
I have tracked down the source of the problem, and it's not at all what I expected.
One of our customers has some sort of issue with their phone, and it seems that every time the phone connects to SIP/TCP, it opens a new socket, and never closes the old one. After two days, there are now 3000 established TCP sockets from that customer (to the sofia profile).
I will be paying them a visit on Monday morning. Now that I have changed the ulimit settings, this doesn't bring the system down at least.
Is there any way I can limit the number of sockets a specific endpoint (or at least a specific source IP) can open?
Yup, this can be easily done with iptables;
http://www.cyberciti.biz/faq/iptables-connection-limits-howto/ http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/ https://sites.google.com/site/admin4life/singlehostiptables
You can also do it with most firewall appliances, i.e. if you have a Cisco ASA in front of your boxes.
I'm hoping to upgrade to a current version, perhaps in the next two weeks when everything is quiet. I had 1.2.8 running for nearly 90 days, and that's a record for us. Needless to say, I was in no hurry to touch anything.
Thanks!
On 13-12-13 04:50 PM, Anthony Minessale wrote: > You still should not be on 1.2.8 =D > > > > On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > As per our discussion on the conference call, this was caused by too > many sessions and not enough file descriptors. > > This can be fixed by changing the ulimit as explained here; > http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom... > > Cal > > > On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale > <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com> <mailto:anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>>> > wrote: > > You may want to start by updating to a more recent version to > rule out anything already improved. > It may be environmental since it started happening after it was > once working better but its hard to tell and hard to debug older > versions. > > > > On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media > Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > I'll be on the FS conference call for the next 30 mins if > you want to ask any questions about this btw. > > Cal > > > On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity > Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > > > > On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre > <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net> > <mailto:fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>>> wrote: > > Hello, > > I'm having a problem where FS seems to exhaust it's > open file limit > after about 7 hours of operation. Once this > happens, the database and > curl connections no longer function, and I usually > lose the ability to > get a CLI connection. Existing calls usually stay > up, until I kill -9 > and launch again. > > I'm using FreeSWITCH Version > 1.2.8+git~20130403T221701Z~79be96aa8e > on Centos 2.6.18-348.el5 > > This machine had been running solid for at least 90 > days without a hitch > - I had almost hit 500K sessions, when this started > happening two days > ago. I have changed very little on the machine - > the odd dialplan > route, and nothing that seems to coincide with that > time. > > > It looks like CURL is not closing sockets properly, > as netstat shows > between 2000 - 4000 http connections in TIME_WAIT > state. The number > slowly creeps up, then down a little, then up some > more until it all > comes crashing down. > > > Can you confirm if your web server is responding to the > requests? > > One possible explanation is this spike in traffic caused > excessive requests which your web server could not > handle, and thus you hit the ulimit. > > You could increase the ulimit (or set it to unlimited) > as per; > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files... > > However, if your web server is struggling to keep up > with the work load then increasing the max open files > will just make the problem worse. Increasing is fine, as > long as your web server can keep up with the load, > otherwise it will just continue to snowball. > > Try installing some profiling tools on your web server > to monitor the stats, you can use something like New > Relic or AppNeta for this, or even just change your > access log to output the loading/response times into > your access log, then look for high response times. > > You can also tweak kernel TCP stack settings as per; > http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti... > > Hope this helps > > > > ls /proc/15742/fd|wc -l currently returns about 400. > It was 300 an hour > ago. When it gets to 1024, I'm going to have to > restart again. > > Any suggestions as to where I should look? > > -Tim > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> > <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication > Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net <http://irc.freenode.net> <http://irc.freenode.net> #freeswitch ☞ > _http://freeswitch.org/g+_ > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> > <mailto:sip%3A888@conference.freeswitch.org <mailto:sip%253A888@conference.freeswitch.org>> ☎ +19193869900 <tel:%2B19193869900> > <tel:%2B19193869900> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net <http://irc.freenode.net> <http://irc.freenode.net> #freeswitch ☞ > _http://freeswitch.org/g+_ > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> > <mailto:sip%3A888@conference.freeswitch.org <mailto:sip%253A888@conference.freeswitch.org>> ☎ +19193869900 <tel:%2B19193869900> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org