mod_xml_curl creating too many open files
Hello, I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again. I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5 This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time. It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down. ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again. Any suggestions as to where I should look? -Tim
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests? One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit. You could increase the ulimit (or set it to unlimited) as per; http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files... However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball. Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times. You can also tweak kernel TCP stack settings as per; http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti... Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw. Cal On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Tim I would first check the connectivity Switch ports? Lloyd On Dec 13, 2013 4:14 PM, "Cal Leeming [Simplicity Media Ltd]" < cal.leeming@simplicitymedialtd.co.uk> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I thought that too, but I ran a ping test against the machine (to that interface) when it happened. It didn't lose a single packet. -Tim On 13-12-13 04:19 PM, Lloyd Aloysius wrote:
Tim
I would first check the connectivity
Switch ports?
Lloyd
On Dec 13, 2013 4:14 PM, "Cal Leeming [Simplicity Media Ltd]" <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per; http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per; http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
There are for sure some important performance issues between 1.2.8 and latest stable. There even more in master. On Fri, Dec 13, 2013 at 3:26 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
I thought that too, but I ran a ping test against the machine (to that interface) when it happened. It didn't lose a single packet.
-Tim
On 13-12-13 04:19 PM, Lloyd Aloysius wrote:
Tim
I would first check the connectivity
Switch ports?
Lloyd
On Dec 13, 2013 4:14 PM, "Cal Leeming [Simplicity Media Ltd]" <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart
again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
You may want to start by updating to a more recent version to rule out anything already improved. It may be environmental since it started happening after it was once working better but its hard to tell and hard to debug older versions. On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
As per our discussion on the conference call, this was caused by too many sessions and not enough file descriptors. This can be fixed by changing the ulimit as explained here; http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom... Cal On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
You may want to start by updating to a more recent version to rule out anything already improved. It may be environmental since it started happening after it was once working better but its hard to tell and hard to debug older versions.
On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
You still should not be on 1.2.8 =D On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
As per our discussion on the conference call, this was caused by too many sessions and not enough file descriptors.
This can be fixed by changing the ulimit as explained here;
http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom...
Cal
On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
You may want to start by updating to a more recent version to rule out anything already improved. It may be environmental since it started happening after it was once working better but its hard to tell and hard to debug older versions.
On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Hello everyone, Thanks for the help on the conference call! I have tracked down the source of the problem, and it's not at all what I expected. One of our customers has some sort of issue with their phone, and it seems that every time the phone connects to SIP/TCP, it opens a new socket, and never closes the old one. After two days, there are now 3000 established TCP sockets from that customer (to the sofia profile). I will be paying them a visit on Monday morning. Now that I have changed the ulimit settings, this doesn't bring the system down at least. Is there any way I can limit the number of sockets a specific endpoint (or at least a specific source IP) can open? I'm hoping to upgrade to a current version, perhaps in the next two weeks when everything is quiet. I had 1.2.8 running for nearly 90 days, and that's a record for us. Needless to say, I was in no hurry to touch anything. Thanks! On 13-12-13 04:50 PM, Anthony Minessale wrote:
You still should not be on 1.2.8 =D
On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
As per our discussion on the conference call, this was caused by too many sessions and not enough file descriptors.
This can be fixed by changing the ulimit as explained here; http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom...
Cal
On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> wrote:
You may want to start by updating to a more recent version to rule out anything already improved. It may be environmental since it started happening after it was once working better but its hard to tell and hard to debug older versions.
On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per; http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per; http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ _http://freeswitch.org/g+_
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:%2B19193869900>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ _http://freeswitch.org/g+_
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Sun, Dec 15, 2013 at 2:05 AM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hello everyone,
Thanks for the help on the conference call!
I have tracked down the source of the problem, and it's not at all what I expected.
One of our customers has some sort of issue with their phone, and it seems that every time the phone connects to SIP/TCP, it opens a new socket, and never closes the old one. After two days, there are now 3000 established TCP sockets from that customer (to the sofia profile).
I will be paying them a visit on Monday morning. Now that I have changed the ulimit settings, this doesn't bring the system down at least.
Is there any way I can limit the number of sockets a specific endpoint (or at least a specific source IP) can open?
Yup, this can be easily done with iptables; http://www.cyberciti.biz/faq/iptables-connection-limits-howto/ http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/ https://sites.google.com/site/admin4life/singlehostiptables You can also do it with most firewall appliances, i.e. if you have a Cisco ASA in front of your boxes.
I'm hoping to upgrade to a current version, perhaps in the next two weeks when everything is quiet. I had 1.2.8 running for nearly 90 days, and that's a record for us. Needless to say, I was in no hurry to touch anything.
Thanks!
On 13-12-13 04:50 PM, Anthony Minessale wrote:
You still should not be on 1.2.8 =D
On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
As per our discussion on the conference call, this was caused by too many sessions and not enough file descriptors.
This can be fixed by changing the ulimit as explained here;
http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom...
Cal
On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>> wrote:
You may want to start by updating to a more recent version to rule out anything already improved. It may be environmental since it started happening after it was once working better but its hard to tell and hard to debug older versions.
On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
I'll be on the FS conference call for the next 30 mins if you want to ask any questions about this btw.
Cal
On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>> wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per;
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per;
http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ _http://freeswitch.org/g+_
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:%2B19193869900>
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ _http://freeswitch.org/g+_
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks for the link! I set some limits so this can't happen again. I also blocked tcp to the internal profile from that client until I can go and fix their phones. Fortunately, all their phones except two are using udp (that ought to narrow it down). I have a Juniper J Series in front of everything, but I seem to recall that there was some reason I couldn't set connection limits there. I use it for all the other firewall stuff though. Thanks for saving my bacon! -Tim On 13-12-14 10:18 PM, Cal Leeming [Simplicity Media Ltd] wrote:
On Sun, Dec 15, 2013 at 2:05 AM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello everyone,
Thanks for the help on the conference call!
I have tracked down the source of the problem, and it's not at all what I expected.
One of our customers has some sort of issue with their phone, and it seems that every time the phone connects to SIP/TCP, it opens a new socket, and never closes the old one. After two days, there are now 3000 established TCP sockets from that customer (to the sofia profile).
I will be paying them a visit on Monday morning. Now that I have changed the ulimit settings, this doesn't bring the system down at least.
Is there any way I can limit the number of sockets a specific endpoint (or at least a specific source IP) can open?
Yup, this can be easily done with iptables;
http://www.cyberciti.biz/faq/iptables-connection-limits-howto/ http://www.mauromascia.com/en/blog/limiting-concurrent-connections-per-ip/ https://sites.google.com/site/admin4life/singlehostiptables
You can also do it with most firewall appliances, i.e. if you have a Cisco ASA in front of your boxes.
I'm hoping to upgrade to a current version, perhaps in the next two weeks when everything is quiet. I had 1.2.8 running for nearly 90 days, and that's a record for us. Needless to say, I was in no hurry to touch anything.
Thanks!
On 13-12-13 04:50 PM, Anthony Minessale wrote: > You still should not be on 1.2.8 =D > > > > On Fri, Dec 13, 2013 at 3:45 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > As per our discussion on the conference call, this was caused by too > many sessions and not enough file descriptors. > > This can be fixed by changing the ulimit as explained here; > http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#Recom... > > Cal > > > On Fri, Dec 13, 2013 at 9:24 PM, Anthony Minessale > <anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com> <mailto:anthony.minessale@gmail.com <mailto:anthony.minessale@gmail.com>>> > wrote: > > You may want to start by updating to a more recent version to > rule out anything already improved. > It may be environmental since it started happening after it was > once working better but its hard to tell and hard to debug older > versions. > > > > On Fri, Dec 13, 2013 at 3:11 PM, Cal Leeming [Simplicity Media > Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > I'll be on the FS conference call for the next 30 mins if > you want to ask any questions about this btw. > > Cal > > > On Fri, Dec 13, 2013 at 9:10 PM, Cal Leeming [Simplicity > Media Ltd] <cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk> > <mailto:cal.leeming@simplicitymedialtd.co.uk <mailto:cal.leeming@simplicitymedialtd.co.uk>>> wrote: > > > > > On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre > <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net> > <mailto:fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>>> wrote: > > Hello, > > I'm having a problem where FS seems to exhaust it's > open file limit > after about 7 hours of operation. Once this > happens, the database and > curl connections no longer function, and I usually > lose the ability to > get a CLI connection. Existing calls usually stay > up, until I kill -9 > and launch again. > > I'm using FreeSWITCH Version > 1.2.8+git~20130403T221701Z~79be96aa8e > on Centos 2.6.18-348.el5 > > This machine had been running solid for at least 90 > days without a hitch > - I had almost hit 500K sessions, when this started > happening two days > ago. I have changed very little on the machine - > the odd dialplan > route, and nothing that seems to coincide with that > time. > > > It looks like CURL is not closing sockets properly, > as netstat shows > between 2000 - 4000 http connections in TIME_WAIT > state. The number > slowly creeps up, then down a little, then up some > more until it all > comes crashing down. > > > Can you confirm if your web server is responding to the > requests? > > One possible explanation is this spike in traffic caused > excessive requests which your web server could not > handle, and thus you hit the ulimit. > > You could increase the ulimit (or set it to unlimited) > as per; > http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files... > > However, if your web server is struggling to keep up > with the work load then increasing the max open files > will just make the problem worse. Increasing is fine, as > long as your web server can keep up with the load, > otherwise it will just continue to snowball. > > Try installing some profiling tools on your web server > to monitor the stats, you can use something like New > Relic or AppNeta for this, or even just change your > access log to output the loading/response times into > your access log, then look for high response times. > > You can also tweak kernel TCP stack settings as per; > http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti... > > Hope this helps > > > > ls /proc/15742/fd|wc -l currently returns about 400. > It was 300 an hour > ago. When it gets to 1024, I'm going to have to > restart again. > > Any suggestions as to where I should look? > > -Tim > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> > <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication > Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net <http://irc.freenode.net> <http://irc.freenode.net> #freeswitch ☞ > _http://freeswitch.org/g+_ > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> > <mailto:sip%3A888@conference.freeswitch.org <mailto:sip%253A888@conference.freeswitch.org>> ☎ +19193869900 <tel:%2B19193869900> > <tel:%2B19193869900> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net <http://irc.freenode.net> <http://irc.freenode.net> #freeswitch ☞ > _http://freeswitch.org/g+_ > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> > <mailto:sip%3A888@conference.freeswitch.org <mailto:sip%253A888@conference.freeswitch.org>> ☎ +19193869900 <tel:%2B19193869900> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org <mailto:consulting@freeswitch.org> > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, it seems to be holding up okay. It will quit at 5am, when there is nobody on the phone. It seems to be very predictable. We have about 500 phones registering every 20 minutes. The webserver gets 2-3 requests per second, occasionally with bursts of 20/s, but it seems to be more than capable of handling it. If the load drops off, should the sockets not eventually time out and close? That's an easy test to do. I just hopped into the conference. -Tim On 13-12-13 04:10 PM, Cal Leeming [Simplicity Media Ltd] wrote:
On Fri, Dec 13, 2013 at 8:37 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a problem where FS seems to exhaust it's open file limit after about 7 hours of operation. Once this happens, the database and curl connections no longer function, and I usually lose the ability to get a CLI connection. Existing calls usually stay up, until I kill -9 and launch again.
I'm using FreeSWITCH Version 1.2.8+git~20130403T221701Z~79be96aa8e on Centos 2.6.18-348.el5
This machine had been running solid for at least 90 days without a hitch - I had almost hit 500K sessions, when this started happening two days ago. I have changed very little on the machine - the odd dialplan route, and nothing that seems to coincide with that time.
It looks like CURL is not closing sockets properly, as netstat shows between 2000 - 4000 http connections in TIME_WAIT state. The number slowly creeps up, then down a little, then up some more until it all comes crashing down.
Can you confirm if your web server is responding to the requests?
One possible explanation is this spike in traffic caused excessive requests which your web server could not handle, and thus you hit the ulimit.
You could increase the ulimit (or set it to unlimited) as per; http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files...
However, if your web server is struggling to keep up with the work load then increasing the max open files will just make the problem worse. Increasing is fine, as long as your web server can keep up with the load, otherwise it will just continue to snowball.
Try installing some profiling tools on your web server to monitor the stats, you can use something like New Relic or AppNeta for this, or even just change your access log to output the loading/response times into your access log, then look for high response times.
You can also tweak kernel TCP stack settings as per; http://www.cyberciti.biz/faq/linux-command-forcibly-close-socket-ports-in-ti...
Hope this helps
ls /proc/15742/fd|wc -l currently returns about 400. It was 300 an hour ago. When it gets to 1024, I'm going to have to restart again.
Any suggestions as to where I should look?
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (4)
-
Anthony Minessale
-
Cal Leeming [Simplicity Media Ltd]
-
Lloyd Aloysius
-
Tim St. Pierre