I am running the most current version of Freeswitch from git on Ubuntu Linux 9.10 64 bit. I have a pair of these systems that are handling no more than 12 simultaneous calls each. Over time, Freeswitch begins to take up more and more CPU, and after about 2 weeks, it is using up so much CPU that call quality starts to degrade. It will be churning away even when there are no active calls. A restart of Freeswitch cures the problem for a couple of weeks. Can someone give me pointers on where to start debugging this issue? Or perhaps a better remedy than a restart? Thanks! Kevin Raison
if you are running the most current version of FS but it takes 2 weeks to notice the problem then the newest your FS could be today is 2 weeks old? =D are you getting your updates from GIT? What I recommend is when you have the problem where there is noticeable CPU usage, run top -H and look at the cpu usage on a per-thread basis. When you find the one that is using a lot of CPU, make note of it. Then get a gcore and dump it into GDB so you can see what the thread in question is doing.. you can get a gcore from the fs build root with ./support-d/fscore_pb gcore <yourname> Then compare the results on the url provided from that script with the info from top -H On Fri, Aug 26, 2011 at 2:45 PM, Kevin Raison <raison@chatsubo.net> wrote:
I am running the most current version of Freeswitch from git on Ubuntu Linux 9.10 64 bit. I have a pair of these systems that are handling no more than 12 simultaneous calls each. Over time, Freeswitch begins to take up more and more CPU, and after about 2 weeks, it is using up so much CPU that call quality starts to degrade. It will be churning away even when there are no active calls. A restart of Freeswitch cures the problem for a couple of weeks. Can someone give me pointers on where to start debugging this issue? Or perhaps a better remedy than a restart?
Thanks!
Kevin Raison
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
if you are running the most current version of FS but it takes 2 weeks to notice the problem then the newest your FS could be today is 2 weeks old? =D are you getting your updates from GIT?
I have gone through several iterations of pulling new updates from git, rebuilding and restarting freeswitch and waiting while the CPU usage rises. I will rebuild again today before I restart as well. Hopefully the problem will just go away this time. Otherwise, I will let you know what I find using your method; thanks for the help! -Kevin
What I recommend is when you have the problem where there is noticeable CPU usage, run top -H and look at the cpu usage on a per-thread basis. When you find the one that is using a lot of CPU, make note of it. Then get a gcore and dump it into GDB so you can see what the thread in question is doing..
you can get a gcore from the fs build root with ./support-d/fscore_pb gcore<yourname>
Then compare the results on the url provided from that script with the info from top -H
what about the recipe i gave you to figure it out? you can just send me the results if you want me to examine it as long as you can collect it. On Fri, Aug 26, 2011 at 3:41 PM, Kevin Raison <raison@chatsubo.net> wrote:
if you are running the most current version of FS but it takes 2 weeks to notice the problem then the newest your FS could be today is 2 weeks old? =D are you getting your updates from GIT?
I have gone through several iterations of pulling new updates from git, rebuilding and restarting freeswitch and waiting while the CPU usage rises. I will rebuild again today before I restart as well. Hopefully the problem will just go away this time. Otherwise, I will let you know what I find using your method; thanks for the help!
-Kevin
What I recommend is when you have the problem where there is noticeable CPU usage, run top -H and look at the cpu usage on a per-thread basis. When you find the one that is using a lot of CPU, make note of it. Then get a gcore and dump it into GDB so you can see what the thread in question is doing..
you can get a gcore from the fs build root with ./support-d/fscore_pb gcore<yourname>
Then compare the results on the url provided from that script with the info from top -H
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Thanks, Anthony, but I was able to determine that a mod_event_socket-based app was not properly closing connections to Freeswitch. The Freeswitch threads that were eating up CPU were those handling the external application's unclosed sessions. Now that the external app is closing connections properly, I am seeing no more CLOSED_WAIT connection states, and hopefully as the week progresses, Freeswitch won't eat up CPU as it was before. Cheers and thanks again for the pointers, Kevin On 08/26/2011 01:06 PM, Anthony Minessale wrote:
if you are running the most current version of FS but it takes 2 weeks to notice the problem then the newest your FS could be today is 2 weeks old? =D are you getting your updates from GIT?
What I recommend is when you have the problem where there is noticeable CPU usage, run top -H and look at the cpu usage on a per-thread basis. When you find the one that is using a lot of CPU, make note of it. Then get a gcore and dump it into GDB so you can see what the thread in question is doing..
you can get a gcore from the fs build root with ./support-d/fscore_pb gcore<yourname>
Then compare the results on the url provided from that script with the info from top -H
On Fri, Aug 26, 2011 at 2:45 PM, Kevin Raison<raison@chatsubo.net> wrote:
I am running the most current version of Freeswitch from git on Ubuntu Linux 9.10 64 bit. I have a pair of these systems that are handling no more than 12 simultaneous calls each. Over time, Freeswitch begins to take up more and more CPU, and after about 2 weeks, it is using up so much CPU that call quality starts to degrade. It will be churning away even when there are no active calls. A restart of Freeswitch cures the problem for a couple of weeks. Can someone give me pointers on where to start debugging this issue? Or perhaps a better remedy than a restart?
Thanks!
Kevin Raison
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (2)
-
Anthony Minessale
-
Kevin Raison