unexplained RAM usage increase
Hello, I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding). The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%. I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense? Regards, Jeff
Out of curiosity does your distro use uclibc, eglibc, or glibc? On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Kristian Kielhofner
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc. - Jeff On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com>wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
I just updated from repo.profhost.eu. The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%. I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%. That's great, but does it make any sense? Does an rtp-timer-name of 'none' pose any risks? - Jeff On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com>wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
What are the calls doing? Bridged? to an IVR? If so, which one. If you do a smaller number of calls does the same thing happen. If you can do it with 1 or 2 calls you could use valgrind. valgrind --tool=memcheck --log-file-exactly=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch On Mon, Jan 28, 2013 at 9:59 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu. The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com>wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Ok the package timestamps/versions there don¹t do us a lot of good, we need to know the version line from the FreeSWITCH CLI.. On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1) The calls are bridged, from one sofia profile to another. Unfortunately two concurrent calls doesn't seem to trigger the same behavior. - Jeff On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging. I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time. SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box. For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time. On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote: Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release. If you you feel some leak has appeared suddenly, why can't you do git bisect and find it? On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote: Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote: Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Mario, Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long... On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don¹t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
> I just updated from repo.profhost.eu <http://repo.profhost.eu/> > <http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent > timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. > At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After > disconnecting the calls the usage returned to 8.5%. > > > I started toggling config items to see if I could impact this. I found > one that seems to have an effect: rtp-timer-name in the sofia profile > config. By changing it from 'soft' to 'none', the CPU utilization with > 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at > 5.8%. > > That's great, but does it make any sense? > > Does an rtp-timer-name of 'none' pose any risks? > > > > - Jeff > > > > On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com > <http://jpyle@fidelityvoice.com/> > wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com <http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc? On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report. TEST TEST TEST HEAD... Today's head is tomorrow's stable.... On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice@freeswitch.org> wrote:
Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu/> < http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com < http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix. Mario G On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.
TEST TEST TEST HEAD... Today's head is tomorrow's stable....
On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice@freeswitch.org> wrote: Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote: Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote: Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu/> <http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com <http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If its still there on head, try valgrind. valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg On Tue, Jan 29, 2013 at 8:09 PM, Mario G <mario_fs@mgtech.com> wrote:
I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix. Mario G
On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.
TEST TEST TEST HEAD... Today's head is tomorrow's stable....
On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice@freeswitch.org> wrote:
Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu/> < http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com < http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Jeff and Anthony, an update: I have the commit pegged and valgrind log at http://jira.freeswitch.org/browse/FS-5095. Reading Jeff's jira I don't think they are related but you never know. Sorry to take so long but git bisects resulted in many versions that would not build (I opened about 5 jiras last 2 months) and I had to hand patch to test. Family emergencies took a big toll. I put about 30 hours into this since November and tested day/night the last 6 days, have many pages of notes and logs. I hope the jira has enough to solve it. BTW, I used the valgrind line below. Mario G On Jan 29, 2013, at 6:15 PM, Anthony Minessale wrote:
If its still there on head, try valgrind.
valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg
On Tue, Jan 29, 2013 at 8:09 PM, Mario G <mario_fs@mgtech.com> wrote: I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix. Mario G
On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.
TEST TEST TEST HEAD... Today's head is tomorrow's stable....
On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice@freeswitch.org> wrote: Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote: Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote: Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote: Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu/> <http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com <http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Still need some more .. see Jira for questions.... If all else fails maybe we can visit the box on ssh to examine the situation. On Mon, Feb 11, 2013 at 7:22 PM, Mario G <mario_fs@mgtech.com> wrote:
Jeff and Anthony, an update: I have the commit pegged and valgrind log at http://jira.freeswitch.org/browse/FS-5095. Reading Jeff's jira I don't think they are related but you never know. Sorry to take so long but git bisects resulted in many versions that would not build (I opened about 5 jiras last 2 months) and I had to hand patch to test. Family emergencies took a big toll. I put about 30 hours into this since November and tested day/night the last 6 days, have many pages of notes and logs. I hope the jira has enough to solve it. BTW, I used the valgrind line below. Mario G
On Jan 29, 2013, at 6:15 PM, Anthony Minessale wrote:
If its still there on head, try valgrind.
valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg
On Tue, Jan 29, 2013 at 8:09 PM, Mario G <mario_fs@mgtech.com> wrote:
I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix. Mario G
On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.
TEST TEST TEST HEAD... Today's head is tomorrow's stable....
On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice@freeswitch.org> wrote:
Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu/> < http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com< http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com < http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Ken, I assume you mean the RTP error (FS 4900 4921, 4907), yes it has been fixed for a while. I thought I read somewhere it will go into 1.2.6. Sorry to go off-topic on this thread. Mario G On Jan 29, 2013, at 5:23 PM, Ken Rice wrote:
Mario,
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long...
On 1/29/13 6:40 PM, "Mario G" <mario_fs@mgtech.com> wrote:
Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them. Mario G
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs@mgtech.com> wrote:
Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb. Mario G
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote: > Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI.. > > > On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote: > >> I just updated from repo.profhost.eu <http://repo.profhost.eu/> <http://repo.profhost.eu <http://repo.profhost.eu/> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%. >> >> >> I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%. >> >> That's great, but does it make any sense? >> >> Does an rtp-timer-name of 'none' pose any risks? >> >> >> >> - Jeff >> >> >> >> On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote: It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com <http://kris@kriskinc.com/> > wrote: Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/>
Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/>
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/>
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I gave you bad information before. It does happen with just one or two calls, it just takes a long time. Valgrind was not an option on the embedded box I was able to reproduce the issue on a regular x86_64 machine running squeeze, FreeSWITCH Version 1.3.4-n20130128T034121Z-1~squeeze+1 (-n20130128T034121Z-1~squeeze+1). I have the vg.log file produced with two calls up for approximately 35 minutes. No other SIP activity occurred during that time period. Is it Jira time, or is there other data you'd like to see? - Jeff On Tue, Jan 29, 2013 at 12:04 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Were are you getting the package from? 1.3 is on 13 right now so 4 is a bit too old. Can you build HEAD from source and reproduce a log you can post to jira? On Tue, Jan 29, 2013 at 5:15 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
I gave you bad information before. It does happen with just one or two calls, it just takes a long time. Valgrind was not an option on the embedded box I was able to reproduce the issue on a regular x86_64 machine running squeeze, FreeSWITCH Version 1.3.4-n20130128T034121Z-1~squeeze+1 (-n20130128T034121Z-1~squeeze+1).
I have the vg.log file produced with two calls up for approximately 35 minutes. No other SIP activity occurred during that time period. Is it Jira time, or is there other data you'd like to see?
- Jeff
On Tue, Jan 29, 2013 at 12:04 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org> wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I installed via apt-get from http://repo.profhost.eu/debian. The same issue occurs on a fresh download from HEAD, yielding v1.3.13b. Jira FS-5065 <http://jira.freeswitch.org/browse/FS-5065>. Log uploads to follow once some calls have had enough time to generate some data. - Jeff On Tue, Jan 29, 2013 at 6:32 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Were are you getting the package from? 1.3 is on 13 right now so 4 is a bit too old. Can you build HEAD from source and reproduce a log you can post to jira?
On Tue, Jan 29, 2013 at 5:15 PM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
I gave you bad information before. It does happen with just one or two calls, it just takes a long time. Valgrind was not an option on the embedded box I was able to reproduce the issue on a regular x86_64 machine running squeeze, FreeSWITCH Version 1.3.4-n20130128T034121Z-1~squeeze+1 (-n20130128T034121Z-1~squeeze+1).
I have the vg.log file produced with two calls up for approximately 35 minutes. No other SIP activity occurred during that time period. Is it Jira time, or is there other data you'd like to see?
- Jeff
On Tue, Jan 29, 2013 at 12:04 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Can you systematically increase the current call count and see where you do see something? On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle@fidelityvoice.com>wrote:
Version: FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
The calls are bridged, from one sofia profile to another.
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
- Jeff
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice@freeswitch.org>wrote:
Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle@fidelityvoice.com> wrote:
I just updated from repo.profhost.eu <http://repo.profhost.eu> . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
That's great, but does it make any sense?
Does an rtp-timer-name of 'none' pose any risks?
- Jeff
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.
- Jeff
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner < kris@kriskinc.com> wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle@fidelityvoice.com> wrote:
Hello,
I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX processor (i386). I can sustain 30 concurrent calls averaging around 70% CPU utilization by the freeswitch process, measured by top. Bypass media and proxy media are disabled. PCMU is forced on both endpoints (no transcoding).
The problem is the RAM usage over time. The board has 256M. Idle, freeswitch occupies around 4% after a fresh restart. A minute or so after 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%. After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage to 6-8%.
I've not tried to troubleshoot an issue like this before. Is valgrind the next step, or would something else make more sense?
Regards, Jeff
participants (5)
-
Anthony Minessale
-
Jeff Pyle
-
Ken Rice
-
Kristian Kielhofner
-
Mario G