<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix.<div>Mario G<br><div><br><div><div>On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind. From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p<div style="">
I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.</div><div style=""><br></div><div style="">TEST TEST TEST HEAD... Today's head is tomorrow's stable....</div>
<div style=""><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <span dir="ltr"><<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<font face="Monaco, Courier New"><span style="font-size:11pt">Mario,<br>
<br>
Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long... <br><div class="im">
<br>
<br>
On 1/29/13 6:40 PM, "Mario G" <<a href="http://mario_fs@mgtech.com/" target="_blank">mario_fs@mgtech.com</a>> wrote:<br>
<br>
</div></span></font><blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt">Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them.<br>
Mario G<br>
<br>
On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:<br>
<br>
</span></font></div><blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt">Things like this are sad. We depend on testing and reporting for our releases. If you wait months to bring up a problem. It will spoil the whole release.<br>
<br>
If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?<br>
<br>
<br>
<br>
On Tue, Jan 29, 2013 at 12:58 PM, Mario G <<a href="http://mario_fs@mgtech.com/" target="_blank">mario_fs@mgtech.com</a>> wrote:<br>
</span></font></div><blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt">Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb.<br>
<font color="#888888">Mario G <br>
</font><br>
On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:<br>
<br>
</span></font></div><blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt">Can you systematically increase the current call count and see where you do see something?<br>
On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.<br>
<br>
I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.<br>
<br>
SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory. Its usually possible to identify the high watermark on a particular box.<br>
<br>
For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.<br>
<br>
<br>
<br>
<br>
<br>
On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <<a href="http://jpyle@fidelityvoice.com/" target="_blank">jpyle@fidelityvoice.com</a>> wrote:<br>
</span></font></div><blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt">Version:<br>
FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)<br>
<br>
The calls are bridged, from one sofia profile to another.<br>
<br>
Unfortunately two concurrent calls doesn't seem to trigger the same behavior.<br>
<font color="#888888"><br>
<br>
- Jeff<br>
</font><br>
<br>
On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <<a href="http://krice@freeswitch.org/" target="_blank">krice@freeswitch.org</a>> wrote:<br>
</span></font></div><blockquote><font face="Monaco, Courier New"><span style="font-size:11pt"><div class="im">Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..<br>
<br>
<br></div>
On 1/28/13 9:59 PM, "Jeff Pyle" <<a href="http://jpyle@fidelityvoice.com/" target="_blank">jpyle@fidelityvoice.com</a> <<a href="http://jpyle@fidelityvoice.com/" target="_blank">http://jpyle@fidelityvoice.com/</a>> > wrote:<br>
<br>
</span></font><blockquote><font face="Monaco, Courier New"><span style="font-size:11pt">I just updated from <a href="http://repo.profhost.eu/" target="_blank">repo.profhost.eu</a> <<a href="http://repo.profhost.eu/" target="_blank">http://repo.profhost.eu/</a>> <<a href="http://repo.profhost.eu/" target="_blank">http://repo.profhost.eu</a> <<a href="http://repo.profhost.eu/" target="_blank">http://repo.profhost.eu/</a>> > . The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT. Same behavior. At 5 minutes it was using 12.5% RAM. At 40 minutes, 60.4%. After disconnecting the calls the usage returned to 8.5%.<div class="im">
<br>
<br>
<br>
I started toggling config items to see if I could impact this. I found one that seems to have an effect: rtp-timer-name in the sofia profile config. By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.<br>
<br>
That's great, but does it make any sense?<br>
<br>
Does an rtp-timer-name of 'none' pose any risks?<br>
<br>
<br>
<br>
- Jeff<br>
<br>
<br>
<br></div><div class="im">
On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <<a href="http://jpyle@fidelityvoice.com/" target="_blank">jpyle@fidelityvoice.com</a> <<a href="http://jpyle@fidelityvoice.com/" target="_blank">http://jpyle@fidelityvoice.com/</a>> > wrote:<br>
</div></span></font></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><font face="Monaco, Courier New"><span style="font-size:11pt"><div class="im">It's on Voyage Linux, a cousin of Debian. I believe it uses glibc.<br>
<font color="#888888"><br>
<br>
- Jeff<br>
</font><br>
<br></div><div class="im">
On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <<a href="http://kris@kriskinc.com/" target="_blank">kris@kriskinc.com</a> <<a href="http://kris@kriskinc.com/" target="_blank">http://kris@kriskinc.com/</a>> > wrote:<br>
Out of curiosity does your distro use uclibc, eglibc, or glibc?<br>
<br></div><div class="im">
On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <<a href="http://jpyle@fidelityvoice.com/" target="_blank">jpyle@fidelityvoice.com</a> <<a href="http://jpyle@fidelityvoice.com/" target="_blank">http://jpyle@fidelityvoice.com/</a>> > wrote:<br>
> Hello,<br>
><br>
> I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX<br>
> processor (i386). I can sustain 30 concurrent calls averaging around 70%<br>
> CPU utilization by the freeswitch process, measured by top. Bypass media<br>
> and proxy media are disabled. PCMU is forced on both endpoints (no<br>
> transcoding).<br>
><br>
> The problem is the RAM usage over time. The board has 256M. Idle,<br>
> freeswitch occupies around 4% after a fresh restart. A minute or so after<br>
> 30 calls are nailed up the RAM usage is about 7.2%. After 5 minutes, 13.6%.<br>
> After 60 minutes, near 65%. Disconnecting the calls returns the RAM usage<br>
> to 6-8%.<br>
><br>
> I've not tried to troubleshoot an issue like this before. Is valgrind the<br>
> next step, or would something else make more sense?<br>
><br>
><br>
> Regards,<br>
> Jeff <br>
</div></span></font><blockquote><blockquote><blockquote><blockquote><blockquote><blockquote><font face="Monaco, Courier New"><span style="font-size:11pt"><br>
</span></font></blockquote><font face="Monaco, Courier New"><span style="font-size:11pt"><div class="im"><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="http://consulting@freeswitch.org/" target="_blank">consulting@freeswitch.org</a><br>
</div><a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a> <<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com/</a>> <br>
<div class="im">
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
</div><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a> <<a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com/</a>> <br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a> <<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a>> <br>
<a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a> <<a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org/</a>> <br>
<a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a> <<a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a>> <br><div class="im">
<br>
FreeSWITCH-users mailing list<br>
<a href="http://FreeSWITCH-users@lists.freeswitch.org/" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
</div><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a> <<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a>> <br>
<br>
</span></font></blockquote><font face="Monaco, Courier New"><span style="font-size:11pt"><br>
<br>
</span></font></blockquote></blockquote></blockquote></blockquote><div class="im"><font face="Monaco, Courier New"><span style="font-size:11pt"><br>
-- <br>
Ken<br>
<font color="#0000FF"><u><a href="http://www.FreeSWITCH.org/" target="_blank">http://www.FreeSWITCH.org</a><br>
<a href="http://www.ClueCon.com/" target="_blank">http://www.ClueCon.com</a><br>
<a href="http://www.OSTAG.org/" target="_blank">http://www.OSTAG.org</a><br>
</u></font><a href="http://irc.freenode.net/" target="_blank">irc.freenode.net</a> #freeswitch<br>
</span></font>
</div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire">http://twitter.com/FreeSWITCH_wire</a><br><br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net/">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900
</div>
_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>http://www.freeswitchsolutions.com<br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>http://www.cudatel.com<br><br>Official FreeSWITCH Sites<br>http://www.freeswitch.org<br>http://wiki.freeswitch.org<br>http://www.cluecon.com<br><br>FreeSWITCH-users mailing list<br>FreeSWITCH-users@lists.freeswitch.org<br>http://lists.freeswitch.org/mailman/listinfo/freeswitch-users<br>UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users<br>http://www.freeswitch.org<br></blockquote></div><br></div></div></body></html>