[Freeswitch-users] unexplained RAM usage increase

Wed Jan 30 05:09:49 MSK 2013

I have updated and if by some miracle it's fixed I will report back. Storage increases with no calls so I assume it's registrations (internal/external?). Please understand this is probably this worse week of the year for me to work on FS (or anything else). I promise to try and narrow down the commit that started it. The memory issue started at a bad time when I could not work on FS (emergencies) and by the time I updated other errors crept in that prevented use and I had to open JIRAs for them, so testing memory was out. Thanks for letting me know about todays fix.
Mario G

On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:

> There is a commit today that found some sql leaks but I would not label them as massive. Its worth it to check and maybe run valgrind.  From our perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this glaring problem for months", breaks our heart =p
> I was just about to push the tag to unlock the 1.2.6 you so desperately want but now I am halting it to confirm this report.
> 
> TEST TEST TEST HEAD... Today's head is tomorrow's stable....
> 
> 
> 
> On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice at freeswitch.org> wrote:
> Mario,
> 
> Is it fixed in the master branch? If it is fixed there it will be rolled down to 1.2 branch before long... 
> 
> 
> On 1/29/13 6:40 PM, "Mario G" <mario_fs at mgtech.com> wrote:
> 
> Anthony, I tried bisect and the problem is that there are other issues (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in head that prevented it from working for a good amount of time. I started jiras for them and they are fixed, but doing bisect brings them back so it's had been really hard to pinpoint the memory issue. Believe me, I have put a LOT of time into trying to narrow it down before opening a JIRA and will continue to do so. Right now it's hard since there were personal emergencies the last 2 months so other pressing things had to take priority. Still, I am working on FreeSwitch keeping up-to-date with head in case other issues pop up I can open a JIRA on them.
> Mario G
> 
> On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
> 
> Things like this are sad.  We depend on testing and reporting for our releases.  If you wait months to bring up a problem.  It will spoil the whole release.
> 
> If you you feel some leak has appeared suddenly, why can't you do git bisect and find it?
> 
> 
> 
> On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs at mgtech.com> wrote:
> Probably not be related, but you never know: on OSX since Nov/Dec there has been a memory leak on 1.2 and head that occurs for in/out/and registrations. I had to triple memory and recycle FreeSwitch every 2-3 days since then. Will open a Jira when I can obtain detailed info and possibly run valgrind. I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb.
> Mario G 
> 
> On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
> 
> Can you systematically increase the current call count and see where you do see something?
> On a system with limited ram you can also consider stripping all the .so files in the mods and lib dir but you will need to put non-stripped ones in for any debugging.
> 
> I don't really see a correlation on how not using a timer could trigger an sustained memory increase so that's why I'd like you to step up the number and see if you can find a number of calls that tops out because usually there always is a magic number where it will hover and go up and down a meg at a time.
> 
> SIP calls are required to keep state data around for at least 30 seconds after a call ends and there are a number of pools in the code that inflate once and do not return the memory.  Its usually possible to identify the high watermark on a particular box.
> 
> For instance the machine we host the conference call on launches using 25 megs and hovers at about 350 megs once it has accumulated all the pool memory it needs over time.
> 
> 
>  
> 
> 
> On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle at fidelityvoice.com> wrote:
> Version:
>   FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1 (-n20130122T122521Z-1~squeeze+1)
> 
> The calls are bridged, from one sofia profile to another.
> 
> Unfortunately two concurrent calls doesn't seem to trigger the same behavior.
> 
> 
> - Jeff
> 
> 
> On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice at freeswitch.org> wrote:
> Ok the package timestamps/versions there don’t do us a lot of good, we need to know the version line from the FreeSWITCH CLI..
> 
> 
> On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle at fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
> 
> I just updated from repo.profhost.eu <http://repo.profhost.eu/>  <http://repo.profhost.eu <http://repo.profhost.eu/> > .  The most recent timestamp on the packages was 2013-01-28 03:41:21 GMT.  Same behavior.  At 5 minutes it was using 12.5% RAM.  At 40 minutes, 60.4%.  After disconnecting the calls the usage returned to 8.5%.
> 
> 
> 
> I started toggling config items to see if I could impact this.  I found one that seems to have an effect:  rtp-timer-name in the sofia profile config.  By changing it from 'soft' to 'none', the CPU utilization with 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
> 
> That's great, but does it make any sense?
> 
> Does an rtp-timer-name of 'none' pose any risks?
> 
> 
> 
> - Jeff
>   
> 
> 
> On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle at fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
> It's on Voyage Linux, a cousin of Debian.  I believe it uses glibc.
> 
> 
> - Jeff
> 
> 
> On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris at kriskinc.com <http://kris@kriskinc.com/> > wrote:
> Out of curiosity does your distro use uclibc, eglibc, or glibc?
> 
> On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle at fidelityvoice.com <http://jpyle@fidelityvoice.com/> > wrote:
> > Hello,
> >
> > I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX
> > processor (i386).  I can sustain 30 concurrent calls averaging around 70%
> > CPU utilization by the freeswitch process, measured by top.  Bypass media
> > and proxy media are disabled.  PCMU is forced on both endpoints (no
> > transcoding).
> >
> > The problem is the RAM usage over time.  The board has 256M.  Idle,
> > freeswitch occupies around 4% after a fresh restart.  A minute or so after
> > 30 calls are nailed up the RAM usage is about 7.2%.  After 5 minutes, 13.6%.
> > After 60 minutes, near 65%.  Disconnecting the calls returns the RAM usage
> > to 6-8%.
> >
> > I've not tried to troubleshoot an issue like this before.  Is valgrind the
> > next step, or would something else make more sense?
> >
> >
> > Regards,
> > Jeff 
> 
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> 
> 
> 
>  </> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org <http://www.freeswitch.org/> 
> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> 
> http://www.cluecon.com <http://www.cluecon.com/> 
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org <http://www.freeswitch.org/> 
> 
> 
> 
> 
> -- 
> Ken
> http://www.FreeSWITCH.org
> http://www.ClueCon.com
> http://www.OSTAG.org
> irc.freenode.net #freeswitch
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
> 
> 
> 
> 
> -- 
> Anthony Minessale II
> 
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
> 
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
> 
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130129/f66be240/attachment.html 