[Freeswitch-users] unexplained RAM usage increase

[Anthony Minessale]

If its still there on head, try valgrind.

valgrind --tool=memcheck --log-file=vg.log --leak-check=full
--leak-resolution=high --show-reachable=yes
/usr/local/freeswitch/bin/freeswitch -vg


On Tue, Jan 29, 2013 at 8:09 PM, Mario G <mario_fs at mgtech.com> wrote:

> I have updated and if by some miracle it's fixed I will report back.
> Storage increases with no calls so I assume it's registrations
> (internal/external?). Please understand this is probably this worse week of
> the year for me to work on FS (or anything else). I promise to try and
> narrow down the commit that started it. The memory issue started at a bad
> time when I could not work on FS (emergencies) and by the time I updated
> other errors crept in that prevented use and I had to open JIRAs for them,
> so testing memory was out. Thanks for letting me know about todays fix.
> Mario G
>
>
> On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
>
> There is a commit today that found some sql leaks but I would not label
> them as massive. Its worth it to check and maybe run valgrind.  From our
> perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this
> glaring problem for months", breaks our heart =p
> I was just about to push the tag to unlock the 1.2.6 you so desperately
> want but now I am halting it to confirm this report.
>
> TEST TEST TEST HEAD... Today's head is tomorrow's stable....
>
>
>
> On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice at freeswitch.org> wrote:
>
>>  Mario,
>>
>> Is it fixed in the master branch? If it is fixed there it will be rolled
>> down to 1.2 branch before long...
>>
>>
>> On 1/29/13 6:40 PM, "Mario G" <mario_fs at mgtech.com> wrote:
>>
>> Anthony, I tried bisect and the problem is that there are other issues
>> (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other
>> issues in head that prevented it from working for a good amount of time. I
>> started jiras for them and they are fixed, but doing bisect brings them
>> back so it's had been really hard to pinpoint the memory issue. Believe me,
>> I have put a LOT of time into trying to narrow it down before opening a
>> JIRA and will continue to do so. Right now it's hard since there were
>> personal emergencies the last 2 months so other pressing things had to take
>> priority. Still, I am working on FreeSwitch keeping up-to-date with head in
>> case other issues pop up I can open a JIRA on them.
>> Mario G
>>
>> On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
>>
>> Things like this are sad.  We depend on testing and reporting for our
>> releases.  If you wait months to bring up a problem.  It will spoil the
>> whole release.
>>
>> If you you feel some leak has appeared suddenly, why can't you do git
>> bisect and find it?
>>
>>
>>
>> On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs at mgtech.com> wrote:
>>
>> Probably not be related, but you never know: on OSX since Nov/Dec there
>> has been a memory leak on 1.2 and head that occurs for in/out/and
>> registrations. I had to triple memory and recycle FreeSwitch every 2-3 days
>> since then. Will open a Jira when I can obtain detailed info and possibly
>> run valgrind. I am also also waiting to update the main FS computer from
>> 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb.
>> Mario G
>>
>> On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
>>
>> Can you systematically increase the current call count and see where you
>> do see something?
>> On a system with limited ram you can also consider stripping all the .so
>> files in the mods and lib dir but you will need to put non-stripped ones in
>> for any debugging.
>>
>> I don't really see a correlation on how not using a timer could trigger
>> an sustained memory increase so that's why I'd like you to step up the
>> number and see if you can find a number of calls that tops out because
>> usually there always is a magic number where it will hover and go up and
>> down a meg at a time.
>>
>> SIP calls are required to keep state data around for at least 30 seconds
>> after a call ends and there are a number of pools in the code that inflate
>> once and do not return the memory.  Its usually possible to identify the
>> high watermark on a particular box.
>>
>> For instance the machine we host the conference call on launches using 25
>> megs and hovers at about 350 megs once it has accumulated all the pool
>> memory it needs over time.
>>
>>
>>
>>
>>
>> On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle at fidelityvoice.com>
>> wrote:
>>
>> Version:
>>   FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1
>> (-n20130122T122521Z-1~squeeze+1)
>>
>> The calls are bridged, from one sofia profile to another.
>>
>> Unfortunately two concurrent calls doesn't seem to trigger the same
>> behavior.
>>
>>
>> - Jeff
>>
>>
>> On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice at freeswitch.org> wrote:
>>
>> Ok the package timestamps/versions there don’t do us a lot of good, we
>> need to know the version line from the FreeSWITCH CLI..
>>
>>
>> On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle at fidelityvoice.com <
>> http://jpyle@fidelityvoice.com/> > wrote:
>>
>> I just updated from repo.profhost.eu <http://repo.profhost.eu/>  <
>> http://repo.profhost.eu <http://repo.profhost.eu/> > .  The most recent
>> timestamp on the packages was 2013-01-28 03:41:21 GMT.  Same behavior.  At
>> 5 minutes it was using 12.5% RAM.  At 40 minutes, 60.4%.  After
>> disconnecting the calls the usage returned to 8.5%.
>>
>>
>>
>> I started toggling config items to see if I could impact this.  I found
>> one that seems to have an effect:  rtp-timer-name in the sofia profile
>> config.  By changing it from 'soft' to 'none', the CPU utilization with 30
>> calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
>>
>> That's great, but does it make any sense?
>>
>> Does an rtp-timer-name of 'none' pose any risks?
>>
>>
>>
>> - Jeff
>>
>>
>>
>> On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle at fidelityvoice.com <
>> http://jpyle@fidelityvoice.com/> > wrote:
>>
>> It's on Voyage Linux, a cousin of Debian.  I believe it uses glibc.
>>
>>
>> - Jeff
>>
>>
>> On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris at kriskinc.com <
>> http://kris@kriskinc.com/> > wrote:
>> Out of curiosity does your distro use uclibc, eglibc, or glibc?
>>
>> On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle at fidelityvoice.com <
>> http://jpyle@fidelityvoice.com/> > wrote:
>> > Hello,
>> >
>> > I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode
>> LX
>> > processor (i386).  I can sustain 30 concurrent calls averaging around
>> 70%
>> > CPU utilization by the freeswitch process, measured by top.  Bypass
>> media
>> > and proxy media are disabled.  PCMU is forced on both endpoints (no
>> > transcoding).
>> >
>> > The problem is the RAM usage over time.  The board has 256M.  Idle,
>> > freeswitch occupies around 4% after a fresh restart.  A minute or so
>> after
>> > 30 calls are nailed up the RAM usage is about 7.2%.  After 5 minutes,
>> 13.6%.
>> > After 60 minutes, near 65%.  Disconnecting the calls returns the RAM
>> usage
>> > to 6-8%.
>> >
>> > I've not tried to troubleshoot an issue like this before.  Is valgrind
>> the
>> > next step, or would something else make more sense?
>> >
>> >
>> > Regards,
>> > Jeff
>>
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>
>> 
>>  </>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> http://wiki.freeswitch.org <http://wiki.freeswitch.org/>
>> http://www.cluecon.com <http://www.cluecon.com/>
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>
>>
>>
>>
>> --
>> Ken
>> *http://www.FreeSWITCH.org
>> http://www.ClueCon.com
>> http://www.OSTAG.org
>> *irc.freenode.net #freeswitch
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>


-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130129/5a1f429d/attachment-0001.html