[Freeswitch-users] unexplained RAM usage increase

Tue Feb 12 07:16:01 MSK 2013

Still need some more .. see Jira for questions....
If all else fails maybe we can visit the box on ssh to examine the
situation.

On Mon, Feb 11, 2013 at 7:22 PM, Mario G <mario_fs at mgtech.com> wrote:

> Jeff and Anthony, an update: I have the commit pegged and valgrind log at
> http://jira.freeswitch.org/browse/FS-5095.  Reading Jeff's jira I don't
> think they are related but you never know. Sorry to take so long but git
> bisects resulted in many versions that would not build (I opened about 5
> jiras last 2 months) and I had to hand patch to test. Family emergencies
> took a big toll. I put about 30 hours into this since November and tested
> day/night the last 6 days, have many pages of notes and logs. I hope the
> jira has enough to solve it. BTW, I used the valgrind line below.
> Mario G
>
> On Jan 29, 2013, at 6:15 PM, Anthony Minessale wrote:
>
> If its still there on head, try valgrind.
>
> valgrind --tool=memcheck --log-file=vg.log --leak-check=full
> --leak-resolution=high --show-reachable=yes
> /usr/local/freeswitch/bin/freeswitch -vg
>
>
> On Tue, Jan 29, 2013 at 8:09 PM, Mario G <mario_fs at mgtech.com> wrote:
>
>> I have updated and if by some miracle it's fixed I will report back.
>> Storage increases with no calls so I assume it's registrations
>> (internal/external?). Please understand this is probably this worse week of
>> the year for me to work on FS (or anything else). I promise to try and
>> narrow down the commit that started it. The memory issue started at a bad
>> time when I could not work on FS (emergencies) and by the time I updated
>> other errors crept in that prevented use and I had to open JIRAs for them,
>> so testing memory was out. Thanks for letting me know about todays fix.
>> Mario G
>>
>>
>> On Jan 29, 2013, at 5:41 PM, Anthony Minessale wrote:
>>
>> There is a commit today that found some sql leaks but I would not label
>> them as massive. Its worth it to check and maybe run valgrind.  From our
>> perspective; Any time someone mentions nonchalantly, "Oh yeah I've had this
>> glaring problem for months", breaks our heart =p
>> I was just about to push the tag to unlock the 1.2.6 you so desperately
>> want but now I am halting it to confirm this report.
>>
>> TEST TEST TEST HEAD... Today's head is tomorrow's stable....
>>
>>
>>
>> On Tue, Jan 29, 2013 at 7:23 PM, Ken Rice <krice at freeswitch.org> wrote:
>>
>>>  Mario,
>>>
>>> Is it fixed in the master branch? If it is fixed there it will be rolled
>>> down to 1.2 branch before long...
>>>
>>>
>>> On 1/29/13 6:40 PM, "Mario G" <mario_fs at mgtech.com> wrote:
>>>
>>> Anthony, I tried bisect and the problem is that there are other issues
>>> (RTP error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other
>>> issues in head that prevented it from working for a good amount of time. I
>>> started jiras for them and they are fixed, but doing bisect brings them
>>> back so it's had been really hard to pinpoint the memory issue. Believe me,
>>> I have put a LOT of time into trying to narrow it down before opening a
>>> JIRA and will continue to do so. Right now it's hard since there were
>>> personal emergencies the last 2 months so other pressing things had to take
>>> priority. Still, I am working on FreeSwitch keeping up-to-date with head in
>>> case other issues pop up I can open a JIRA on them.
>>> Mario G
>>>
>>> On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
>>>
>>> Things like this are sad.  We depend on testing and reporting for our
>>> releases.  If you wait months to bring up a problem.  It will spoil the
>>> whole release.
>>>
>>> If you you feel some leak has appeared suddenly, why can't you do git
>>> bisect and find it?
>>>
>>>
>>>
>>> On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>
>>> Probably not be related, but you never know: on OSX since Nov/Dec there
>>> has been a memory leak on 1.2 and head that occurs for in/out/and
>>> registrations. I had to triple memory and recycle FreeSwitch every 2-3 days
>>> since then. Will open a Jira when I can obtain detailed info and possibly
>>> run valgrind. I am also also waiting to update the main FS computer from
>>> 10.6.8 to 10.8.3 to see its effect. Hopefully in Feb.
>>> Mario G
>>>
>>> On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
>>>
>>> Can you systematically increase the current call count and see where you
>>> do see something?
>>> On a system with limited ram you can also consider stripping all the .so
>>> files in the mods and lib dir but you will need to put non-stripped ones in
>>> for any debugging.
>>>
>>> I don't really see a correlation on how not using a timer could trigger
>>> an sustained memory increase so that's why I'd like you to step up the
>>> number and see if you can find a number of calls that tops out because
>>> usually there always is a magic number where it will hover and go up and
>>> down a meg at a time.
>>>
>>> SIP calls are required to keep state data around for at least 30 seconds
>>> after a call ends and there are a number of pools in the code that inflate
>>> once and do not return the memory.  Its usually possible to identify the
>>> high watermark on a particular box.
>>>
>>> For instance the machine we host the conference call on launches using
>>> 25 megs and hovers at about 350 megs once it has accumulated all the pool
>>> memory it needs over time.
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle at fidelityvoice.com>
>>> wrote:
>>>
>>> Version:
>>>   FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1
>>> (-n20130122T122521Z-1~squeeze+1)
>>>
>>> The calls are bridged, from one sofia profile to another.
>>>
>>> Unfortunately two concurrent calls doesn't seem to trigger the same
>>> behavior.
>>>
>>>
>>> - Jeff
>>>
>>>
>>> On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice at freeswitch.org> wrote:
>>>
>>> Ok the package timestamps/versions there don’t do us a lot of good, we
>>> need to know the version line from the FreeSWITCH CLI..
>>>
>>>
>>> On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle at fidelityvoice.com <
>>> http://jpyle@fidelityvoice.com/> > wrote:
>>>
>>> I just updated from repo.profhost.eu <http://repo.profhost.eu/>  <
>>> http://repo.profhost.eu <http://repo.profhost.eu/> > .  The most recent
>>> timestamp on the packages was 2013-01-28 03:41:21 GMT.  Same behavior.  At
>>> 5 minutes it was using 12.5% RAM.  At 40 minutes, 60.4%.  After
>>> disconnecting the calls the usage returned to 8.5%.
>>>
>>>
>>>
>>> I started toggling config items to see if I could impact this.  I found
>>> one that seems to have an effect:  rtp-timer-name in the sofia profile
>>> config.  By changing it from 'soft' to 'none', the CPU utilization with 30
>>> calls dropped from ~70% to ~46%, and the RAM usage is rock solid at 5.8%.
>>>
>>> That's great, but does it make any sense?
>>>
>>> Does an rtp-timer-name of 'none' pose any risks?
>>>
>>>
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle at fidelityvoice.com <
>>> http://jpyle@fidelityvoice.com/> > wrote:
>>>
>>> It's on Voyage Linux, a cousin of Debian.  I believe it uses glibc.
>>>
>>>
>>> - Jeff
>>>
>>>
>>> On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris at kriskinc.com<
>>> http://kris@kriskinc.com/> > wrote:
>>> Out of curiosity does your distro use uclibc, eglibc, or glibc?
>>>
>>> On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle at fidelityvoice.com <
>>> http://jpyle@fidelityvoice.com/> > wrote:
>>> > Hello,
>>> >
>>> > I'm running HEAD version from Jan 22 on an Alix board with an AMD
>>> Geode LX
>>> > processor (i386).  I can sustain 30 concurrent calls averaging around
>>> 70%
>>> > CPU utilization by the freeswitch process, measured by top.  Bypass
>>> media
>>> > and proxy media are disabled.  PCMU is forced on both endpoints (no
>>> > transcoding).
>>> >
>>> > The problem is the RAM usage over time.  The board has 256M.  Idle,
>>> > freeswitch occupies around 4% after a fresh restart.  A minute or so
>>> after
>>> > 30 calls are nailed up the RAM usage is about 7.2%.  After 5 minutes,
>>> 13.6%.
>>> > After 60 minutes, near 65%.  Disconnecting the calls returns the RAM
>>> usage
>>> > to 6-8%.
>>> >
>>> > I've not tried to troubleshoot an issue like this before.  Is valgrind
>>> the
>>> > next step, or would something else make more sense?
>>> >
>>> >
>>> > Regards,
>>> > Jeff
>>>
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>
>>>
>>> 
>>>  </>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>> http://wiki.freeswitch.org <http://wiki.freeswitch.org/>
>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>
>>>
>>>
>>>
>>> --
>>> Ken
>>> *http://www.FreeSWITCH.org
>>> http://www.ClueCon.com
>>> http://www.OSTAG.org
>>> *irc.freenode.net #freeswitch
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>> IRC: irc.freenode.net #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org
>> googletalk:conf+888 at conference.freeswitch.org
>> pstn:+19193869900
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire

AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130211/3bce52e9/attachment-0001.html 