[Freeswitch-users] unexplained RAM usage increase

Wed Jan 30 04:23:01 MSK 2013

Mario,

Is it fixed in the master branch? If it is fixed there it will be rolled
down to 1.2 branch before long...

On 1/29/13 6:40 PM, "Mario G" <mario_fs at mgtech.com> wrote:

> Anthony, I tried bisect and the problem is that there are other issues (RTP
> error still in 1.2 stable, waiting for 1.2.6), and about 4-5 other issues in
> head that prevented it from working for a good amount of time. I started jiras
> for them and they are fixed, but doing bisect brings them back so it's had
> been really hard to pinpoint the memory issue. Believe me, I have put a LOT of
> time into trying to narrow it down before opening a JIRA and will continue to
> do so. Right now it's hard since there were personal emergencies the last 2
> months so other pressing things had to take priority. Still, I am working on
> FreeSwitch keeping up-to-date with head in case other issues pop up I can open
> a JIRA on them.
> Mario G
> 
> On Jan 29, 2013, at 3:14 PM, Anthony Minessale wrote:
> 
>> Things like this are sad.  We depend on testing and reporting for our
>> releases.  If you wait months to bring up a problem.  It will spoil the whole
>> release.
>> 
>> If you you feel some leak has appeared suddenly, why can't you do git bisect
>> and find it?
>> 
>> 
>> 
>> On Tue, Jan 29, 2013 at 12:58 PM, Mario G <mario_fs at mgtech.com> wrote:
>>> Probably not be related, but you never know: on OSX since Nov/Dec there has
>>> been a memory leak on 1.2 and head that occurs for in/out/and registrations.
>>> I had to triple memory and recycle FreeSwitch every 2-3 days since then.
>>> Will open a Jira when I can obtain detailed info and possibly run valgrind.
>>> I am also also waiting to update the main FS computer from 10.6.8 to 10.8.3
>>> to see its effect. Hopefully in Feb.
>>> Mario G 
>>> 
>>> On Jan 29, 2013, at 9:04 AM, Anthony Minessale wrote:
>>> 
>>>> Can you systematically increase the current call count and see where you do
>>>> see something?
>>>> On a system with limited ram you can also consider stripping all the .so
>>>> files in the mods and lib dir but you will need to put non-stripped ones in
>>>> for any debugging.
>>>> 
>>>> I don't really see a correlation on how not using a timer could trigger an
>>>> sustained memory increase so that's why I'd like you to step up the number
>>>> and see if you can find a number of calls that tops out because usually
>>>> there always is a magic number where it will hover and go up and down a meg
>>>> at a time.
>>>> 
>>>> SIP calls are required to keep state data around for at least 30 seconds
>>>> after a call ends and there are a number of pools in the code that inflate
>>>> once and do not return the memory.  Its usually possible to identify the
>>>> high watermark on a particular box.
>>>> 
>>>> For instance the machine we host the conference call on launches using 25
>>>> megs and hovers at about 350 megs once it has accumulated all the pool
>>>> memory it needs over time.
>>>> 
>>>> 
>>>>  
>>>> 
>>>> 
>>>> On Tue, Jan 29, 2013 at 8:33 AM, Jeff Pyle <jpyle at fidelityvoice.com> wrote:
>>>>> Version:
>>>>>   FreeSWITCH version: 1.3.4-n20130122T122521Z-1~squeeze+1
>>>>> (-n20130122T122521Z-1~squeeze+1)
>>>>> 
>>>>> The calls are bridged, from one sofia profile to another.
>>>>> 
>>>>> Unfortunately two concurrent calls doesn't seem to trigger the same
>>>>> behavior.
>>>>> 
>>>>> 
>>>>> - Jeff
>>>>> 
>>>>> 
>>>>> On Mon, Jan 28, 2013 at 11:21 PM, Ken Rice <krice at freeswitch.org> wrote:
>>>>>> Ok the package timestamps/versions there don¹t do us a lot of good, we
>>>>>> need to know the version line from the FreeSWITCH CLI..
>>>>>> 
>>>>>> 
>>>>>> On 1/28/13 9:59 PM, "Jeff Pyle" <jpyle at fidelityvoice.com
>>>>>> <http://jpyle@fidelityvoice.com/> > wrote:
>>>>>> 
>>>>>>> I just updated from repo.profhost.eu <http://repo.profhost.eu/>
>>>>>>> <http://repo.profhost.eu <http://repo.profhost.eu/> > .  The most recent
>>>>>>> timestamp on the packages was 2013-01-28 03:41:21 GMT.  Same behavior.
>>>>>>> At 5 minutes it was using 12.5% RAM.  At 40 minutes, 60.4%.  After
>>>>>>> disconnecting the calls the usage returned to 8.5%.
>>>>>>> 
>>>>>>> 
>>>>>>> I started toggling config items to see if I could impact this.  I found
>>>>>>> one that seems to have an effect:  rtp-timer-name in the sofia profile
>>>>>>> config.  By changing it from 'soft' to 'none', the CPU utilization with
>>>>>>> 30 calls dropped from ~70% to ~46%, and the RAM usage is rock solid at
>>>>>>> 5.8%.
>>>>>>> 
>>>>>>> That's great, but does it make any sense?
>>>>>>> 
>>>>>>> Does an rtp-timer-name of 'none' pose any risks?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> - Jeff
>>>>>>>   
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Jan 28, 2013 at 8:43 PM, Jeff Pyle <jpyle at fidelityvoice.com
>>>>>>> <http://jpyle@fidelityvoice.com/> > wrote:
It's on Voyage Linux, a cousin of Debian.  I believe it uses glibc.

- Jeff

On Mon, Jan 28, 2013 at 6:45 PM, Kristian Kielhofner <kris at kriskinc.com
<http://kris@kriskinc.com/> > wrote:
Out of curiosity does your distro use uclibc, eglibc, or glibc?

On Mon, Jan 28, 2013 at 5:41 PM, Jeff Pyle <jpyle at fidelityvoice.com
<http://jpyle@fidelityvoice.com/> > wrote:
> Hello,
>
> I'm running HEAD version from Jan 22 on an Alix board with an AMD Geode LX
> processor (i386).  I can sustain 30 concurrent calls averaging around 70%
> CPU utilization by the freeswitch process, measured by top.  Bypass media
> and proxy media are disabled.  PCMU is forced on both endpoints (no
> transcoding).
>
> The problem is the RAM usage over time.  The board has 256M.  Idle,
> freeswitch occupies around 4% after a fresh restart.  A minute or so after
> 30 calls are nailed up the RAM usage is about 7.2%.  After 5 minutes, 13.6%.
> After 60 minutes, near 65%.  Disconnecting the calls returns the RAM usage
> to 6-8%.
>
> I've not tried to troubleshoot an issue like this before.  Is valgrind the
> next step, or would something else make more sense?
>
>
> Regards,
> Jeff 
>>>>>> 
>>>>> 
>>>>> 
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>> 
>>>>> 
>>>>>  </>
>>>>> 
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>> http://wiki.freeswitch.org <http://wiki.freeswitch.org/>
>>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>> 
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>> 
>>>> 
>>>> 

-- 
Ken
http://www.FreeSWITCH.org
http://www.ClueCon.com
http://www.OSTAG.org
irc.freenode.net #freeswitch

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130129/2376c926/attachment-0001.html 