[Freeswitch-users] Are there any known memory leaks with reloadxml and large XML configs?

Thu Apr 14 21:58:46 MSD 2016

Have you done any debugging like running valgrind or ASAN on a server?
Did you try to reproduce it in a lab yet?

We can't really do any leg work for you so we would need a more specific
reproduction case.
We do offer commercial options as you are probably already are aware of
where we could deploy consultants to do more of the leg work for you.

Do you only reloadxml or do you reload any modules.
Also you should ALWAYS file a jira.  We don't want to field issues on the
list.  Its not a problem to close NOT A BUG if it turns out that way.

On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul at mst.edu> wrote:

> The indication I'm reacting to of leak isn't "one in memory".... The below
> is showing FIFTY THREE of them.
>
> -- Nathan
>
> On 04/14/2016 12:38 PM, Michael Jerris wrote:
> > A parsed xml tree in a core file would be normal.  We always keep one in
> memory, thats why you have to reload to get a new one, so it would be in
> the core file every time, not an indication of a leak.
> >
> >
> >> On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul at mst.edu> wrote:
> >>
> >> I've got a persistent issue with a slow (sometimes not that slow)
> memory leak in FS. Our environment does frequent
> >> reconfigs built around reloadxml and when the leak gets bad enough, I
> start getting random reports of slipping audio
> >> sync. I've put in weekly forced failovers, but if we have a week with
> more changes than typical (we're still in middle
> >> of mass migration off of CCM) - the leak growth can be too fast:
> >>
> >> freeswi+ 17977 12.1 27.2 15247652 2213648 ?    S<l  Apr02 2155:56
> /local/freeswitch/server/bin/freeswitch -u freeswitch
> >> -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf
> /local/freeswitch/server/conf -db
> >> /local/freeswitch/server/db -run /local/freeswitch/data -log
> /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
> >>
> >>
> >> The reason I ask if the leak is related to reloadxml - when doing some
> analysis on the resulting core dump on the (no
> >> longer active) server, I'm finding examples like this when doing a
> 'strings corefile | sort | uniq -c | sort -n':
> >>
> >> ...
> >>    74642 !-- type(skinny) --
> >>    76811 /buttons
> >>    76811 "InvalidHash
> >>    76812 "a1-hash
> >>    76836 /skinny
> >>    76853 buttons
> >> ...
> >>   105277 !-- password won't be used by cisco devices --
> >> ...
> >>   105277 "vm-mailfrom
> >>   105277 "vm-password
> >>
> >> [root at freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml
> >> 1431
> >>
> >> [root at freesw-p1 data]# ls -al *.fsxml
> >> -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28
> freeswitch-prod.xml.fsxml
> >> [root at freesw-p1 data]# wc -l *.fsxml
> >> 105025 freeswitch-prod.xml.fsxml
> >>
> >
> > this is just a copy of the compiled xml file.  it is 100k lines long.
> I'm guessing you have lots of static users in here.
> >
> >>
> >
> > None of this would indicate what you are suggesting necessarily
> >
> >> It's like it's leaking large numbers of complete copies of the XML.
> When I look directly at the core dump, it looks to
> >> me like the strings are in the parsed state of the XML. (Below slightly
> masked copy and paste from viewing dump with less.)
> >
> > Thats what i would accept.
> >
> >>
> >>
> >> -------------
> >> ^@!-- xxx-xxx-xxxx --^@
> >> ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@
> >>      ^@params^@
> >>          ^@!-- password won't be used by cisco devices --^@
> >>          ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@
> >>          ^@param^@name^@"vm-mailfrom^@ value^@"voicemail at mst.edu^@^@^@
> >> -----------
> >>
> >> Would really appreciate any ideas on how I might mitigate this leaking
> or if there is anything that could be done to
> >> help diagnose it further to help address the underlying issue.
> >>
> >>
> >> I'm happy to open a JIRA on this, but will NOT be able to test this
> with latest master as I can't just experiment with
> >> the live production environment.
> >>
> >> -- Nathan
> >>
> >> ------------------------------------------------------------
> >> Nathan Neulinger                       nneul at mst.edu
> >> Missouri S&T Information Technology    (573) 612-1412
> >> System Administrator - Architect
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://confluence.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
>
> --
> ------------------------------------------------------------
> Nathan Neulinger                       nneul at mst.edu
> Missouri S&T Information Technology    (573) 612-1412
> System Administrator - Architect
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160414/e1fd4808/attachment-0001.html 