[Freeswitch-users] Are there any known memory leaks with reloadxml and large XML configs?

Thu Apr 14 22:18:11 MSD 2016

P.S.
Check all the modules you use to make sure when they open the XML registry
that they close it again.

On Thu, Apr 14, 2016 at 12:58 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:

> Have you done any debugging like running valgrind or ASAN on a server?
> Did you try to reproduce it in a lab yet?
>
> We can't really do any leg work for you so we would need a more specific
> reproduction case.
> We do offer commercial options as you are probably already are aware of
> where we could deploy consultants to do more of the leg work for you.
>
> Do you only reloadxml or do you reload any modules.
> Also you should ALWAYS file a jira.  We don't want to field issues on the
> list.  Its not a problem to close NOT A BUG if it turns out that way.
>
>
>
>
>
>
> On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul at mst.edu> wrote:
>
>> The indication I'm reacting to of leak isn't "one in memory".... The
>> below is showing FIFTY THREE of them.
>>
>> -- Nathan
>>
>> On 04/14/2016 12:38 PM, Michael Jerris wrote:
>> > A parsed xml tree in a core file would be normal.  We always keep one
>> in memory, thats why you have to reload to get a new one, so it would be in
>> the core file every time, not an indication of a leak.
>> >
>> >
>> >> On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul at mst.edu> wrote:
>> >>
>> >> I've got a persistent issue with a slow (sometimes not that slow)
>> memory leak in FS. Our environment does frequent
>> >> reconfigs built around reloadxml and when the leak gets bad enough, I
>> start getting random reports of slipping audio
>> >> sync. I've put in weekly forced failovers, but if we have a week with
>> more changes than typical (we're still in middle
>> >> of mass migration off of CCM) - the leak growth can be too fast:
>> >>
>> >> freeswi+ 17977 12.1 27.2 15247652 2213648 ?    S<l  Apr02 2155:56
>> /local/freeswitch/server/bin/freeswitch -u freeswitch
>> >> -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server
>> -conf /local/freeswitch/server/conf -db
>> >> /local/freeswitch/server/db -run /local/freeswitch/data -log
>> /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
>> >>
>> >>
>> >> The reason I ask if the leak is related to reloadxml - when doing some
>> analysis on the resulting core dump on the (no
>> >> longer active) server, I'm finding examples like this when doing a
>> 'strings corefile | sort | uniq -c | sort -n':
>> >>
>> >> ...
>> >>    74642 !-- type(skinny) --
>> >>    76811 /buttons
>> >>    76811 "InvalidHash
>> >>    76812 "a1-hash
>> >>    76836 /skinny
>> >>    76853 buttons
>> >> ...
>> >>   105277 !-- password won't be used by cisco devices --
>> >> ...
>> >>   105277 "vm-mailfrom
>> >>   105277 "vm-password
>> >>
>> >> [root at freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml
>> >> 1431
>> >>
>> >> [root at freesw-p1 data]# ls -al *.fsxml
>> >> -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28
>> freeswitch-prod.xml.fsxml
>> >> [root at freesw-p1 data]# wc -l *.fsxml
>> >> 105025 freeswitch-prod.xml.fsxml
>> >>
>> >
>> > this is just a copy of the compiled xml file.  it is 100k lines long.
>> I'm guessing you have lots of static users in here.
>> >
>> >>
>> >
>> > None of this would indicate what you are suggesting necessarily
>> >
>> >> It's like it's leaking large numbers of complete copies of the XML.
>> When I look directly at the core dump, it looks to
>> >> me like the strings are in the parsed state of the XML. (Below
>> slightly masked copy and paste from viewing dump with less.)
>> >
>> > Thats what i would accept.
>> >
>> >>
>> >>
>> >> -------------
>> >> ^@!-- xxx-xxx-xxxx --^@
>> >> ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@
>> >>      ^@params^@
>> >>          ^@!-- password won't be used by cisco devices --^@
>> >>          ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@
>> >>          ^@param^@name^@"vm-mailfrom^@ value^@"voicemail at mst.edu^@^@^@
>> >> -----------
>> >>
>> >> Would really appreciate any ideas on how I might mitigate this leaking
>> or if there is anything that could be done to
>> >> help diagnose it further to help address the underlying issue.
>> >>
>> >>
>> >> I'm happy to open a JIRA on this, but will NOT be able to test this
>> with latest master as I can't just experiment with
>> >> the live production environment.
>> >>
>> >> -- Nathan
>> >>
>> >> ------------------------------------------------------------
>> >> Nathan Neulinger                       nneul at mst.edu
>> >> Missouri S&T Information Technology    (573) 612-1412
>> >> System Administrator - Architect
>> >>
>> >>
>> _________________________________________________________________________
>> >> Professional FreeSWITCH Consulting Services:
>> >> consulting at freeswitch.org
>> >> http://www.freeswitchsolutions.com
>> >>
>> >> Official FreeSWITCH Sites
>> >> http://www.freeswitch.org
>> >> http://confluence.freeswitch.org
>> >> http://www.cluecon.com
>> >>
>> >> FreeSWITCH-users mailing list
>> >> FreeSWITCH-users at lists.freeswitch.org
>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >> UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >> http://www.freeswitch.org
>> >
>> >
>> >
>> _________________________________________________________________________
>> > Professional FreeSWITCH Consulting Services:
>> > consulting at freeswitch.org
>> > http://www.freeswitchsolutions.com
>> >
>> > Official FreeSWITCH Sites
>> > http://www.freeswitch.org
>> > http://confluence.freeswitch.org
>> > http://www.cluecon.com
>> >
>> > FreeSWITCH-users mailing list
>> > FreeSWITCH-users at lists.freeswitch.org
>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> > UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> > http://www.freeswitch.org
>> >
>>
>> --
>> ------------------------------------------------------------
>> Nathan Neulinger                       nneul at mst.edu
>> Missouri S&T Information Technology    (573) 612-1412
>> System Administrator - Architect
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
> https://www.youtube.com/watch?v=9XXgW34t40s
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160414/1a03038c/attachment-0001.html 