[Freeswitch-users] Are there any known memory leaks with reloadxml and large XML configs?

Fri Apr 15 17:37:15 MSD 2016

Thanks for the assistance. Your additional suggestion of where to look led
me right to an additional memory leak in mod_skinny - and was as a result
able to reproduce a scenario that could trigger it. I don't know for sure
that it fixes the symptom I've seen on production, but I suspect this is at
least a significant part of it. A fix has been pushed into master for it.

JIRA issue updated accordingly.

-- Nathan

On Thu, Apr 14, 2016 at 11:41 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:

> Why are you still discussing this here when you have a jira open?  I am
> not sure how to properly explain the IMMENSE difficulty of working on bugs
> over email.  There is no right reason to side with email when trying to
> decide between jira and email.  If a jira turns out not to be a bug, one
> click to close it.  When an email turns out to be a bug, frick, now we have
> to copy all the data over.  Pass it on, file jiras when you have a
> problem.  We have broadcasted it here for years..:.....
>
> The xml registry is ref counted if it reloads while its open, it probably
> means something you use opens the xml root and does not close it.  I said
> this on your jira.
>
>
> On Thursday, April 14, 2016, Nathan Neulinger <nneul at mst.edu> wrote:
>
>> Running some more post analysis over the raw content of the core dump,
>> and extracting each parsed blob of XML, it looks
>> like almost all of them are complete copies (and not partial like if it
>> ran out of room and had to resize+relocate).
>>
>> I extracted all of the chunks to individual files, and looking at each
>> one in turn based on sequential position in
>> memory, I am seeing reasonably expected small/tiny changes from one to
>> the next that would correspond to incremental
>> changes in our configuration UI (stuff like individual changes to a line
>> key, button position, voicemail address,
>> password, etc.)
>>
>>
>> What in the system results in a full copy of the parsed XML structure?
>> Does every access do some sort of temporary copy
>> like you'd get with SQL and transaction isolation?
>>
>>
>> -- Nathan
>>
>> On 04/14/2016 01:25 PM, Nathan Neulinger wrote:
>> > FS-9074 opened.
>> >
>> > I've attempted to reproduce in lab environment, but not with much
>> success since I can't readily reproduce the
>> > activity/usage level from the production environment. On production
>> env, I can readily see the resident memory growing
>> > when I reloadxml on the active server, but it doesn't do it every time,
>> so it's clearly some sort of combination of
>> > events/timing related.
>> >
>> > On the lab setup, I typically see about 3 x duplicates of the strings
>> from the XML in a generated core.
>> >
>> > No modules are being reloaded during normal operations. Just a rewrite
>> of the configs followed by a reloadxml.
>> >
>> >
>> > As far as the commercial option - have asked y'all several times for
>> what our options would be for that sort of service,
>> > but have never gotten any clear answer particularly given the custom
>> deployment we have. Would be very interested in
>> > presenting that to management here if you can give me some proposed
>> service offerings and costs.
>> >
>> > -- Nathan
>> >
>> > On 04/14/2016 12:58 PM, Anthony Minessale wrote:
>> >> Have you done any debugging like running valgrind or ASAN on a server?
>> >> Did you try to reproduce it in a lab yet?
>> >>
>> >> We can't really do any leg work for you so we would need a more
>> specific reproduction case.
>> >> We do offer commercial options as you are probably already are aware
>> of where we could deploy consultants to do more of
>> >> the leg work for you.
>> >>
>> >> Do you only reloadxml or do you reload any modules.
>> >> Also you should ALWAYS file a jira.  We don't want to field issues on
>> the list.  Its not a problem to close NOT A BUG if
>> >> it turns out that way.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul at mst.edu
>> <mailto:nneul at mst.edu>> wrote:
>> >>
>> >>     The indication I'm reacting to of leak isn't "one in memory"....
>> The below is showing FIFTY THREE of them.
>> >>
>> >>     -- Nathan
>> >
>> >
>>
>> --
>> ------------------------------------------------------------
>> Nathan Neulinger                       nneul at mst.edu
>> Missouri S&T Information Technology    (573) 612-1412
>> System Administrator - Architect
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
> https://www.youtube.com/watch?v=9XXgW34t40s
> https://www.youtube.com/watch?v=NLaDpGQuZDA
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160415/25bf4be1/attachment.html 