<div dir="ltr">Have you done any debugging like running valgrind or ASAN on a server?<div>Did you try to reproduce it in a lab yet?</div><div><br></div><div>We can't really do any leg work for you so we would need a more specific reproduction case.</div><div>We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.</div><div><br></div><div>Do you only reloadxml or do you reload any modules.</div><div>Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <span dir="ltr"><<a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.<br>
<span class="HOEnZb"><font color="#888888"><br>
-- Nathan<br>
</font></span><div><div class="h5"><br>
On 04/14/2016 12:38 PM, Michael Jerris wrote:<br>
> A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.<br>
><br>
><br>
>> On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <<a href="mailto:nneul@mst.edu">nneul@mst.edu</a>> wrote:<br>
>><br>
>> I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent<br>
>> reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio<br>
>> sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle<br>
>> of mass migration off of CCM) - the leak growth can be too fast:<br>
>><br>
>> freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch<br>
>> -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db<br>
>> /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap<br>
>><br>
>><br>
>> The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no<br>
>> longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':<br>
>><br>
>> ...<br>
>> 74642 !-- type(skinny) --<br>
>> 76811 /buttons<br>
>> 76811 "InvalidHash<br>
>> 76812 "a1-hash<br>
>> 76836 /skinny<br>
>> 76853 buttons<br>
>> ...<br>
>> 105277 !-- password won't be used by cisco devices --<br>
>> ...<br>
>> 105277 "vm-mailfrom<br>
>> 105277 "vm-password<br>
>><br>
>> [root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml<br>
>> 1431<br>
>><br>
>> [root@freesw-p1 data]# ls -al *.fsxml<br>
>> -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml<br>
>> [root@freesw-p1 data]# wc -l *.fsxml<br>
>> 105025 freeswitch-prod.xml.fsxml<br>
>><br>
><br>
> this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.<br>
><br>
>><br>
><br>
> None of this would indicate what you are suggesting necessarily<br>
><br>
>> It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to<br>
>> me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)<br>
><br>
> Thats what i would accept.<br>
><br>
>><br>
>><br>
>> -------------<br>
>> ^@!-- xxx-xxx-xxxx --^@<br>
>> ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@<br>
>> ^@params^@<br>
>> ^@!-- password won't be used by cisco devices --^@<br>
>> ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@<br>
>> ^@param^@name^@"vm-mailfrom^@ value^@"<a href="mailto:voicemail@mst.edu">voicemail@mst.edu</a>^@^@^@<br>
>> -----------<br>
>><br>
>> Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to<br>
>> help diagnose it further to help address the underlying issue.<br>
>><br>
>><br>
>> I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with<br>
>> the live production environment.<br>
>><br>
>> -- Nathan<br>
>><br>
>> ------------------------------------------------------------<br>
>> Nathan Neulinger <a href="mailto:nneul@mst.edu">nneul@mst.edu</a><br>
>> Missouri S&T Information Technology <a href="tel:%28573%29%20612-1412" value="+15736121412">(573) 612-1412</a><br>
>> System Administrator - Architect<br>
>><br>
>> _________________________________________________________________________<br>
>> Professional FreeSWITCH Consulting Services:<br>
>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
>><br>
>> Official FreeSWITCH Sites<br>
>> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
>> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
>> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
>><br>
>> FreeSWITCH-users mailing list<br>
>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
</div></div>--<br>
<div class="HOEnZb"><div class="h5">------------------------------------------------------------<br>
Nathan Neulinger <a href="mailto:nneul@mst.edu">nneul@mst.edu</a><br>
Missouri S&T Information Technology <a href="tel:%28573%29%20612-1412" value="+15736121412">(573) 612-1412</a><br>
System Administrator - Architect<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ +19193869900 </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</div>