<div dir="ltr">P.S.<div>Check all the modules you use to make sure when they open the XML registry that they close it again.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 14, 2016 at 12:58 PM, Anthony Minessale <span dir="ltr">&lt;<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Have you done any debugging like running valgrind or ASAN on a server?<div>Did you try to reproduce it in a lab yet?</div><div><br></div><div>We can&#39;t really do any leg work for you so we would need a more specific reproduction case.</div><div>We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.</div><div><br></div><div>Do you only reloadxml or do you reload any modules.</div><div>Also you should ALWAYS file a jira.  We don&#39;t want to field issues on the list.  Its not a problem to close NOT A BUG if it turns out that way.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <span dir="ltr">&lt;<a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The indication I&#39;m reacting to of leak isn&#39;t &quot;one in memory&quot;.... The below is showing FIFTY THREE of them.<br>
<span><font color="#888888"><br>
-- Nathan<br>
</font></span><div><div><br>
On 04/14/2016 12:38 PM, Michael Jerris wrote:<br>
&gt; A parsed xml tree in a core file would be normal.  We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.<br>
&gt;<br>
&gt;<br>
&gt;&gt; On Apr 14, 2016, at 12:49 PM, Nathan Neulinger &lt;<a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a>&gt; wrote:<br>
&gt;&gt;<br>
&gt;&gt; I&#39;ve got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent<br>
&gt;&gt; reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio<br>
&gt;&gt; sync. I&#39;ve put in weekly forced failovers, but if we have a week with more changes than typical (we&#39;re still in middle<br>
&gt;&gt; of mass migration off of CCM) - the leak growth can be too fast:<br>
&gt;&gt;<br>
&gt;&gt; freeswi+ 17977 12.1 27.2 15247652 2213648 ?    S&lt;l  Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch<br>
&gt;&gt; -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db<br>
&gt;&gt; /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no<br>
&gt;&gt; longer active) server, I&#39;m finding examples like this when doing a &#39;strings corefile | sort | uniq -c | sort -n&#39;:<br>
&gt;&gt;<br>
&gt;&gt; ...<br>
&gt;&gt;    74642 !-- type(skinny) --<br>
&gt;&gt;    76811 /buttons<br>
&gt;&gt;    76811 &quot;InvalidHash<br>
&gt;&gt;    76812 &quot;a1-hash<br>
&gt;&gt;    76836 /skinny<br>
&gt;&gt;    76853 buttons<br>
&gt;&gt; ...<br>
&gt;&gt;   105277 !-- password won&#39;t be used by cisco devices --<br>
&gt;&gt; ...<br>
&gt;&gt;   105277 &quot;vm-mailfrom<br>
&gt;&gt;   105277 &quot;vm-password<br>
&gt;&gt;<br>
&gt;&gt; [root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml<br>
&gt;&gt; 1431<br>
&gt;&gt;<br>
&gt;&gt; [root@freesw-p1 data]# ls -al *.fsxml<br>
&gt;&gt; -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml<br>
&gt;&gt; [root@freesw-p1 data]# wc -l *.fsxml<br>
&gt;&gt; 105025 freeswitch-prod.xml.fsxml<br>
&gt;&gt;<br>
&gt;<br>
&gt; this is just a copy of the compiled xml file.  it is 100k lines long.  I&#39;m guessing you have lots of static users in here.<br>
&gt;<br>
&gt;&gt;<br>
&gt;<br>
&gt; None of this would indicate what you are suggesting necessarily<br>
&gt;<br>
&gt;&gt; It&#39;s like it&#39;s leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to<br>
&gt;&gt; me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)<br>
&gt;<br>
&gt; Thats what i would accept.<br>
&gt;<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; -------------<br>
&gt;&gt; ^@!-- xxx-xxx-xxxx --^@<br>
&gt;&gt; ^@user^@id^@&quot;xxxxxxxxxxxxxxxxxxxxxxxx^@^@<br>
&gt;&gt;      ^@params^@<br>
&gt;&gt;          ^@!-- password won&#39;t be used by cisco devices --^@<br>
&gt;&gt;          ^@param^@name^@&quot;password^@ value^@&quot;XXXXXXXXXXXXXXXXX^@^@^@<br>
&gt;&gt;          ^@param^@name^@&quot;vm-mailfrom^@ value^@&quot;<a href="mailto:voicemail@mst.edu" target="_blank">voicemail@mst.edu</a>^@^@^@<br>
&gt;&gt; -----------<br>
&gt;&gt;<br>
&gt;&gt; Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to<br>
&gt;&gt; help diagnose it further to help address the underlying issue.<br>
&gt;&gt;<br>
&gt;&gt;<br>
&gt;&gt; I&#39;m happy to open a JIRA on this, but will NOT be able to test this with latest master as I can&#39;t just experiment with<br>
&gt;&gt; the live production environment.<br>
&gt;&gt;<br>
&gt;&gt; -- Nathan<br>
&gt;&gt;<br>
&gt;&gt; ------------------------------------------------------------<br>
&gt;&gt; Nathan Neulinger                       <a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a><br>
&gt;&gt; Missouri S&amp;T Information Technology    <a href="tel:%28573%29%20612-1412" value="+15736121412" target="_blank">(573) 612-1412</a><br>
&gt;&gt; System Administrator - Architect<br>
&gt;&gt;<br>
&gt;&gt; _________________________________________________________________________<br>
&gt;&gt; Professional FreeSWITCH Consulting Services:<br>
&gt;&gt; <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
&gt;&gt; <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;&gt;<br>
&gt;&gt; Official FreeSWITCH Sites<br>
&gt;&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;&gt; <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt;&gt; <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;&gt;<br>
&gt;&gt; FreeSWITCH-users mailing list<br>
&gt;&gt; <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
&gt;&gt; <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt;&gt; UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt;&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;<br>
&gt;<br>
&gt; _________________________________________________________________________<br>
&gt; Professional FreeSWITCH Consulting Services:<br>
&gt; <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
&gt; <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
&gt;<br>
&gt; Official FreeSWITCH Sites<br>
&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt; <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
&gt; <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
&gt;<br>
&gt; FreeSWITCH-users mailing list<br>
&gt; <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
&gt; <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
&gt; UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
&gt; <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
&gt;<br>
<br>
</div></div>--<br>
<div><div>------------------------------------------------------------<br>
Nathan Neulinger                       <a href="mailto:nneul@mst.edu" target="_blank">nneul@mst.edu</a><br>
Missouri S&amp;T Information Technology    <a href="tel:%28573%29%20612-1412" value="+15736121412" target="_blank">(573) 612-1412</a><br>
System Administrator - Architect<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a>  ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a>  ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a>  ☎ <a href="tel:%2B19193869900" value="+19193869900" target="_blank">+19193869900</a> </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a>  ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a>  ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div><div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br></div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a>  ☎ +19193869900 </div><div><br></div></div></div><div><a href="https://www.youtube.com/watch?v=9XXgW34t40s" style="color:rgb(17,85,204);font-size:12.8000001907349px" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a></div><div><a href="https://www.youtube.com/watch?v=NLaDpGQuZDA" target="_blank">https://www.youtube.com/watch?v=NLaDpGQuZDA</a><br></div></div></div></div></div></div></div>
</div>