The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them. -- Nathan On 04/14/2016 12:38 PM, Michael Jerris wrote:
A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.
On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul@mst.edu> wrote:
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast:
freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':
... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password
[root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431
[root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml
this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.
None of this would indicate what you are suggesting necessarily
It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)
Thats what i would accept.
------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ -----------
Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to help diagnose it further to help address the underlying issue.
I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect