Are there any known memory leaks with reloadxml and large XML configs?
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast: freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n': ... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password [root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431 [root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.) ------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ ----------- Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to help diagnose it further to help address the underlying issue. I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment. -- Nathan ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.
On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul@mst.edu> wrote:
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast:
freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':
... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password
[root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431
[root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml
this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.
None of this would indicate what you are suggesting necessarily
It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)
Thats what i would accept.
------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ -----------
Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to help diagnose it further to help address the underlying issue.
I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them. -- Nathan On 04/14/2016 12:38 PM, Michael Jerris wrote:
A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.
On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul@mst.edu> wrote:
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast:
freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':
... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password
[root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431
[root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml
this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.
None of this would indicate what you are suggesting necessarily
It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)
Thats what i would accept.
------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ -----------
Would really appreciate any ideas on how I might mitigate this leaking or if there is anything that could be done to help diagnose it further to help address the underlying issue.
I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet? We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you. Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way. On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
On 04/14/2016 12:38 PM, Michael Jerris wrote:
A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.
On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul@mst.edu> wrote:
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast:
freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':
... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password
[root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431
[root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml
this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.
None of this would indicate what you are suggesting necessarily
It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)
Thats what i would accept.
------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ -----------
Would really appreciate any ideas on how I might mitigate this leaking
or if there is anything that could be done to
help diagnose it further to help address the underlying issue.
I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
P.S. Check all the modules you use to make sure when they open the XML registry that they close it again. On Thu, Apr 14, 2016 at 12:58 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet?
We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.
Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.
On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
On 04/14/2016 12:38 PM, Michael Jerris wrote:
A parsed xml tree in a core file would be normal. We always keep one in memory, thats why you have to reload to get a new one, so it would be in the core file every time, not an indication of a leak.
On Apr 14, 2016, at 12:49 PM, Nathan Neulinger <nneul@mst.edu> wrote:
I've got a persistent issue with a slow (sometimes not that slow) memory leak in FS. Our environment does frequent reconfigs built around reloadxml and when the leak gets bad enough, I start getting random reports of slipping audio sync. I've put in weekly forced failovers, but if we have a week with more changes than typical (we're still in middle of mass migration off of CCM) - the leak growth can be too fast:
freeswi+ 17977 12.1 27.2 15247652 2213648 ? S<l Apr02 2155:56 /local/freeswitch/server/bin/freeswitch -u freeswitch -core -cfgname freeswitch-prod.xml -base /local/freeswitch/server -conf /local/freeswitch/server/conf -db /local/freeswitch/server/db -run /local/freeswitch/data -log /local/freeswitch/data -rp -nf -nc -nonat -nonatmap
The reason I ask if the leak is related to reloadxml - when doing some analysis on the resulting core dump on the (no longer active) server, I'm finding examples like this when doing a 'strings corefile | sort | uniq -c | sort -n':
... 74642 !-- type(skinny) -- 76811 /buttons 76811 "InvalidHash 76812 "a1-hash 76836 /skinny 76853 buttons ... 105277 !-- password won't be used by cisco devices -- ... 105277 "vm-mailfrom 105277 "vm-password
[root@freesw-p1 data]# grep -c InvalidHash freeswitch-prod.xml.fsxml 1431
[root@freesw-p1 data]# ls -al *.fsxml -rw------- 1 freeswitch freeswitch 4985669 Apr 14 08:28 freeswitch-prod.xml.fsxml [root@freesw-p1 data]# wc -l *.fsxml 105025 freeswitch-prod.xml.fsxml
this is just a copy of the compiled xml file. it is 100k lines long. I'm guessing you have lots of static users in here.
None of this would indicate what you are suggesting necessarily
It's like it's leaking large numbers of complete copies of the XML. When I look directly at the core dump, it looks to me like the strings are in the parsed state of the XML. (Below slightly masked copy and paste from viewing dump with less.)
Thats what i would accept.
------------- ^@!-- xxx-xxx-xxxx --^@ ^@user^@id^@"xxxxxxxxxxxxxxxxxxxxxxxx^@^@ ^@params^@ ^@!-- password won't be used by cisco devices --^@ ^@param^@name^@"password^@ value^@"XXXXXXXXXXXXXXXXX^@^@^@ ^@param^@name^@"vm-mailfrom^@ value^@"voicemail@mst.edu^@^@^@ -----------
Would really appreciate any ideas on how I might mitigate this leaking
or if there is anything that could be done to
help diagnose it further to help address the underlying issue.
I'm happy to open a JIRA on this, but will NOT be able to test this with latest master as I can't just experiment with the live production environment.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
FS-9074 opened. I've attempted to reproduce in lab environment, but not with much success since I can't readily reproduce the activity/usage level from the production environment. On production env, I can readily see the resident memory growing when I reloadxml on the active server, but it doesn't do it every time, so it's clearly some sort of combination of events/timing related. On the lab setup, I typically see about 3 x duplicates of the strings from the XML in a generated core. No modules are being reloaded during normal operations. Just a rewrite of the configs followed by a reloadxml. As far as the commercial option - have asked y'all several times for what our options would be for that sort of service, but have never gotten any clear answer particularly given the custom deployment we have. Would be very interested in presenting that to management here if you can give me some proposed service offerings and costs. -- Nathan On 04/14/2016 12:58 PM, Anthony Minessale wrote:
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet?
We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.
Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.
On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu <mailto:nneul@mst.edu>> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
Running some more post analysis over the raw content of the core dump, and extracting each parsed blob of XML, it looks like almost all of them are complete copies (and not partial like if it ran out of room and had to resize+relocate). I extracted all of the chunks to individual files, and looking at each one in turn based on sequential position in memory, I am seeing reasonably expected small/tiny changes from one to the next that would correspond to incremental changes in our configuration UI (stuff like individual changes to a line key, button position, voicemail address, password, etc.) What in the system results in a full copy of the parsed XML structure? Does every access do some sort of temporary copy like you'd get with SQL and transaction isolation? -- Nathan On 04/14/2016 01:25 PM, Nathan Neulinger wrote:
FS-9074 opened.
I've attempted to reproduce in lab environment, but not with much success since I can't readily reproduce the activity/usage level from the production environment. On production env, I can readily see the resident memory growing when I reloadxml on the active server, but it doesn't do it every time, so it's clearly some sort of combination of events/timing related.
On the lab setup, I typically see about 3 x duplicates of the strings from the XML in a generated core.
No modules are being reloaded during normal operations. Just a rewrite of the configs followed by a reloadxml.
As far as the commercial option - have asked y'all several times for what our options would be for that sort of service, but have never gotten any clear answer particularly given the custom deployment we have. Would be very interested in presenting that to management here if you can give me some proposed service offerings and costs.
-- Nathan
On 04/14/2016 12:58 PM, Anthony Minessale wrote:
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet?
We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.
Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.
On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu <mailto:nneul@mst.edu>> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
Why are you still discussing this here when you have a jira open? I am not sure how to properly explain the IMMENSE difficulty of working on bugs over email. There is no right reason to side with email when trying to decide between jira and email. If a jira turns out not to be a bug, one click to close it. When an email turns out to be a bug, frick, now we have to copy all the data over. Pass it on, file jiras when you have a problem. We have broadcasted it here for years..:..... The xml registry is ref counted if it reloads while its open, it probably means something you use opens the xml root and does not close it. I said this on your jira. On Thursday, April 14, 2016, Nathan Neulinger <nneul@mst.edu> wrote:
Running some more post analysis over the raw content of the core dump, and extracting each parsed blob of XML, it looks like almost all of them are complete copies (and not partial like if it ran out of room and had to resize+relocate).
I extracted all of the chunks to individual files, and looking at each one in turn based on sequential position in memory, I am seeing reasonably expected small/tiny changes from one to the next that would correspond to incremental changes in our configuration UI (stuff like individual changes to a line key, button position, voicemail address, password, etc.)
What in the system results in a full copy of the parsed XML structure? Does every access do some sort of temporary copy like you'd get with SQL and transaction isolation?
-- Nathan
On 04/14/2016 01:25 PM, Nathan Neulinger wrote:
FS-9074 opened.
I've attempted to reproduce in lab environment, but not with much success since I can't readily reproduce the activity/usage level from the production environment. On production env, I can readily see the resident memory growing when I reloadxml on the active server, but it doesn't do it every time, so it's clearly some sort of combination of events/timing related.
On the lab setup, I typically see about 3 x duplicates of the strings from the XML in a generated core.
No modules are being reloaded during normal operations. Just a rewrite of the configs followed by a reloadxml.
As far as the commercial option - have asked y'all several times for what our options would be for that sort of service, but have never gotten any clear answer particularly given the custom deployment we have. Would be very interested in presenting that to management here if you can give me some proposed service offerings and costs.
-- Nathan
On 04/14/2016 12:58 PM, Anthony Minessale wrote:
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet?
We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.
Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.
On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu <javascript:;> <mailto:nneul@mst.edu <javascript:;>>> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu <javascript:;> Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <javascript:;> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <javascript:;> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Thanks for the assistance. Your additional suggestion of where to look led me right to an additional memory leak in mod_skinny - and was as a result able to reproduce a scenario that could trigger it. I don't know for sure that it fixes the symptom I've seen on production, but I suspect this is at least a significant part of it. A fix has been pushed into master for it. JIRA issue updated accordingly. -- Nathan On Thu, Apr 14, 2016 at 11:41 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Why are you still discussing this here when you have a jira open? I am not sure how to properly explain the IMMENSE difficulty of working on bugs over email. There is no right reason to side with email when trying to decide between jira and email. If a jira turns out not to be a bug, one click to close it. When an email turns out to be a bug, frick, now we have to copy all the data over. Pass it on, file jiras when you have a problem. We have broadcasted it here for years..:.....
The xml registry is ref counted if it reloads while its open, it probably means something you use opens the xml root and does not close it. I said this on your jira.
On Thursday, April 14, 2016, Nathan Neulinger <nneul@mst.edu> wrote:
Running some more post analysis over the raw content of the core dump, and extracting each parsed blob of XML, it looks like almost all of them are complete copies (and not partial like if it ran out of room and had to resize+relocate).
I extracted all of the chunks to individual files, and looking at each one in turn based on sequential position in memory, I am seeing reasonably expected small/tiny changes from one to the next that would correspond to incremental changes in our configuration UI (stuff like individual changes to a line key, button position, voicemail address, password, etc.)
What in the system results in a full copy of the parsed XML structure? Does every access do some sort of temporary copy like you'd get with SQL and transaction isolation?
-- Nathan
On 04/14/2016 01:25 PM, Nathan Neulinger wrote:
FS-9074 opened.
I've attempted to reproduce in lab environment, but not with much success since I can't readily reproduce the activity/usage level from the production environment. On production env, I can readily see the resident memory growing when I reloadxml on the active server, but it doesn't do it every time, so it's clearly some sort of combination of events/timing related.
On the lab setup, I typically see about 3 x duplicates of the strings from the XML in a generated core.
No modules are being reloaded during normal operations. Just a rewrite of the configs followed by a reloadxml.
As far as the commercial option - have asked y'all several times for what our options would be for that sort of service, but have never gotten any clear answer particularly given the custom deployment we have. Would be very interested in presenting that to management here if you can give me some proposed service offerings and costs.
-- Nathan
On 04/14/2016 12:58 PM, Anthony Minessale wrote:
Have you done any debugging like running valgrind or ASAN on a server? Did you try to reproduce it in a lab yet?
We can't really do any leg work for you so we would need a more specific reproduction case. We do offer commercial options as you are probably already are aware of where we could deploy consultants to do more of the leg work for you.
Do you only reloadxml or do you reload any modules. Also you should ALWAYS file a jira. We don't want to field issues on the list. Its not a problem to close NOT A BUG if it turns out that way.
On Thu, Apr 14, 2016 at 12:50 PM, Nathan Neulinger <nneul@mst.edu <mailto:nneul@mst.edu>> wrote:
The indication I'm reacting to of leak isn't "one in memory".... The below is showing FIFTY THREE of them.
-- Nathan
-- ------------------------------------------------------------ Nathan Neulinger nneul@mst.edu Missouri S&T Information Technology (573) 612-1412 System Administrator - Architect
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (3)
-
Anthony Minessale
-
Michael Jerris
-
Nathan Neulinger