[Freeswitch-dev] How can I detect the cause of a potential memory leak?
Hello, I'm experiencing what I think is a memory leak in a production system with a recent git (2-Nov), with FS consuming >5GB and increasing after a few days with never more than 10 calls at once. While the users are not experiencing problems so far, I'm a bit worried. So I have a few questions: - How can I be sure it's a leak and not some memory pool thing that FS would free when the system needs (system has 10GB total)? - If I unload modules one by one, is the memory used by this module freed immediately? - What would be a "good" way to try and narrow down the cause? Is valgrind a good tool for that? Of course I'll try with GIT HEAD and try my best to find the cause before considering to add a new Jira issue, otherwise it's quite useless. Thanks, François.
valgrind is your best bet: valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg run this on normal traffic for a while and get the log file. if you unload mods it would not help with a leak but it would with a swelling pool. On Mon, Nov 14, 2011 at 11:30 AM, François Delawarde < fdelawarde@wirelessmundi.com> wrote:
Hello,
I'm experiencing what I think is a memory leak in a production system with a recent git (2-Nov), with FS consuming >5GB and increasing after a few days with never more than 10 calls at once.
While the users are not experiencing problems so far, I'm a bit worried. So I have a few questions:
- How can I be sure it's a leak and not some memory pool thing that FS would free when the system needs (system has 10GB total)?
- If I unload modules one by one, is the memory used by this module freed immediately?
- What would be a "good" way to try and narrow down the cause? Is valgrind a good tool for that?
Of course I'll try with GIT HEAD and try my best to find the cause before considering to add a new Jira issue, otherwise it's quite useless.
Thanks, François.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
I did a small valgrind test with git HEAD: 1. turn on, make 5 calls or so 2. leave FS on for the night (without calls) 3. shut it down cleanly in the morning During execution, some "Warning: invalid file descriptor -1 in syscall close()" (4-5 for each call + some extras). After shutting down, something strikes me in the vg.log file: "definitely lost: 173,407,885 bytes in 84,680 blocks" Is this normal? valgrind log: http://pastebin.freeswitch.org/17779 Thanks, François. On Mon, 2011-11-14 at 11:41 -0600, Anthony Minessale wrote:
valgrind is your best bet:
valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg
run this on normal traffic for a while and get the log file.
if you unload mods it would not help with a leak but it would with a swelling pool.
On Mon, Nov 14, 2011 at 11:30 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: Hello,
I'm experiencing what I think is a memory leak in a production system with a recent git (2-Nov), with FS consuming >5GB and increasing after a few days with never more than 10 calls at once.
While the users are not experiencing problems so far, I'm a bit worried. So I have a few questions:
- How can I be sure it's a leak and not some memory pool thing that FS would free when the system needs (system has 10GB total)?
- If I unload modules one by one, is the memory used by this module freed immediately?
- What would be a "good" way to try and narrow down the cause? Is valgrind a good tool for that?
Of course I'll try with GIT HEAD and try my best to find the cause before considering to add a new Jira issue, otherwise it's quite useless.
Thanks, François.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
That report showed me 2 leaks. 1) in mod_cdr_sqlite where it was expanding a variable template and not checking if it needed to free the results. 2) in mod_dptools in the user channel code. Wait, there's more. Fixing the leak in the user channel then led me to realize there was a regression in the change that introduced that leak that was breaking the code that copies all the variables from the {} into the far end channel. anyway, try out GIT head. On Tue, Nov 15, 2011 at 3:25 AM, François Delawarde < fdelawarde@wirelessmundi.com> wrote:
I did a small valgrind test with git HEAD: 1. turn on, make 5 calls or so 2. leave FS on for the night (without calls) 3. shut it down cleanly in the morning
During execution, some "Warning: invalid file descriptor -1 in syscall close()" (4-5 for each call + some extras).
After shutting down, something strikes me in the vg.log file: "definitely lost: 173,407,885 bytes in 84,680 blocks"
Is this normal?
valgrind log: http://pastebin.freeswitch.org/17779
Thanks, François.
On Mon, 2011-11-14 at 11:41 -0600, Anthony Minessale wrote:
valgrind is your best bet:
valgrind --tool=memcheck --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg
run this on normal traffic for a while and get the log file.
if you unload mods it would not help with a leak but it would with a swelling pool.
On Mon, Nov 14, 2011 at 11:30 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: Hello,
I'm experiencing what I think is a memory leak in a production system with a recent git (2-Nov), with FS consuming >5GB and increasing after a few days with never more than 10 calls at once.
While the users are not experiencing problems so far, I'm a bit worried. So I have a few questions:
- How can I be sure it's a leak and not some memory pool thing that FS would free when the system needs (system has 10GB total)?
- If I unload modules one by one, is the memory used by this module freed immediately?
- What would be a "good" way to try and narrow down the cause? Is valgrind a good tool for that?
Of course I'll try with GIT HEAD and try my best to find the cause before considering to add a new Jira issue, otherwise it's quite useless.
Thanks, François.
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-dev
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Cool! I should do that more often... I'll first try to figure out how you found those leaks from the report. Thanks! François. On Tue, 2011-11-15 at 10:02 -0600, Anthony Minessale wrote:
That report showed me 2 leaks.
1) in mod_cdr_sqlite where it was expanding a variable template and not checking if it needed to free the results. 2) in mod_dptools in the user channel code.
Wait, there's more.
Fixing the leak in the user channel then led me to realize there was a regression in the change that introduced that leak that was breaking the code that copies all the variables from the {} into the far end channel.
anyway, try out GIT head.
On Tue, Nov 15, 2011 at 3:25 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: I did a small valgrind test with git HEAD: 1. turn on, make 5 calls or so 2. leave FS on for the night (without calls) 3. shut it down cleanly in the morning
During execution, some "Warning: invalid file descriptor -1 in syscall close()" (4-5 for each call + some extras).
After shutting down, something strikes me in the vg.log file: "definitely lost: 173,407,885 bytes in 84,680 blocks"
Is this normal?
valgrind log: http://pastebin.freeswitch.org/17779
Thanks, François.
On Mon, 2011-11-14 at 11:41 -0600, Anthony Minessale wrote: > valgrind is your best bet: > > valgrind --tool=memcheck --log-file=vg.log --leak-check=full > --leak-resolution=high > --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg > > > run this on normal traffic for a while and get the log file. > > > if you unload mods it would not help with a leak but it would with a > swelling pool. > > > > On Mon, Nov 14, 2011 at 11:30 AM, François Delawarde > <fdelawarde@wirelessmundi.com> wrote: > Hello, > > I'm experiencing what I think is a memory leak in a production > system > with a recent git (2-Nov), with FS consuming >5GB and > increasing after a > few days with never more than 10 calls at once. > > While the users are not experiencing problems so far, I'm a > bit worried. > So I have a few questions: > > - How can I be sure it's a leak and not some memory pool thing > that FS > would free when the system needs (system has 10GB total)? > > - If I unload modules one by one, is the memory used by this > module > freed immediately? > > - What would be a "good" way to try and narrow down the cause? > Is > valgrind a good tool for that? > > Of course I'll try with GIT HEAD and try my best to find the > cause > before considering to add a new Jira issue, otherwise it's > quite > useless. > > > Thanks, > François. > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-dev mailing list > FreeSWITCH-dev@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev > http://www.freeswitch.org > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > Twitter: http://twitter.com/FreeSWITCH_wire > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > googletalk:conf+888@conference.freeswitch.org > pstn:+19193869900 > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-dev mailing list > FreeSWITCH-dev@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev > http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
Master Anthony thanks again, your teachings were not in vain: http://jira.freeswitch.org/browse/FS-3702 Regards, François. On Tue, 2011-11-15 at 18:02 +0100, François Delawarde wrote:
Cool! I should do that more often...
I'll first try to figure out how you found those leaks from the report.
Thanks! François.
On Tue, 2011-11-15 at 10:02 -0600, Anthony Minessale wrote:
That report showed me 2 leaks.
1) in mod_cdr_sqlite where it was expanding a variable template and not checking if it needed to free the results. 2) in mod_dptools in the user channel code.
Wait, there's more.
Fixing the leak in the user channel then led me to realize there was a regression in the change that introduced that leak that was breaking the code that copies all the variables from the {} into the far end channel.
anyway, try out GIT head.
On Tue, Nov 15, 2011 at 3:25 AM, François Delawarde <fdelawarde@wirelessmundi.com> wrote: I did a small valgrind test with git HEAD: 1. turn on, make 5 calls or so 2. leave FS on for the night (without calls) 3. shut it down cleanly in the morning
During execution, some "Warning: invalid file descriptor -1 in syscall close()" (4-5 for each call + some extras).
After shutting down, something strikes me in the vg.log file: "definitely lost: 173,407,885 bytes in 84,680 blocks"
Is this normal?
valgrind log: http://pastebin.freeswitch.org/17779
Thanks, François.
On Mon, 2011-11-14 at 11:41 -0600, Anthony Minessale wrote: > valgrind is your best bet: > > valgrind --tool=memcheck --log-file=vg.log --leak-check=full > --leak-resolution=high > --show-reachable=yes /usr/local/freeswitch/bin/freeswitch -vg > > > run this on normal traffic for a while and get the log file. > > > if you unload mods it would not help with a leak but it would with a > swelling pool. > > > > On Mon, Nov 14, 2011 at 11:30 AM, François Delawarde > <fdelawarde@wirelessmundi.com> wrote: > Hello, > > I'm experiencing what I think is a memory leak in a production > system > with a recent git (2-Nov), with FS consuming >5GB and > increasing after a > few days with never more than 10 calls at once. > > While the users are not experiencing problems so far, I'm a > bit worried. > So I have a few questions: > > - How can I be sure it's a leak and not some memory pool thing > that FS > would free when the system needs (system has 10GB total)? > > - If I unload modules one by one, is the memory used by this > module > freed immediately? > > - What would be a "good" way to try and narrow down the cause? > Is > valgrind a good tool for that? > > Of course I'll try with GIT HEAD and try my best to find the > cause > before considering to add a new Jira issue, otherwise it's > quite > useless. > > > Thanks, > François. > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-dev mailing list > FreeSWITCH-dev@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev > http://www.freeswitch.org > > > > > -- > Anthony Minessale II > > FreeSWITCH http://www.freeswitch.org/ > ClueCon http://www.cluecon.com/ > Twitter: http://twitter.com/FreeSWITCH_wire > > AIM: anthm > MSN:anthony_minessale@hotmail.com > GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com > IRC: irc.freenode.net #freeswitch > > FreeSWITCH Developer Conference > sip:888@conference.freeswitch.org > googletalk:conf+888@conference.freeswitch.org > pstn:+19193869900 > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-dev mailing list > FreeSWITCH-dev@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev > http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
participants (2)
-
Anthony Minessale
-
François Delawarde