Hello *, a memory leak showed up in our loadtests. It's (still) the same setup as in the http://jira.freeswitch.org/browse/MODSOFIA-22 bugfix. One thing I'd like to add is that "fsctl shutdown restart" was unable to shutdown freeswitch. The last line printed is "switch_core_memory.c:567 Stopping memory pool queue." attached file is a the collected and graphed output of some "ps waux" command. "sz" should be in "physical pages" (2k?). I rerun the test and this time it coredumped trying to malloc() space for some playback. Anything else you need (full backtraces?) to dig into it? Cheers Beni.
600k is not a leak? FS can use as much as a gig of ram or more depending on what you are doing. you may want to install a fresh copy of FS, removing all your old files etc and make sure they build clean. We also have not had much luck running on ubuntu which is more of a desktop centric OS. I recommend you try your application on 64 bit CentOS which is the platform all of our paid customers use. On Thu, Aug 27, 2009 at 8:15 AM, Benedikt Fraunhofer < fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello *,
a memory leak showed up in our loadtests. It's (still) the same setup as in the http://jira.freeswitch.org/browse/MODSOFIA-22 bugfix.
One thing I'd like to add is that "fsctl shutdown restart" was unable to shutdown freeswitch. The last line printed is "switch_core_memory.c:567 Stopping memory pool queue."
attached file is a the collected and graphed output of some "ps waux" command. "sz" should be in "physical pages" (2k?).
I rerun the test and this time it coredumped trying to malloc() space for some playback.
Anything else you need (full backtraces?) to dig into it?
Cheers Beni.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hello Anthony, 2009/8/27 Anthony Minessale <anthony.minessale@gmail.com>:
600k is not a leak? FS can use as much as a gig of ram or more depending on what you are doing.
it's 600*1024 * 4* 1024 /1024/1024 => ~ 2.4 Gig that's what i tried to express with the "physical pages" in my first post.
you may want to install a fresh copy of FS, removing all your old files etc and make sure they build clean.
i "make current" to get the SOFIA-22 bugfix.
We also have not had much luck running on ubuntu which is more of a desktop centric OS.
really? even the "server" edition? (not the netbook remix :) Is 64bit so grave different if i don't need to address more than 4 Gig of ram per process?
I recommend you try your application on 64 bit CentOS which is the platform all of our paid customers use.
hmm. our operations-people wont be happy with that. But as i carefully followed the other mailing-list-threads it came to my mind that i could be the culprit: somehow the original example.xml came back into the directory tree and included a lot of things, most notably the hash-inserts for last-dailed-numbers and stuff. I stripped it down to what it was before and gave it another try. thx again and (hopefully) sorry Beni.
Anthony can you ( or anyone else alao ). Please elaborate on what makes centos 5.3 o much better for Freeswitch. Is there some specific library vesiion on centos that makes a massive difference ? Reason I ask ... I personally only have a preference for debian, but others may have policy mandated Os's For their companies, and it would be great to have some info about this. I'd imagine this actually boils down to a requirement for libfoo version X ... And if we ran those library versions on other OS's we'd be fine ? Jay
I recommend you try your application on 64 bit CentOS which is the platform all of our paid customers use.
hmm. our operations-people wont be happy with that.
Mostly based on that is what we developed it with and continue to use and have had the best luck and least problems with. I am sure you are right about it being certain version of the core toolchain I will guess it's the combo of libc and kernel and the fact that CentOS has a purist attitude towards patching such code. Some people use stable debian with no issues, other have reported problems resolved by switching to CentOS so presumably you could roll your own debian that was similar but we don't have the resources to try to find out what makes what happen etc. If you can configure debian to work, it's fine, if we get some inexplicable problem on debian we will start to wonder though. it's happened more than once with people doing nothing other than changing OS =D On Thu, Aug 27, 2009 at 4:55 PM, Jay Binks <jaybinks@gmail.com> wrote:
Anthony can you ( or anyone else alao ). Please elaborate on what makes centos 5.3 o much better for Freeswitch.
Is there some specific library vesiion on centos that makes a massive difference ?
Reason I ask ... I personally only have a preference for debian, but others may have policy mandated Os's For their companies, and it would be great to have some info about this.
I'd imagine this actually boils down to a requirement for libfoo version X ... And if we ran those library versions on other OS's we'd be fine ?
Jay
I recommend you try your application on 64 bit CentOS which is the platform all of our paid customers use.
hmm. our operations-people wont be happy with that.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Jay Binks <jaybinks@gmail.com> wrote:
Reason I ask ... I personally only have a preference for debian, but others may have policy mandated Os's For their companies, and it would be great to have some info about this.
The only problem I've had with FreeSWITCH under Debian Squeeze and Sid involves TLS-related segmentation faults that appear to be related to something in the version of libssl supplied with Debian. The same problem can't be reproduced on Fedora, for example, but it does occur under Debian Lenny as well as Debian Squeeze and Sid (i.e., "testing" and "unstable", respectively). besides this, all of the issues that I have encountered turned out to be (usually short-lived) bugs in FreeSWITCH or one of the libraries included in the source tree - they're mostly FreeSWITCH issues. I should point out that the FreeSWITCH developers are very good at avoiding the introduction of bugs into their code and that known bugs get fixed. It appears to be an unchanging fact about programming that with a large and complex project, even given highly knowledgeable, experienced, committed and talented developers (as we have with FreeSWITCH), sometimes, bugs do slip through.
Cool, feel free to report these bugs in Jira. If they are indeed FreeSWITCH bugs. On Fri, Aug 28, 2009 at 12:10 AM, Jason White <jason@jasonjgw.net> wrote:
Jay Binks <jaybinks@gmail.com> wrote:
Reason I ask ... I personally only have a preference for debian, but others may have policy mandated Os's For their companies, and it would be great to have some info about this.
The only problem I've had with FreeSWITCH under Debian Squeeze and Sid involves TLS-related segmentation faults that appear to be related to something in the version of libssl supplied with Debian. The same problem can't be reproduced on Fedora, for example, but it does occur under Debian Lenny as well as Debian Squeeze and Sid (i.e., "testing" and "unstable", respectively).
besides this, all of the issues that I have encountered turned out to be (usually short-lived) bugs in FreeSWITCH or one of the libraries included in the source tree - they're mostly FreeSWITCH issues.
I should point out that the FreeSWITCH developers are very good at avoiding the introduction of bugs into their code and that known bugs get fixed. It appears to be an unchanging fact about programming that with a large and complex project, even given highly knowledgeable, experienced, committed and talented developers (as we have with FreeSWITCH), sometimes, bugs do slip through.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hello *, still no luck. Removing the default dialplan with all those hash-inserts made the memory-usage-curve flatter but it oom-coredumped stating: freeswitch: src/switch_core_memory.c:443: switch_core_perform_alloc: Assertion `ptr != ((void *)0)' failed. after eating up 3.2 gig ram+swap. Any chance or idea to track that down or any hint where to look at? The other two freeswitch-machines in this scenario play well, they're dumb load-sinks. what makes this machine different is on one hand the lua-scripts that are run on it and on the other hand it's the only machine who gets it's commands via xml_rpc. Cheers Beni.
Use valgrind. /b On Aug 31, 2009, at 7:22 AM, Benedikt Fraunhofer wrote:
The other two freeswitch-machines in this scenario play well, they're dumb load-sinks. what makes this machine different is on one hand the lua-scripts that are run on it and on the other hand it's the only machine who gets it's commands via xml_rpc.
Isn't there a known issue with lua+sql leaking memory on some platforms? On Mon, Aug 31, 2009 at 8:32 AM, Brian West<brian@freeswitch.org> wrote:
Use valgrind.
/b
On Aug 31, 2009, at 7:22 AM, Benedikt Fraunhofer wrote:
The other two freeswitch-machines in this scenario play well, they're dumb load-sinks. what makes this machine different is on one hand the lua-scripts that are run on it and on the other hand it's the only machine who gets it's commands via xml_rpc.
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
Hello *, 2009/8/31 Rupa Schomaker <rupa@rupa.com>:
Isn't there a known issue with lua+sql leaking memory on some platforms?
just lua, no sql in use :)
On Mon, Aug 31, 2009 at 8:32 AM, Brian West<brian@freeswitch.org> wrote:
Use valgrind.
i tried that... in the beginning valgrind segfaultet several times with some error message that it's unable to monitor so many threads. vg_alloc_ThreadState: no free slots available Increase VG_N_THREADS, rebuild and try again. valgrind: the 'impossible' happened: VG_N_THREADS is too low it happeneded with 400 concurrent calls pretty soon, with 100 it took a while, but at least it consumed half a gig of ram. Can someone interpret the file it generated so far? i put it at http://ns42.ath.cx/B0GdWh/vg.log.bz2 (5k compressed, 452k uncompressed) or is there just not enough information contained? I'll give it another shot with 50 concurrent calls but i guess this will take ages... Thx in advance Beni.
What are you doing in these lua scripts? Because there are a few things you can do in the lua script itself that will cause you to leak like crazy due to improper use. /b On Sep 2, 2009, at 7:59 AM, Benedikt Fraunhofer wrote:
Hello *,
2009/8/31 Rupa Schomaker <rupa@rupa.com>:
Isn't there a known issue with lua+sql leaking memory on some platforms?
just lua, no sql in use :)
Hello Brian, 2009/9/2 Brian West <brian@freeswitch.org>:
What are you doing in these lua scripts? Because there are a few things you can do in the lua script itself that will cause you to leak like crazy due to improper use. /b
the setup is the same as in http://jira.freeswitch.org/browse/MODSOFIA-22 one is ----- local reason = session:getVariable("originate_disposition"); session:setAutoHangup(false); if(reason) then if(reason == "NO_ANSWER") then -- nothing end if(reason == "USER_BUSY") then -- nothing end end freeswitch.consoleLog(... ------ anotherone is ---- local sess = "nil"; if(argv[1]) then sess=argv[1]; end freeswitch.consoleLog(... api = freeswitch.API(); local res = api:execute("sched_api" ... freeswitch.consoleLog(... ---- and the scheduled script does --- function log(msg) freeswitch.consoleLog("notice", "c2c-hangup-timeout.lua: " .. msg .. "\n"); end local sess = argv[1]; if(sess) then freeswitch.consoleLog("INFO", "hangup-timeout.lua for uuid " .. sess .. "\n"); api = freeswitch.API(); local stillValid = api:execute("uuid_getvar", sess .. " Dummy-DoesChannelExists"); if(stillValid:sub(1,4) == "-ERR") then log("session uuid " .. sess .. " disappeared (nothing bad)"); else -- this is important!!! Otherwise the aleg get's just hung up! api:execute("uuid_media", sess); api:execute("uuid_transfer", sess .. " -both timeout"); end else -- /if(sess) log("called with nil session?"); end -- /if(sess) --- i don't have to do some kind of "free()" on all local variables, do i? Thx. Beni.
You have to reduce the load when running valgrind. On Wed, Sep 2, 2009 at 7:59 AM, Benedikt Fraunhofer < fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello *,
2009/8/31 Rupa Schomaker <rupa@rupa.com>:
Isn't there a known issue with lua+sql leaking memory on some platforms?
just lua, no sql in use :)
On Mon, Aug 31, 2009 at 8:32 AM, Brian West<brian@freeswitch.org> wrote:
Use valgrind.
i tried that... in the beginning valgrind segfaultet several times with some error message that it's unable to monitor so many threads.
vg_alloc_ThreadState: no free slots available Increase VG_N_THREADS, rebuild and try again.
valgrind: the 'impossible' happened: VG_N_THREADS is too low
it happeneded with 400 concurrent calls pretty soon, with 100 it took a while, but at least it consumed half a gig of ram.
Can someone interpret the file it generated so far? i put it at
http://ns42.ath.cx/B0GdWh/vg.log.bz2
(5k compressed, 452k uncompressed) or is there just not enough information contained?
I'll give it another shot with 50 concurrent calls but i guess this will take ages...
Thx in advance
Beni.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
run it slower and make sure it shuts down clean. valgrind --tool=memcheck --log-file-exactly=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /path/to/freeswitch -vg On Wed, Sep 2, 2009 at 10:11 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
You have to reduce the load when running valgrind.
On Wed, Sep 2, 2009 at 7:59 AM, Benedikt Fraunhofer < fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello *,
2009/8/31 Rupa Schomaker <rupa@rupa.com>:
Isn't there a known issue with lua+sql leaking memory on some platforms?
just lua, no sql in use :)
On Mon, Aug 31, 2009 at 8:32 AM, Brian West<brian@freeswitch.org> wrote:
Use valgrind.
i tried that... in the beginning valgrind segfaultet several times with some error message that it's unable to monitor so many threads.
vg_alloc_ThreadState: no free slots available Increase VG_N_THREADS, rebuild and try again.
valgrind: the 'impossible' happened: VG_N_THREADS is too low
it happeneded with 400 concurrent calls pretty soon, with 100 it took a while, but at least it consumed half a gig of ram.
Can someone interpret the file it generated so far? i put it at
http://ns42.ath.cx/B0GdWh/vg.log.bz2
(5k compressed, 452k uncompressed) or is there just not enough information contained?
I'll give it another shot with 50 concurrent calls but i guess this will take ages...
Thx in advance
Beni.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hello Anthony, 2009/9/2 Anthony Minessale <anthony.minessale@gmail.com>:
run it slower and make sure it shuts down clean.
i already reduced load 37% but that didnt help, now i'm down to 25% and it's running.
valgrind --tool=memcheck --log-file-exactly=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /path/to/freeswitch -vg
that's the line in the wiki? I used that but "log-file-exactly" is not available in my valgrind-3.3.0-Debian, just "log-file". Does this make any difference? Also it wont shut down cleanly but oom-coredump (some assertion failed) and if i stop the loadgen after some time and try to 'fsctl shutdown' it wont cleanly bring it down, too. (see first post, last line printed is switch_core_memory.c:567 Stopping memory pool queue. ) Thx. Beni.
i mean valgrind is very intensive so you must run very slow 1-5cps yes if you have a version that only has log-file you can use that. if you find me on irc and send me the credentials privately I will examine your box for you. On Wed, Sep 2, 2009 at 10:26 AM, Benedikt Fraunhofer < fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello Anthony,
2009/9/2 Anthony Minessale <anthony.minessale@gmail.com>:
run it slower and make sure it shuts down clean.
i already reduced load 37% but that didnt help, now i'm down to 25% and it's running.
valgrind --tool=memcheck --log-file-exactly=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /path/to/freeswitch -vg
that's the line in the wiki? I used that but "log-file-exactly" is not available in my valgrind-3.3.0-Debian, just "log-file". Does this make any difference?
Also it wont shut down cleanly but oom-coredump (some assertion failed) and if i stop the loadgen after some time and try to 'fsctl shutdown' it wont cleanly bring it down, too. (see first post, last line printed is
switch_core_memory.c:567 Stopping memory pool queue. )
Thx.
Beni.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hello Anthony, 2009/9/2 Anthony Minessale <anthony.minessale@gmail.com>:
yes if you have a version that only has log-file you can use that.
if you find me on irc and send me the credentials privately I will examine your box for you.
thanks for that offer, but the box is pretty deep inside our internal network with no routing to the outside, several stepping-stones in between and all that "security" stuff. I finally found the right amount of load where the memory leak builds up quickly enough and was able to stop freeswitch before it started swapping. The result is available on http://ns42.ath.cx/B0GdWh/vg-2.log.bz2 (19k) Thx in advance Beni.
Worst offenders (leakers over 100K). The last one is the worst (672M) -- looks like a lua script. What are you doing in lua again? ==28624== 105,725 bytes in 1,804 blocks are still reachable in loss record 497 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x50384F2: xmlrpc_strdupnull (asprintf.c:92) ==28624== by 0x503F86D: RequestRead (http.c:57) ==28624== by 0x5044413: ??? (server.c:538) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 116,772 bytes in 3,156 blocks are definitely lost in loss record 498 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x64A12EA: ??? (mod_dptools.c:633) ==28624== by 0x409AA45: switch_core_session_exec (switch_core_session.c:1476) ==28624== by 0x409AF88: switch_core_session_execute_application (switch_core_session.c:1398) ==28624== by 0x409E674: switch_core_session_run (switch_core_state_machine.c:166) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== ==28624== ==28624== 119,658 (119,621 direct, 37 indirect) bytes in 3,233 blocks are definitely lost in loss record 499 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x50B6790: sofia_event_callback (sofia.c:3863) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DD75: su_base_port_step (su_base_port.c:454) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 124,209 bytes in 3,357 blocks are still reachable in loss record 500 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x409A5EC: switch_core_session_thread (switch_core_session.c:1086) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 124,290 bytes in 4,143 blocks are still reachable in loss record 501 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x443B957: vasprintf (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x5038532: xmlrpc_vasprintf (asprintf.c:61) ==28624== by 0x5038581: xmlrpc_asprintf (asprintf.c:81) ==28624== by 0x503B881: DateToString (date.c:43) ==28624== by 0x5036D09: handler_hook (mod_xml_rpc.c:733) ==28624== by 0x504456F: ??? (server.c:515) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 137,085 bytes in 3,705 blocks are still reachable in loss record 502 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x409921F: switch_core_session_perform_destroy (switch_core_session.c:947) ==28624== by 0x409A60D: switch_core_session_thread (switch_core_session.c:1088) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 145,589 bytes in 1,837 blocks are possibly lost in loss record 503 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x50384F2: xmlrpc_strdupnull (asprintf.c:92) ==28624== by 0x503F86D: RequestRead (http.c:57) ==28624== by 0x5044413: ??? (server.c:538) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 151,929 (151,922 direct, 7 indirect) bytes in 4,106 blocks are definitely lost in loss record 504 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x40C124A: audio_bridge_on_exchange_media (switch_ivr_bridge.c:503) ==28624== by 0x409DB8B: switch_core_session_run (switch_core_state_machine.c:494) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 200,704 bytes in 1 blocks are still reachable in loss record 505 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410C0EC: apr_palloc (apr_pools.c:300) ==28624== by 0x4101A3A: apr_queue_create (apr_queue.c:129) ==28624== by 0x407FFEA: switch_queue_create (switch_apr.c:897) ==28624== by 0x509F2B5: mod_sofia_load (mod_sofia.c:3371) ==28624== by 0x40AF30D: switch_loadable_module_load_module_ex (switch_loadable_module.c:846) ==28624== by 0x40AFCCF: switch_loadable_module_init (switch_loadable_module.c:1174) ==28624== by 0x40A8320: switch_core_init_and_modload (switch_core.c:1451) ==28624== by 0x804A7EC: main (switch.c:731) ==28624== ==28624== ==28624== 200,704 bytes in 1 blocks are still reachable in loss record 506 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410C0EC: apr_palloc (apr_pools.c:300) ==28624== by 0x4101A3A: apr_queue_create (apr_queue.c:129) ==28624== by 0x407FFEA: switch_queue_create (switch_apr.c:897) ==28624== by 0x509F295: mod_sofia_load (mod_sofia.c:3370) ==28624== by 0x40AF30D: switch_loadable_module_load_module_ex (switch_loadable_module.c:846) ==28624== by 0x40AFCCF: switch_loadable_module_init (switch_loadable_module.c:1174) ==28624== by 0x40A8320: switch_core_init_and_modload (switch_core.c:1451) ==28624== by 0x804A7EC: main (switch.c:731) ==28624== ==28624== ==28624== 225,280 bytes in 11 blocks are still reachable in loss record 507 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x503AC39: ??? (mallocvar.h:43) ==28624== by 0x503ACBD: PoolAlloc (data.c:602) ==28624== by 0x503AD2C: PoolStrdup (data.c:674) ==28624== by 0x5043442: MIMETypeAdd2 (response.c:356) ==28624== by 0x50434DD: MIMETypeAdd (response.c:415) ==28624== by 0x503586D: mod_xml_rpc_runtime (mod_xml_rpc.c:936) ==28624== by 0x40AF772: switch_loadable_module_exec (switch_loadable_module.c:94) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 246,312 bytes in 311 blocks are possibly lost in loss record 508 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x50D7759: sofia_glue_do_invite (sofia_glue.c:1677) ==28624== by 0x50A2400: sofia_on_init (mod_sofia.c:102) ==28624== by 0x409D587: switch_core_session_run (switch_core_state_machine.c:481) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 303,104 bytes in 37 blocks are still reachable in loss record 509 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410BB2F: apr_pool_create_ex (apr_pools.c:300) ==28624== by 0x4092416: switch_core_perform_new_memory_pool (switch_core_memory.c:357) ==28624== by 0x40AF24E: switch_loadable_module_load_module_ex (switch_loadable_module.c:785) ==28624== by 0x40AFCCF: switch_loadable_module_init (switch_loadable_module.c:1174) ==28624== by 0x40A8320: switch_core_init_and_modload (switch_core.c:1451) ==28624== by 0x804A7EC: main (switch.c:731) ==28624== ==28624== ==28624== 399,637 (399,600 direct, 37 indirect) bytes in 10,800 blocks are definitely lost in loss record 510 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x50B2A5E: sofia_handle_sip_i_bye (sofia.c:327) ==28624== by 0x50B4A76: sofia_event_callback (sofia.c:508) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DD75: su_base_port_step (su_base_port.c:454) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== ==28624== ==28624== 444,777 bytes in 12,021 blocks are definitely lost in loss record 511 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x50B6790: sofia_event_callback (sofia.c:3863) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DE27: su_base_port_step (su_base_port.c:473) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 2,261,847 (2,261,810 direct, 37 indirect) bytes in 61,130 blocks are definitely lost in loss record 512 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408A4B7: switch_channel_perform_mark_answered (switch_channel.c:1914) ==28624== by 0x50B8528: sofia_event_callback (sofia.c:3807) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DD75: su_base_port_step (su_base_port.c:454) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 2,413,880 (2,413,843 direct, 37 indirect) bytes in 65,239 blocks are definitely lost in loss record 513 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x50B2A5E: sofia_handle_sip_i_bye (sofia.c:327) ==28624== by 0x50B4A76: sofia_event_callback (sofia.c:508) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DE27: su_base_port_step (su_base_port.c:473) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== ==28624== ==28624== 2,416,640 bytes in 295 blocks are still reachable in loss record 514 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410C0EC: apr_palloc (apr_pools.c:300) ==28624== by 0x41110E6: apr_thread_create (thread.c:150) ==28624== by 0x4080878: switch_thread_create (switch_apr.c:631) ==28624== by 0x6C278E9: lua_thread (mod_lua.cpp:372) ==28624== by 0x6C27948: ??? (mod_lua.cpp:407) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x583A7FC: ??? (mod_commands.c:2426) ==28624== by 0x40A8881: switch_scheduler_execute (switch_scheduler.c:61) ==28624== by 0x40A8DE0: task_thread_loop (switch_scheduler.c:127) ==28624== by 0x40A8EA3: switch_scheduler_task_thread (switch_scheduler.c:168) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== ==28624== ==28624== 2,671,215 (2,671,178 direct, 37 indirect) bytes in 72,194 blocks are definitely lost in loss record 515 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408ACF4: switch_channel_perform_mark_ring_ready (switch_channel.c:1697) ==28624== by 0x50B6E29: sofia_event_callback (sofia.c:3366) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DD75: su_base_port_step (su_base_port.c:454) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 2,826,171 bytes in 76,383 blocks are definitely lost in loss record 516 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x4088664: switch_channel_perform_hangup (switch_channel.c:1674) ==28624== by 0x40C1049: signal_bridge_on_hangup (switch_ivr_bridge.c:710) ==28624== by 0x409D7CF: switch_core_session_run (switch_core_state_machine.c:434) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 2,968,658 bytes in 80,234 blocks are definitely lost in loss record 517 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x692F07B: ??? (mod_dialplan_xml.c:315) ==28624== by 0x409EFFD: switch_core_session_run (switch_core_state_machine.c:109) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 2,974,282 (2,974,245 direct, 37 indirect) bytes in 80,385 blocks are definitely lost in loss record 518 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x40DDF32: switch_ivr_session_transfer (switch_ivr.c:1350) ==28624== by 0x5840B54: ??? (mod_commands.c:2319) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x5036FAA: handler_hook (mod_xml_rpc.c:777) ==28624== by 0x504456F: ??? (server.c:515) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== ==28624== ==28624== 2,976,983 (2,976,909 direct, 74 indirect) bytes in 80,457 blocks are definitely lost in loss record 519 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408B204: switch_channel_set_name (switch_channel.c:602) ==28624== by 0x50D3FDD: sofia_glue_attach_private (sofia_glue.c:527) ==28624== by 0x50A35EF: sofia_outgoing_channel (mod_sofia.c:2854) ==28624== by 0x409B970: switch_core_session_outgoing_channel (switch_core_session.c:410) ==28624== by 0x40C53D8: switch_ivr_originate (switch_ivr_originate.c:1508) ==28624== by 0x64A7F2F: ??? (mod_dptools.c:2092) ==28624== by 0x409AA45: switch_core_session_exec (switch_core_session.c:1476) ==28624== by 0x409AF88: switch_core_session_execute_application (switch_core_session.c:1398) ==28624== ==28624== ==28624== 3,085,393 bytes in 83,389 blocks are definitely lost in loss record 520 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408ACF4: switch_channel_perform_mark_ring_ready (switch_channel.c:1697) ==28624== by 0x50B6E29: sofia_event_callback (sofia.c:3366) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DE27: su_base_port_step (su_base_port.c:473) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 3,110,961 (3,107,016 direct, 3,945 indirect) bytes in 3,923 blocks are definitely lost in loss record 521 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x50D7759: sofia_glue_do_invite (sofia_glue.c:1677) ==28624== by 0x50A2400: sofia_on_init (mod_sofia.c:102) ==28624== by 0x409D587: switch_core_session_run (switch_core_state_machine.c:481) ==28624== by 0x409A48E: switch_core_session_thread (switch_core_session.c:1066) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 3,550,402 (3,550,261 direct, 141 indirect) bytes in 95,953 blocks are definitely lost in loss record 522 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408B204: switch_channel_set_name (switch_channel.c:602) ==28624== by 0x50D3FDD: sofia_glue_attach_private (sofia_glue.c:527) ==28624== by 0x50A35EF: sofia_outgoing_channel (mod_sofia.c:2854) ==28624== by 0x409B970: switch_core_session_outgoing_channel (switch_core_session.c:410) ==28624== by 0x40C53D8: switch_ivr_originate (switch_ivr_originate.c:1508) ==28624== by 0x5840AF1: ??? (mod_commands.c:2285) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x5036FAA: handler_hook (mod_xml_rpc.c:777) ==28624== ==28624== ==28624== 3,687,864 (3,687,790 direct, 74 indirect) bytes in 99,670 blocks are definitely lost in loss record 523 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x408A4B7: switch_channel_perform_mark_answered (switch_channel.c:1914) ==28624== by 0x50B8528: sofia_event_callback (sofia.c:3807) ==28624== by 0x5146787: nua_application_event (nua_stack.c:393) ==28624== by 0x519DB28: su_base_port_execute_msgs (su_base_port.c:280) ==28624== by 0x519D8CF: su_base_port_getmsgs (su_base_port.c:202) ==28624== by 0x519DE27: su_base_port_step (su_base_port.c:473) ==28624== by 0x5190968: su_port_step (su_port.h:340) ==28624== by 0x5190938: su_root_step (su_root.c:858) ==28624== ==28624== ==28624== 4,833,280 bytes in 590 blocks are still reachable in loss record 524 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410C0EC: apr_palloc (apr_pools.c:300) ==28624== by 0x41112CF: apr_threadattr_create (thread.c:45) ==28624== by 0x4080953: switch_threadattr_create (switch_apr.c:589) ==28624== by 0x6C27891: lua_thread (mod_lua.cpp:369) ==28624== by 0x6C27948: ??? (mod_lua.cpp:407) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x583A7FC: ??? (mod_commands.c:2426) ==28624== by 0x40A8881: switch_scheduler_execute (switch_scheduler.c:61) ==28624== by 0x40A8DE0: task_thread_loop (switch_scheduler.c:127) ==28624== by 0x40A8EA3: switch_scheduler_task_thread (switch_scheduler.c:168) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== ==28624== ==28624== 6,502,484 (6,502,269 direct, 215 indirect) bytes in 175,737 blocks are definitely lost in loss record 525 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x409921F: switch_core_session_perform_destroy (switch_core_session.c:947) ==28624== by 0x409A60D: switch_core_session_thread (switch_core_session.c:1088) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 6,515,323 (6,515,219 direct, 104 indirect) bytes in 176,087 blocks are definitely lost in loss record 526 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x40E4EA9: switch_log_vprintf (switch_log.c:438) ==28624== by 0x40E5130: switch_log_printf (switch_log.c:308) ==28624== by 0x409A5EC: switch_core_session_thread (switch_core_session.c:1086) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 7,557,720 bytes in 251,924 blocks are definitely lost in loss record 527 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x443B957: vasprintf (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x5038532: xmlrpc_vasprintf (asprintf.c:61) ==28624== by 0x5038581: xmlrpc_asprintf (asprintf.c:81) ==28624== by 0x503B881: DateToString (date.c:43) ==28624== by 0x5036D09: handler_hook (mod_xml_rpc.c:733) ==28624== by 0x504456F: ??? (server.c:515) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 8,700,546 (8,700,436 direct, 110 indirect) bytes in 253,570 blocks are definitely lost in loss record 528 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x444AFCF: strdup (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== by 0x50384F2: xmlrpc_strdupnull (asprintf.c:92) ==28624== by 0x503F86D: RequestRead (http.c:57) ==28624== by 0x5044413: ??? (server.c:538) ==28624== by 0x5039FAF: ??? (conn.c:37) ==28624== by 0x50486F1: ??? (thread_pthread.c:48) ==28624== by 0x42114FA: start_thread (in /lib/tls/i686/cmov/libpthread-2.7.so) ==28624== by 0x44AFE5D: clone (in /lib/tls/i686/cmov/libc-2.7.so) ==28624== ==28624== ==28624== 672,268,288 bytes in 82,064 blocks are still reachable in loss record 529 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410BB2F: apr_pool_create_ex (apr_pools.c:300) ==28624== by 0x4111176: apr_thread_create (thread.c:171) ==28624== by 0x4080878: switch_thread_create (switch_apr.c:631) ==28624== by 0x6C278E9: lua_thread (mod_lua.cpp:372) ==28624== by 0x6C27948: ??? (mod_lua.cpp:407) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x583A7FC: ??? (mod_commands.c:2426) ==28624== by 0x40A8881: switch_scheduler_execute (switch_scheduler.c:61) ==28624== by 0x40A8DE0: task_thread_loop (switch_scheduler.c:127) ==28624== by 0x40A8EA3: switch_scheduler_task_thread (switch_scheduler.c:168) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== ==28624== LEAK SUMMARY: ==28624== definitely lost: 63,113,740 bytes in 1,690,880 blocks. ==28624== indirectly lost: 35,632 bytes in 491 blocks. ==28624== possibly lost: 645,758 bytes in 9,150 blocks. ==28624== still reachable: 681,849,684 bytes in 113,077 blocks. ==28624== suppressed: 0 bytes in 0 blocks. On Fri, Sep 4, 2009 at 7:42 AM, Benedikt Fraunhofer<fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello Anthony,
2009/9/2 Anthony Minessale <anthony.minessale@gmail.com>:
yes if you have a version that only has log-file you can use that.
if you find me on irc and send me the credentials privately I will examine your box for you.
thanks for that offer, but the box is pretty deep inside our internal network with no routing to the outside, several stepping-stones in between and all that "security" stuff.
I finally found the right amount of load where the memory leak builds up quickly enough and was able to stop freeswitch before it started swapping. The result is available on
http://ns42.ath.cx/B0GdWh/vg-2.log.bz2
(19k)
Thx in advance Beni.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
Doesn't that look like a pool that isn't being destroyed? On Fri, Sep 4, 2009 at 9:10 AM, Rupa Schomaker<rupa@rupa.com> wrote:
Worst offenders (leakers over 100K). The last one is the worst (672M) -- looks like a lua script. What are you doing in lua again? ==28624== 672,268,288 bytes in 82,064 blocks are still reachable in loss record 529 of 529 ==28624== at 0x4022AB8: malloc (vg_replace_malloc.c:207) ==28624== by 0x410BB2F: apr_pool_create_ex (apr_pools.c:300) ==28624== by 0x4111176: apr_thread_create (thread.c:171) ==28624== by 0x4080878: switch_thread_create (switch_apr.c:631) ==28624== by 0x6C278E9: lua_thread (mod_lua.cpp:372) ==28624== by 0x6C27948: ??? (mod_lua.cpp:407) ==28624== by 0x40AADFC: switch_api_execute (switch_loadable_module.c:1567) ==28624== by 0x583A7FC: ??? (mod_commands.c:2426) ==28624== by 0x40A8881: switch_scheduler_execute (switch_scheduler.c:61) ==28624== by 0x40A8DE0: task_thread_loop (switch_scheduler.c:127) ==28624== by 0x40A8EA3: switch_scheduler_task_thread (switch_scheduler.c:168) ==28624== by 0x4110E05: dummy_worker (thread.c:138) ==28624== ==28624== LEAK SUMMARY: ==28624== definitely lost: 63,113,740 bytes in 1,690,880 blocks. ==28624== indirectly lost: 35,632 bytes in 491 blocks. ==28624== possibly lost: 645,758 bytes in 9,150 blocks. ==28624== still reachable: 681,849,684 bytes in 113,077 blocks. ==28624== suppressed: 0 bytes in 0 blocks.
-- -Rupa
There are other smaller leakers. xmlrpc is leaking, but the leaks are very small compared to the lua leak. Same with spidermonkey_curl - it is leaking but not too terribly much. I'll hop on #freeswitch in a bit and see if anyone has an idea. On Fri, Sep 4, 2009 at 9:35 AM, Benedikt Fraunhofer<fraunhofer.lists.freeswitch-001@traced.net> wrote:
personally i would blame xmlrpc (which is no xml :) for it.
Just my 2cent
Beni.
-- -Rupa
that looks to me like luarun being called on a script that never terminates. could your script be ending up caught in an endless loop or blocking on something? On Fri, Sep 4, 2009 at 9:42 AM, Rupa Schomaker <rupa@rupa.com> wrote:
There are other smaller leakers. xmlrpc is leaking, but the leaks are very small compared to the lua leak. Same with spidermonkey_curl - it is leaking but not too terribly much. I'll hop on #freeswitch in a bit and see if anyone has an idea.
On Fri, Sep 4, 2009 at 9:35 AM, Benedikt Fraunhofer<fraunhofer.lists.freeswitch-001@traced.net> wrote:
personally i would blame xmlrpc (which is no xml :) for it.
Just my 2cent
Beni.
-- -Rupa
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400
Hello *, the latest bugfixes for luarun (pool allocation) fixed it. It's working now with luarun (friday-monday) and bgapi (monday-today). The last test with the new "&" operator for sched_api is currently running. Thx! Beni.
On Wed, Sep 9, 2009 at 4:58 AM, Benedikt Fraunhofer<fraunhofer.lists.freeswitch-001@traced.net> wrote:
Hello *,
the latest bugfixes for luarun (pool allocation) fixed it. It's working now with luarun (friday-monday) and bgapi (monday-today). The last test with the new "&" operator for sched_api is currently running.
Thx!
Beni.
Great news! Another one bites the dust. :) -- -Rupa
Hello *, "sched_api ... &" works, too. Thx again and looking forward to the next bug :) Beni.
2009/9/4 Rupa Schomaker <rupa@rupa.com>:
Worst offenders (leakers over 100K). The last one is the worst (672M) -- looks like a lua script. What are you doing in lua again?
i feel kinda dumb to double post, but here it is again :) the setup is the same as in http://jira.freeswitch.org/browse/MODSOFIA-22 one is ----- local reason = session:getVariable("originate_disposition"); session:setAutoHangup(false); if(reason) then if(reason == "NO_ANSWER") then -- nothing end if(reason == "USER_BUSY") then -- nothing end end freeswitch.consoleLog(... ------ anotherone is ---- local sess = "nil"; if(argv[1]) then sess=argv[1]; end freeswitch.consoleLog(... api = freeswitch.API(); local res = api:execute("sched_api" ... freeswitch.consoleLog(... ---- and the scheduled script does --- function log(msg) freeswitch.consoleLog("notice", "c2c-hangup-timeout.lua: " .. msg .. "\n"); end local sess = argv[1]; if(sess) then freeswitch.consoleLog("INFO", "hangup-timeout.lua for uuid " .. sess .. "\n"); api = freeswitch.API(); local stillValid = api:execute("uuid_getvar", sess .. " Dummy-DoesChannelExists"); if(stillValid:sub(1,4) == "-ERR") then log("session uuid " .. sess .. " disappeared (nothing bad)"); else -- this is important!!! Otherwise the aleg get's just hung up! api:execute("uuid_media", sess); api:execute("uuid_transfer", sess .. " -both timeout"); end else -- /if(sess) log("called with nil session?"); end -- /if(sess) --- at least there's no fancy db-connection-thingi which could make debugging harder :) Cheers Beni.
participants (7)
-
Anthony Minessale
-
Benedikt Fraunhofer
-
Brian West
-
Diego Viola
-
Jason White
-
Jay Binks
-
Rupa Schomaker