[Freeswitch-users] mem leak and core dump
Anthony Minessale
anthmct at yahoo.com
Sat Nov 24 10:16:43 PST 2007
If the problem persists can you open a bug on jira.
http://jira.freeswitch.org
That way we can track it better and you can attach traces etc.
That crash appears to come from the sofia sip library.
there have been a few updates to that library that may not be applied to your tree can you try the following commands from your unix shell in the main root of the freeswitch src?
svn update
cd libs/sofia-sip
make clean
make
cd ../..
make clean modwipe
make install
what does your dialplan look like that you are testing calls to?
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
iax:guest at conference.freeswitch.org/888
googletalk:conf+888 at conference.freeswitch.org
pstn:213-799-1400
----- Original Message ----
From: Freeswitch Freeswitch <freeswitch at 2ride.com>
To: freeswitch-users at lists.freeswitch.org
Sent: Saturday, November 24, 2007 10:48:28 AM
Subject: Re: [Freeswitch-users] mem leak and core dump
Here are my ulimit settings before I adjusted to meet your recommendations Anthony.
I had increased file descriptors and decreased stack size.
t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 1024
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 100000
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here are the settings I am running with now.
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) unlimited
-c: core file size (blocks) unlimited
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 999999
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here is the stack from my lastest crash.
I noticed some of the libraries did not seem to be built with debug so I cleaned and rebuilt and reinstall everything.
Currently running for 15 minutes, over 5000 total sessions and 300 concurrent.
Finger crossed .......
GNU gdb Red Hat Linux (6.6-8.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.
Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1
Reading symbols from /usr/lib64/libsqlite3.so.0...done.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-
linux-x86-64.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so]
Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so]
Core was generated by `./freeswitch'.
Program terminated with signal 11, Segmentation fault.
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513
513 if (du->du_refresh && du->du_refresh < next)
(gdb) where
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513
#1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533
#2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at su_base_port.c:328
#3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized out>) at su_pthread_port.c:321
#4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003baaed062d in clone () from /lib64/libc.so.6
(gdb)
After this crash I did a clean rebuild and reinstall - since then I have had no subsequent crashes.
On 11/23/07, Anthony Minessale <anthmct at yahoo.com> wrote:
please disable the crash-protection option in switch.conf.xml
and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch..
here are some other ulimits you may want to turn up for high-volume calls:
vi u
also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine.
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon
http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC:
irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
iax:guest at conference.freeswitch.org/888
googletalk:conf+888 at conference.freeswitch.org
pstn:213-799-1400
----- Original Message ----
From: Freeswitch Freeswitch <freeswitch at 2ride.com>
To:
freeswitch-users at lists.freeswitch.org
Sent: Friday, November 23, 2007 10:47:53 AM
Subject: Re: [Freeswitch-users] mem leak and core dump
Caught signal 11 for unmapped thread!
On 11/23/07, Freeswitch Freeswitch <
freeswitch at 2ride.com> wrote:
Hello,
Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system.
Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up.
Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash.
Calling scenario is about as simple as it gets.
gw1 -> freeswitch -> gw2 all sip
codecs, 729 and 711
transcoding off
I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ).
Also notable that I compiled the code with debug symbols.
No core has been generated during crashes. Is there some way to enable core dumps?
Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward.
Any help is very much appreciated.
Best regards,
Tom
On 11/19/07, Michael Jerris
<
mike at jerris.com> wrote:
On 11/19/07,
Freeswitch Freeswitch <freeswitch at 2ride.com> wrote:
Hello Freeswitch'rs,
I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls.
running env:
Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @
2.33GHz GNU/Linux
Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.
I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls.
Also had one core dump.
Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details?
What can I do to help troubleshoot these issues?
Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference.
We have tested quite heavily with mod_console running, that should not be an issue.
Also considering running on one of my fedora core 7 boxes instead of gentoo, again to see if anything changes.
We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using.
Mike
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
____________________________________________________________________________________
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20071124/cb4c19d8/attachment-0002.html
More information about the FreeSWITCH-users
mailing list