[Freeswitch-users] mem leak and core dump
Abdul Hakeem
alhakeem at gmail.com
Sun Nov 25 01:35:23 PST 2007
Hi,
Try reducing the stack size (-) to about 256K and also double the file
descriptors (-n) to 1024048.
Ensure the ulimit commands are in the /etc/profile and bashrc.
Cheers,
AH
_____
From: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of
Freeswitch Freeswitch
Sent: 24 November 2007 16:48
To: freeswitch-users at lists.freeswitch.org
Subject: Re: [Freeswitch-users] mem leak and core dump
Here are my ulimit settings before I adjusted to meet your recommendations
Anthony.
I had increased file descriptors and decreased stack size.
t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 1024
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 100000
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here are the settings I am running with now.
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) unlimited
-c: core file size (blocks) unlimited
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 999999
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here is the stack from my lastest crash.
I noticed some of the libraries did not seem to be built with debug so I
cleaned and rebuilt and reinstall everything.
Currently running for 15 minutes, over 5000 total sessions and 300
concurrent.
Finger crossed .......
GNU gdb Red Hat Linux (6.6-8.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from
/usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.
Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1
Reading symbols from /usr/lib64/libsqlite3.so.0...done.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld- linux-x86-64.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while
reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so]
Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so
Reading symbols from
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while
reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so]
Core was generated by `./freeswitch'.
Program terminated with signal 11, Segmentation fault.
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
a=0x0) at nua_stack.c:513
513 if (du->du_refresh && du->du_refresh < next)
(gdb) where
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
a=0x0) at nua_stack.c:513
#1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>,
timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at
su_timer.c:533
#2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at
su_base_port.c:328
#3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized
out>) at su_pthread_port.c:321
#4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003baaed062d in clone () from /lib64/libc.so.6
(gdb)
After this crash I did a clean rebuild and reinstall - since then I have had
no subsequent crashes.
On 11/23/07, Anthony Minessale <anthmct at yahoo.com> wrote:
please disable the crash-protection option in switch.conf.xml
and execute the command 'ulimit -c unlimited' from your unix shell before
you start freeswitch..
here are some other ulimits you may want to turn up for high-volume calls:
vi u
also you can use the unix command "script" to log all the console output to
a file so you can capture all of the output for us to examine.
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
<mailto:sip:888 at conference.freeswitch.org>
iax:guest at conference.freeswitch.org/888
googletalk:conf+888 at conference.freeswitch.org
pstn:213-799-1400
----- Original Message ----
From: Freeswitch Freeswitch <freeswitch at 2ride.com>
To: freeswitch-users at lists.freeswitch.org
Sent: Friday, November 23, 2007 10:47:53 AM
Subject: Re: [Freeswitch-users] mem leak and core dump
Caught signal 11 for unmapped thread!
On 11/23/07, Freeswitch Freeswitch < <mailto:freeswitch at 2ride.com>
freeswitch at 2ride.com> wrote:
Hello,
Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7
system.
Still chasing random crashes of FS. First time this happened last night
there were around 1000 sessions up.
Crit error was logged referring to a signal caught on an unmapped thread. I
wasn't able to catch the specific signal in the console before it scrolled
off. This morning I had another crash. This time around 200 sessions
running. There was no logging reported during the second crash.
Calling scenario is about as simple as it gets.
gw1 -> freeswitch -> gw2 all sip
codecs, 729 and 711
transcoding off
I have been playing with the various rtp knobs in the sofia conf. Sometimes
running with rewriting rtp timerstamps, sometimes not. Also tried with soft
timer and off. ( can someone point me to an explanation of what the soft
timer does? ).
Also notable that I compiled the code with debug symbols.
No core has been generated during crashes. Is there some way to enable core
dumps?
Overall I am very impressed with FS - would love to get the cause of these
crashes nailed so I can press forward.
Any help is very much appreciated.
Best regards,
Tom
On 11/19/07, Michael Jerris < <mailto:mike at jerris.com> mike at jerris.com>
wrote:
On 11/19/07, Freeswitch Freeswitch <freeswitch at 2ride.com> wrote:
Hello Freeswitch'rs,
I recently began using the latest freeswitch beta code as a pure B2BUA for
the purpose of topology hiding of IP - PSTN calls.
running env:
Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R)
CPU 5140 @ 2.33GHz GNU/Linux
Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr,
xml_cdr, console and syslog.
I am seeing what appears to be a memory leak. Consumed memory continues to
drop linearly based on number of calls.
Also had one core dump.
Can you provide details of the call scenarios you are testing, call rates,
number of concurrent calls and any other details?
What can I do to help troubleshoot these issues?
Today I plan to stop using the CDR, rpc and logging modules to see if it
makes a difference.
We have tested quite heavily with mod_console running, that should not be an
issue.
Also considering running on one of my fedora core 7 boxes instead of gentoo,
again to see if anything changes.
We have had repeated issues with gentoo due to patches made either to gcc or
to libc or the kernel. It is the only os/distro that I highly suggest never
using.
Mike
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
<mailto:Freeswitch-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
<http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_____
Get easy, one-click access to your favorites. Make Yahoo! your homepage.
<http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
<mailto:Freeswitch-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20071125/d762cf66/attachment-0002.html
More information about the FreeSWITCH-users
mailing list