[Freeswitch-users] mem leak and core dump
Freeswitch Freeswitch
freeswitch at 2ride.com
Sat Nov 24 08:48:28 PST 2007
Here are my ulimit settings before I adjusted to meet your recommendations
Anthony.
I had increased file descriptors and decreased stack size.
t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 1024
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 100000
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here are the settings I am running with now.
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) unlimited
-c: core file size (blocks) unlimited
-m: resident set size (kbytes) unlimited
-u: processes 16381
-n: file descriptors 999999
-l: locked-in-memory size (kb) 32
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 16381
-q: bytes in POSIX msg queues 819200
-N 13: 0
-N 14: 0
Here is the stack from my lastest crash.
I noticed some of the libraries did not seem to be built with debug so I
cleaned and rebuilt and reinstall everything.
Currently running for 15 minutes, over 5000 total sessions and 300
concurrent.
Finger crossed .......
GNU gdb Red Hat Linux (6.6-8.fc7rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Using host libthread_db library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from
/usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.
Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1
Reading symbols from /usr/lib64/libsqlite3.so.0...done.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /lib64/libuuid.so.1...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so
Reading symbols from /lib64/libssl.so.6...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libz.so.1...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while
reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so]
Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so
Reading symbols from
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done.
Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so
Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while
reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so]
Core was generated by `./freeswitch'.
Program terminated with signal 11, Segmentation fault.
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
a=0x0) at nua_stack.c:513
513 if (du->du_refresh && du->du_refresh < next)
(gdb) where
#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
a=0x0) at nua_stack.c:513
#1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>,
timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at
su_timer.c:533
#2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at
su_base_port.c:328
#3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized
out>) at su_pthread_port.c:321
#4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003baaed062d in clone () from /lib64/libc.so.6
(gdb)
After this crash I did a clean rebuild and reinstall - since then I have had
no subsequent crashes.
On 11/23/07, Anthony Minessale <anthmct at yahoo.com> wrote:
>
> please disable the crash-protection option in switch.conf.xml
> and execute the command 'ulimit -c unlimited' from your unix shell before
> you start freeswitch..
>
> here are some other ulimits you may want to turn up for high-volume calls:
>
> vi u
> also you can use the unix command "script" to log all the console output
> to a file so you can capture all of the output for us to examine.
>
>
>
>
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> iax:guest at conference.freeswitch.org/888
> googletalk:conf+888 at conference.freeswitch.org
> pstn:213-799-1400
>
>
> ----- Original Message ----
> From: Freeswitch Freeswitch <freeswitch at 2ride.com>
> To: freeswitch-users at lists.freeswitch.org
> Sent: Friday, November 23, 2007 10:47:53 AM
> Subject: Re: [Freeswitch-users] mem leak and core dump
>
> Caught signal 11 for unmapped thread!
>
> On 11/23/07, Freeswitch Freeswitch <freeswitch at 2ride.com> wrote:
> >
> > Hello,
> >
> > Now I have updated to the latest trunk as of Nov 21 and moved to a
> > fedora 7 system.
> >
> > Still chasing random crashes of FS. First time this happened last night
> > there were around 1000 sessions up.
> > Crit error was logged referring to a signal caught on an unmapped
> > thread. I wasn't able to catch the specific signal in the console before it
> > scrolled off. This morning I had another crash. This time around 200
> > sessions running. There was no logging reported during the second crash.
> >
> > Calling scenario is about as simple as it gets.
> >
> > gw1 -> freeswitch -> gw2 all sip
> > codecs, 729 and 711
> > transcoding off
> >
> > I have been playing with the various rtp knobs in the sofia conf.
> > Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried
> > with soft timer and off. ( can someone point me to an explanation of what
> > the soft timer does? ).
> >
> > Also notable that I compiled the code with debug symbols.
> >
> > No core has been generated during crashes. Is there some way to enable
> > core dumps?
> >
> > Overall I am very impressed with FS - would love to get the cause of
> > these crashes nailed so I can press forward.
> >
> > Any help is very much appreciated.
> >
> > Best regards,
> >
> > Tom
> >
> >
> >
> >
> >
> > On 11/19/07, Michael Jerris < mike at jerris.com> wrote:
> >
> > > On 11/19/07, Freeswitch Freeswitch <freeswitch at 2ride.com> wrote:
> > > >
> > > > Hello Freeswitch'rs,
> > > >
> > > > I recently began using the latest freeswitch beta code as a pure
> > > > B2BUA for the purpose of topology hiding of IP - PSTN calls.
> > > >
> > > running env:
> > > > Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R)
> > > > Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux
> > > > Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
> > > >
> > > > The modules I am using are 711 and 729 codecs, sip, directory,
> > > > xml_rpc, cdr, xml_cdr, console and syslog.
> > > >
> > > > I am seeing what appears to be a memory leak. Consumed memory
> > > > continues to drop linearly based on number of calls.
> > > > Also had one core dump.
> > >
> > >
> > > Can you provide details of the call scenarios you are testing, call
> > > rates, number of concurrent calls and any other details?
> > >
> > >
> > > What can I do to help troubleshoot these issues?
> > > >
> > > > Today I plan to stop using the CDR, rpc and logging modules to see
> > > > if it makes a difference.
> > >
> > >
> > > We have tested quite heavily with mod_console running, that should not
> > > be an issue.
> > >
> > > Also considering running on one of my fedora core 7 boxes instead of
> > > > gentoo, again to see if anything changes.
> > >
> > >
> > > We have had repeated issues with gentoo due to patches made either to
> > > gcc or to libc or the kernel. It is the only os/distro that I highly
> > > suggest never using.
> > >
> > > Mike
> > >
> > >
> > > _______________________________________________
> > > Freeswitch-users mailing list
> > > Freeswitch-users at lists.freeswitch.org
> > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > > UNSUBSCRIBE:
> > > http://lists.freeswitch.org/mailman/options/freeswitch-users
> > > http://www.freeswitch.org
> > >
> > >
> >
>
>
> ------------------------------
> Get easy, one-click access to your favorites. Make Yahoo! your homepage.<http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20071124/2f26fe80/attachment-0002.html
More information about the FreeSWITCH-users
mailing list