[Freeswitch-users] mem leak and core dump
Freeswitch Freeswitch
freeswitch at 2ride.com
Sun Nov 25 11:05:29 PST 2007
Here's the stack from my latest crash this morning:
was running with 1400 sessions and the ulimit settings recommended by
Anthony
This was after a svn update last night.
more ideas?
Thanks ....
Core was generated by `./freeswitch'.
Program terminated with signal 11, Segmentation fault.
#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value
optimized out>) at src/switch_utils.c:601
601 sprintf(p, "%u%s", i[x], x == 3 ? "" : ".");
(gdb) where
#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value
optimized out>) at src/switch_utils.c:601
#1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788,
payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at src/switch_rtCore
was generated by `./freeswitch'.
Program terminated with signal 11, Segmentation fault.
#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value
optimized out>) at src/switch_utils.c:601
601 sprintf(p, "%u%s", i[x], x == 3 ? "" : ".");
(gdb) where
#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value
optimized out>) at src/switch_utils.c:601
#1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788,
payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at
src/switch_rtp.c:991
#2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame
(rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230
#3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>,
frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized
out>,
stream_id=<value optimized out>) at mod_sofia.c:492
#4 0x00002aaaaab1611b in switch_core_session_read_frame
(session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at
src/switch_core_io.c:113
#5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>,
obj=<value optimized out>) at src/switch_ivr_bridge.c:168
#6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128)
at src/switch_ivr_bridge.c:237
#7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128)
at src/switch_core_state_machine.c:409
#8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value
optimized out>, obj=0x0) at src/switch_core_session.c:681
#9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003baaed062d in clone () from /lib64/libc.so.6
p.c:991
#2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame
(rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230
#3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>,
frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized
out>,
stream_id=<value optimized out>) at mod_sofia.c:492
#4 0x00002aaaaab1611b in switch_core_session_read_frame
(session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at
src/switch_core_io.c:113
#5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>,
obj=<value optimized out>) at src/switch_ivr_bridge.c:168
#6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128)
at src/switch_ivr_bridge.c:237
#7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128)
at src/switch_core_state_machine.c:409
#8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value
optimized out>, obj=0x0) at src/switch_core_session.c:681
#9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003baaed062d in clone () from /lib64/libc.so.6
On 11/25/07, Abdul Hakeem <alhakeem at gmail.com> wrote:
>
> Hi,
> Try reducing the stack size (-) to about 256K and also double the file
> descriptors (-n) to 1024048.
> Ensure the ulimit commands are in the /etc/profile and bashrc.
> Cheers,
> AH
>
> ------------------------------
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Freeswitch
> Freeswitch
> *Sent:* 24 November 2007 16:48
> *To:* freeswitch-users at lists.freeswitch.org
> *Subject:* Re: [Freeswitch-users] mem leak and core dump
>
> Here are my ulimit settings before I adjusted to meet your recommendations
> Anthony.
> I had increased file descriptors and decreased stack size.
>
> t: cpu time (seconds) unlimited
> -f: file size (blocks) unlimited
> -d: data seg size (kbytes) unlimited
> -s: stack size (kbytes) 1024
> -c: core file size (blocks) 0
> -m: resident set size (kbytes) unlimited
> -u: processes 16381
> -n: file descriptors 100000
> -l: locked-in-memory size (kb) 32
> -v: address space (kb) unlimited
> -x: file locks unlimited
> -i: pending signals 16381
> -q: bytes in POSIX msg queues 819200
> -N 13: 0
> -N 14: 0
>
> Here are the settings I am running with now.
>
> -t: cpu time (seconds) unlimited
> -f: file size (blocks) unlimited
> -d: data seg size (kbytes) unlimited
> -s: stack size (kbytes) unlimited
> -c: core file size (blocks) unlimited
> -m: resident set size (kbytes) unlimited
> -u: processes 16381
> -n: file descriptors 999999
> -l: locked-in-memory size (kb) 32
> -v: address space (kb) unlimited
> -x: file locks unlimited
> -i: pending signals 16381
> -q: bytes in POSIX msg queues 819200
> -N 13: 0
> -N 14: 0
>
>
> Here is the stack from my lastest crash.
> I noticed some of the libraries did not seem to be built with debug so I
> cleaned and rebuilt and reinstall everything.
> Currently running for 15 minutes, over 5000 total sessions and 300
> concurrent.
> Finger crossed .......
>
> GNU gdb Red Hat Linux (6.6-8.fc7rh)
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Reading symbols from /lib64/libm.so.6...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from
> /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.
> Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1
> Reading symbols from /usr/lib64/libsqlite3.so.0...done.
> Loaded symbols for /usr/lib64/libsqlite3.so.0
> Reading symbols from /lib64/libuuid.so.1...done.
> Loaded symbols for /lib64/libuuid.so.1
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libcrypt.so.1...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/libpthread.so.0...done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /usr/lib64/libstdc++.so.6...done.
> Loaded symbols for /usr/lib64/libstdc++.so.6
> Reading symbols from /lib64/libgcc_s.so.1...done.
> Loaded symbols for /lib64/libgcc_s.so.1
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld- linux-x86-64.so.2
> Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so
> Reading symbols from /lib64/libssl.so.6...done.
> Loaded symbols for /lib64/libssl.so.6
> Reading symbols from /lib64/libcrypto.so.6...done.
> Loaded symbols for /lib64/libcrypto.so.6
> Reading symbols from /lib64/libz.so.1...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.
> Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
> Reading symbols from /usr/lib64/libkrb5.so.3...done.
> Loaded symbols for /usr/lib64/libkrb5.so.3
> Reading symbols from /lib64/libcom_err.so.2...done.
> Loaded symbols for /lib64/libcom_err.so.2
> Reading symbols from /usr/lib64/libk5crypto.so.3...done.
> Loaded symbols for /usr/lib64/libk5crypto.so.3
> Reading symbols from /usr/lib64/libkrb5support.so.0...done.
> Loaded symbols for /usr/lib64/libkrb5support.so.0
> Reading symbols from /lib64/libkeyutils.so.1...done.
> Loaded symbols for /lib64/libkeyutils.so.1
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so
> Reading symbols from
> /usr/local/freeswitch1121/mod/mod_conference.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error
> while reading shared library symbols:
> DW_FORM_strp pointing outside of .debug_str section [in module
> /usr/local/freeswitch1121/mod/mod_expr.so]
> Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so
> Reading symbols from
> /usr/local/freeswitch1121/mod/mod_voicemail.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so
> Reading symbols from
> /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done.
> Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so
> Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error
> while reading shared library symbols:
> DW_FORM_strp pointing outside of .debug_str section [in module
> /usr/local/freeswitch1121/mod/mod_expr.so]
> Core was generated by `./freeswitch'.
> Program terminated with signal 11, Segmentation fault.
> #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
> a=0x0) at nua_stack.c:513
> 513 if (du->du_refresh && du->du_refresh < next)
> (gdb) where
> #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>,
> a=0x0) at nua_stack.c:513
> #1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>,
> timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at
> su_timer.c:533
> #2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at
> su_base_port.c:328
> #3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value
> optimized out>) at su_pthread_port.c:321
> #4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0
> #5 0x0000003baaed062d in clone () from /lib64/libc.so.6
> (gdb)
>
> After this crash I did a clean rebuild and reinstall - since then I have
> had no subsequent crashes.
>
>
>
> On 11/23/07, Anthony Minessale <anthmct at yahoo.com> wrote:
> >
> > please disable the crash-protection option in switch.conf.xml
> > and execute the command 'ulimit -c unlimited' from your unix shell
> > before you start freeswitch..
> >
> > here are some other ulimits you may want to turn up for high-volume
> > calls:
> >
> > vi u
> > also you can use the unix command "script" to log all the console output
> > to a file so you can capture all of the output for us to examine.
> >
> >
> >
> >
> > Anthony Minessale II
> >
> > FreeSWITCH http://www.freeswitch.org/
> > ClueCon http://www.cluecon.com/
> >
> > AIM: anthm
> > MSN:anthony_minessale at hotmail.com
> > GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> > IRC: irc.freenode.net #freeswitch
> >
> > FreeSWITCH Developer Conference
> > sip:888 at conference.freeswitch.org
> > iax:guest at conference.freeswitch.org/888
> > googletalk:conf+888 at conference.freeswitch.org
> > pstn:213-799-1400
> >
> >
> > ----- Original Message ----
> > From: Freeswitch Freeswitch <freeswitch at 2ride.com>
> > To: freeswitch-users at lists.freeswitch.org
> > Sent: Friday, November 23, 2007 10:47:53 AM
> > Subject: Re: [Freeswitch-users] mem leak and core dump
> >
> > Caught signal 11 for unmapped thread!
> >
> > On 11/23/07, Freeswitch Freeswitch < freeswitch at 2ride.com> wrote:
> > >
> > > Hello,
> > >
> > > Now I have updated to the latest trunk as of Nov 21 and moved to a
> > > fedora 7 system.
> > >
> > > Still chasing random crashes of FS. First time this happened last
> > > night there were around 1000 sessions up.
> > > Crit error was logged referring to a signal caught on an unmapped
> > > thread. I wasn't able to catch the specific signal in the console before it
> > > scrolled off. This morning I had another crash. This time around 200
> > > sessions running. There was no logging reported during the second crash.
> > >
> > > Calling scenario is about as simple as it gets.
> > >
> > > gw1 -> freeswitch -> gw2 all sip
> > > codecs, 729 and 711
> > > transcoding off
> > >
> > > I have been playing with the various rtp knobs in the sofia conf.
> > > Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried
> > > with soft timer and off. ( can someone point me to an explanation of what
> > > the soft timer does? ).
> > >
> > > Also notable that I compiled the code with debug symbols.
> > >
> > > No core has been generated during crashes. Is there some way to
> > > enable core dumps?
> > >
> > > Overall I am very impressed with FS - would love to get the cause of
> > > these crashes nailed so I can press forward.
> > >
> > > Any help is very much appreciated.
> > >
> > > Best regards,
> > >
> > > Tom
> > >
> > >
> > >
> > >
> > >
> > > On 11/19/07, Michael Jerris < mike at jerris.com> wrote:
> > >
> > > > On 11/19/07, Freeswitch Freeswitch <freeswitch at 2ride.com> wrote:
> > > > >
> > > > > Hello Freeswitch'rs,
> > > > >
> > > > > I recently began using the latest freeswitch beta code as a pure
> > > > > B2BUA for the purpose of topology hiding of IP - PSTN calls.
> > > > >
> > > > running env:
> > > > > Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686
> > > > > Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux
> > > > > Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
> > > > >
> > > > > The modules I am using are 711 and 729 codecs, sip, directory,
> > > > > xml_rpc, cdr, xml_cdr, console and syslog.
> > > > >
> > > > > I am seeing what appears to be a memory leak. Consumed memory
> > > > > continues to drop linearly based on number of calls.
> > > > > Also had one core dump.
> > > >
> > > >
> > > > Can you provide details of the call scenarios you are testing, call
> > > > rates, number of concurrent calls and any other details?
> > > >
> > > >
> > > > What can I do to help troubleshoot these issues?
> > > > >
> > > > > Today I plan to stop using the CDR, rpc and logging modules to see
> > > > > if it makes a difference.
> > > >
> > > >
> > > > We have tested quite heavily with mod_console running, that should
> > > > not be an issue.
> > > >
> > > > Also considering running on one of my fedora core 7 boxes instead of
> > > > > gentoo, again to see if anything changes.
> > > >
> > > >
> > > > We have had repeated issues with gentoo due to patches made either
> > > > to gcc or to libc or the kernel. It is the only os/distro that I highly
> > > > suggest never using.
> > > >
> > > > Mike
> > > >
> > > >
> > > > _______________________________________________
> > > > Freeswitch-users mailing list
> > > > Freeswitch-users at lists.freeswitch.org
> > > > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > > > UNSUBSCRIBE:
> > > > http://lists.freeswitch.org/mailman/options/freeswitch-users
> > > > http://www.freeswitch.org
> > > >
> > > >
> > >
> >
> >
> > ------------------------------
> > Get easy, one-click access to your favorites. Make Yahoo! your homepage.<http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:
> > http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20071125/8f3dffc7/attachment-0002.html
More information about the FreeSWITCH-users
mailing list