please disable the crash-protection option in switch.conf.xml and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch.. here are some other ulimits you may want to turn up for high-volume calls: ulimit -d unlimited ulimit -f unlimited ulimit -i unlimited ulimit -n 999999 ulimit -q unlimited ulimit -u unlimited ulimit -v unlimited ulimit -x unlimited ulimit -s unlimited ulimit -l unlimited also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine. Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400 ----- Original Message ---- From: Freeswitch Freeswitch <freeswitch@2ride.com> To: freeswitch-users@lists.freeswitch.org Sent: Friday, November 23, 2007 10:47:53 AM Subject: Re: [Freeswitch-users] mem leak and core dump Caught signal 11 for unmapped thread! On 11/23/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote: Hello, Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system. Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up. Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash. Calling scenario is about as simple as it gets. gw1 -> freeswitch -> gw2 all sip codecs, 729 and 711 transcoding off I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ). Also notable that I compiled the code with debug symbols. No core has been generated during crashes. Is there some way to enable core dumps? Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward. Any help is very much appreciated. Best regards, Tom On 11/19/07, Michael Jerris < mike@jerris.com> wrote: On 11/19/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote: Hello Freeswitch'rs, I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls. running env: Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M) The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog. I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls. Also had one core dump. Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details? What can I do to help troubleshoot these issues? Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference. We have tested quite heavily with mod_console running, that should not be an issue. Also considering running on one of my fedora core 7 boxes instead of gentoo, again to see if anything changes. We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using. Mike _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org ____________________________________________________________________________________ Get easy, one-click access to your favorites. Make Yahoo! your homepage. http://www.yahoo.com/r/hs
Here are my ulimit settings before I adjusted to meet your recommendations Anthony. I had increased file descriptors and decreased stack size. t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 1024 -c: core file size (blocks) 0 -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 100000 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0 Here are the settings I am running with now. -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) unlimited -c: core file size (blocks) unlimited -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 999999 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0 Here is the stack from my lastest crash. I noticed some of the libraries did not seem to be built with debug so I cleaned and rebuilt and reinstall everything. Currently running for 15 minutes, over 5000 total sessions and 300 concurrent. Finger crossed ....... GNU gdb Red Hat Linux (6.6-8.fc7rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... Using host libthread_db library "/lib64/libthread_db.so.1". Reading symbols from /lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done. Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1 Reading symbols from /usr/lib64/libsqlite3.so.0...done. Loaded symbols for /usr/lib64/libsqlite3.so.0 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/librt.so.1...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libpthread.so.0...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libstdc++.so.6...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libz.so.1...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Core was generated by `./freeswitch'. Program terminated with signal 11, Segmentation fault. #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 513 if (du->du_refresh && du->du_refresh < next) (gdb) where #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 #1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533 #2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at su_base_port.c:328 #3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized out>) at su_pthread_port.c:321 #4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003baaed062d in clone () from /lib64/libc.so.6 (gdb) After this crash I did a clean rebuild and reinstall - since then I have had no subsequent crashes. On 11/23/07, Anthony Minessale <anthmct@yahoo.com> wrote:
please disable the crash-protection option in switch.conf.xml and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch..
here are some other ulimits you may want to turn up for high-volume calls:
vi u also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine.
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400
----- Original Message ---- From: Freeswitch Freeswitch <freeswitch@2ride.com> To: freeswitch-users@lists.freeswitch.org Sent: Friday, November 23, 2007 10:47:53 AM Subject: Re: [Freeswitch-users] mem leak and core dump
Caught signal 11 for unmapped thread!
On 11/23/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote:
Hello,
Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system.
Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up. Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash.
Calling scenario is about as simple as it gets.
gw1 -> freeswitch -> gw2 all sip codecs, 729 and 711 transcoding off
I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ).
Also notable that I compiled the code with debug symbols.
No core has been generated during crashes. Is there some way to enable core dumps?
Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward.
Any help is very much appreciated.
Best regards,
Tom
On 11/19/07, Michael Jerris < mike@jerris.com> wrote:
On 11/19/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote:
Hello Freeswitch'rs,
I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls.
running env:
Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.
I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls. Also had one core dump.
Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details?
What can I do to help troubleshoot these issues?
Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference.
We have tested quite heavily with mod_console running, that should not be an issue.
Also considering running on one of my fedora core 7 boxes instead of
gentoo, again to see if anything changes.
We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using.
Mike
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------ Get easy, one-click access to your favorites. Make Yahoo! your homepage.<http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi, Try reducing the stack size (-) to about 256K and also double the file descriptors (-n) to 1024048. Ensure the ulimit commands are in the /etc/profile and bashrc. Cheers, AH _____ From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Freeswitch Freeswitch Sent: 24 November 2007 16:48 To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] mem leak and core dump Here are my ulimit settings before I adjusted to meet your recommendations Anthony. I had increased file descriptors and decreased stack size. t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 1024 -c: core file size (blocks) 0 -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 100000 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0 Here are the settings I am running with now. -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) unlimited -c: core file size (blocks) unlimited -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 999999 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0 Here is the stack from my lastest crash. I noticed some of the libraries did not seem to be built with debug so I cleaned and rebuilt and reinstall everything. Currently running for 15 minutes, over 5000 total sessions and 300 concurrent. Finger crossed ....... GNU gdb Red Hat Linux (6.6-8.fc7rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... Using host libthread_db library "/lib64/libthread_db.so.1". Reading symbols from /lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done. Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1 Reading symbols from /usr/lib64/libsqlite3.so.0...done. Loaded symbols for /usr/lib64/libsqlite3.so.0 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/librt.so.1...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libpthread.so.0...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libstdc++.so.6...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld- linux-x86-64.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libz.so.1...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Core was generated by `./freeswitch'. Program terminated with signal 11, Segmentation fault. #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 513 if (du->du_refresh && du->du_refresh < next) (gdb) where #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 #1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533 #2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at su_base_port.c:328 #3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized out>) at su_pthread_port.c:321 #4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003baaed062d in clone () from /lib64/libc.so.6 (gdb) After this crash I did a clean rebuild and reinstall - since then I have had no subsequent crashes. On 11/23/07, Anthony Minessale <anthmct@yahoo.com> wrote: please disable the crash-protection option in switch.conf.xml and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch.. here are some other ulimits you may want to turn up for high-volume calls: vi u also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine. Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip:888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400 ----- Original Message ---- From: Freeswitch Freeswitch <freeswitch@2ride.com> To: freeswitch-users@lists.freeswitch.org Sent: Friday, November 23, 2007 10:47:53 AM Subject: Re: [Freeswitch-users] mem leak and core dump Caught signal 11 for unmapped thread! On 11/23/07, Freeswitch Freeswitch < <mailto:freeswitch@2ride.com> freeswitch@2ride.com> wrote: Hello, Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system. Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up. Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash. Calling scenario is about as simple as it gets. gw1 -> freeswitch -> gw2 all sip codecs, 729 and 711 transcoding off I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ). Also notable that I compiled the code with debug symbols. No core has been generated during crashes. Is there some way to enable core dumps? Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward. Any help is very much appreciated. Best regards, Tom On 11/19/07, Michael Jerris < <mailto:mike@jerris.com> mike@jerris.com> wrote: On 11/19/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote: Hello Freeswitch'rs, I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls. running env: Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M) The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog. I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls. Also had one core dump. Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details? What can I do to help troubleshoot these issues? Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference. We have tested quite heavily with mod_console running, that should not be an issue. Also considering running on one of my fedora core 7 boxes instead of gentoo, again to see if anything changes. We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using. Mike _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _____ Get easy, one-click access to your favorites. Make Yahoo! your homepage. <http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs> _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org <mailto:Freeswitch-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Here's the stack from my latest crash this morning: was running with 1400 sessions and the ulimit settings recommended by Anthony This was after a svn update last night. more ideas? Thanks .... Core was generated by `./freeswitch'. Program terminated with signal 11, Segmentation fault. #0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601 601 sprintf(p, "%u%s", i[x], x == 3 ? "" : "."); (gdb) where #0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601 #1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788, payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at src/switch_rtCore was generated by `./freeswitch'. Program terminated with signal 11, Segmentation fault. #0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601 601 sprintf(p, "%u%s", i[x], x == 3 ? "" : "."); (gdb) where #0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601 #1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788, payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at src/switch_rtp.c:991 #2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame (rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230 #3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>, frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized out>, stream_id=<value optimized out>) at mod_sofia.c:492 #4 0x00002aaaaab1611b in switch_core_session_read_frame (session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at src/switch_core_io.c:113 #5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:168 #6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128) at src/switch_ivr_bridge.c:237 #7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128) at src/switch_core_state_machine.c:409 #8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value optimized out>, obj=0x0) at src/switch_core_session.c:681 #9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0 #10 0x0000003baaed062d in clone () from /lib64/libc.so.6 p.c:991 #2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame (rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230 #3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>, frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized out>, stream_id=<value optimized out>) at mod_sofia.c:492 #4 0x00002aaaaab1611b in switch_core_session_read_frame (session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at src/switch_core_io.c:113 #5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:168 #6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128) at src/switch_ivr_bridge.c:237 #7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128) at src/switch_core_state_machine.c:409 #8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value optimized out>, obj=0x0) at src/switch_core_session.c:681 #9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0 #10 0x0000003baaed062d in clone () from /lib64/libc.so.6 On 11/25/07, Abdul Hakeem <alhakeem@gmail.com> wrote:
Hi, Try reducing the stack size (-) to about 256K and also double the file descriptors (-n) to 1024048. Ensure the ulimit commands are in the /etc/profile and bashrc. Cheers, AH
------------------------------ *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Freeswitch Freeswitch *Sent:* 24 November 2007 16:48 *To:* freeswitch-users@lists.freeswitch.org *Subject:* Re: [Freeswitch-users] mem leak and core dump
Here are my ulimit settings before I adjusted to meet your recommendations Anthony. I had increased file descriptors and decreased stack size.
t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 1024 -c: core file size (blocks) 0 -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 100000 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0
Here are the settings I am running with now.
-t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) unlimited -c: core file size (blocks) unlimited -m: resident set size (kbytes) unlimited -u: processes 16381 -n: file descriptors 999999 -l: locked-in-memory size (kb) 32 -v: address space (kb) unlimited -x: file locks unlimited -i: pending signals 16381 -q: bytes in POSIX msg queues 819200 -N 13: 0 -N 14: 0
Here is the stack from my lastest crash. I noticed some of the libraries did not seem to be built with debug so I cleaned and rebuilt and reinstall everything. Currently running for 15 minutes, over 5000 total sessions and 300 concurrent. Finger crossed .......
GNU gdb Red Hat Linux (6.6-8.fc7rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... Using host libthread_db library "/lib64/libthread_db.so.1". Reading symbols from /lib64/libm.so.6...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done. Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1 Reading symbols from /usr/lib64/libsqlite3.so.0...done. Loaded symbols for /usr/lib64/libsqlite3.so.0 Reading symbols from /lib64/libuuid.so.1...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /lib64/librt.so.1...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libdl.so.2...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libcrypt.so.1...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libpthread.so.0...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libstdc++.so.6...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld- linux-x86-64.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so Reading symbols from /lib64/libssl.so.6...done. Loaded symbols for /lib64/libssl.so.6 Reading symbols from /lib64/libcrypto.so.6...done. Loaded symbols for /lib64/libcrypto.so.6 Reading symbols from /lib64/libz.so.1...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libkrb5support.so.0...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so Reading symbols from /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done. Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols: DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so] Core was generated by `./freeswitch'. Program terminated with signal 11, Segmentation fault. #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 513 if (du->du_refresh && du->du_refresh < next) (gdb) where #0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513 #1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533 #2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at su_base_port.c:328 #3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized out>) at su_pthread_port.c:321 #4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003baaed062d in clone () from /lib64/libc.so.6 (gdb)
After this crash I did a clean rebuild and reinstall - since then I have had no subsequent crashes.
On 11/23/07, Anthony Minessale <anthmct@yahoo.com> wrote:
please disable the crash-protection option in switch.conf.xml and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch..
here are some other ulimits you may want to turn up for high-volume calls:
vi u also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine.
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:213-799-1400
----- Original Message ---- From: Freeswitch Freeswitch <freeswitch@2ride.com> To: freeswitch-users@lists.freeswitch.org Sent: Friday, November 23, 2007 10:47:53 AM Subject: Re: [Freeswitch-users] mem leak and core dump
Caught signal 11 for unmapped thread!
On 11/23/07, Freeswitch Freeswitch < freeswitch@2ride.com> wrote:
Hello,
Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system.
Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up. Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash.
Calling scenario is about as simple as it gets.
gw1 -> freeswitch -> gw2 all sip codecs, 729 and 711 transcoding off
I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ).
Also notable that I compiled the code with debug symbols.
No core has been generated during crashes. Is there some way to enable core dumps?
Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward.
Any help is very much appreciated.
Best regards,
Tom
On 11/19/07, Michael Jerris < mike@jerris.com> wrote:
On 11/19/07, Freeswitch Freeswitch <freeswitch@2ride.com> wrote:
Hello Freeswitch'rs,
I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls.
running env:
Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @ 2.33GHz GNU/Linux Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)
The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.
I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls. Also had one core dump.
Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details?
What can I do to help troubleshoot these issues?
Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference.
We have tested quite heavily with mod_console running, that should not be an issue.
Also considering running on one of my fedora core 7 boxes instead of
gentoo, again to see if anything changes.
We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using.
Mike
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------ Get easy, one-click access to your favorites. Make Yahoo! your homepage.<http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs>
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (3)
-
Abdul Hakeem
-
Anthony Minessale
-
Freeswitch Freeswitch