Here are my ulimit settings before I adjusted to meet your recommendations Anthony.<br>I had increased file descriptors and decreased stack size.<br><br>t: cpu time (seconds) unlimited<br>-f: file size (blocks) unlimited
<br>-d: data seg size (kbytes) unlimited<br>-s: stack size (kbytes) 1024<br>-c: core file size (blocks) 0<br>-m: resident set size (kbytes) unlimited<br>-u: processes 16381<br>-n: file descriptors 100000
<br>-l: locked-in-memory size (kb) 32<br>-v: address space (kb) unlimited<br>-x: file locks unlimited<br>-i: pending signals 16381<br>-q: bytes in POSIX msg queues 819200<br>-N 13: 0
<br>-N 14: 0<br><br>Here are the settings I am running with now.<br><br>-t: cpu time (seconds) unlimited<br>-f: file size (blocks) unlimited<br>-d: data seg size (kbytes) unlimited
<br>-s: stack size (kbytes) unlimited<br>-c: core file size (blocks) unlimited<br>-m: resident set size (kbytes) unlimited<br>-u: processes 16381<br>-n: file descriptors 999999<br>-l: locked-in-memory size (kb) 32
<br>-v: address space (kb) unlimited<br>-x: file locks unlimited<br>-i: pending signals 16381<br>-q: bytes in POSIX msg queues 819200<br>-N 13: 0<br>-N 14: 0
<br><br><br>Here is the stack from my lastest crash.<br>I noticed some of the libraries did not seem to be built with debug so I cleaned and rebuilt and reinstall everything.<br>Currently running for 15 minutes, over 5000 total sessions and 300 concurrent.
<br>Finger crossed .......<br><br>GNU gdb Red Hat Linux (6.6-8.fc7rh)<br>Copyright (C) 2006 Free Software Foundation, Inc.<br>GDB is free software, covered by the GNU General Public License, and you are<br>welcome to change it and/or distribute copies of it under certain conditions.
<br>Type "show copying" to see the conditions.<br>There is absolutely no warranty for GDB. Type "show warranty" for details.<br>This GDB was configured as "x86_64-redhat-linux-gnu"...<br>Using host libthread_db library "/lib64/libthread_db.so.1".
<br>Reading symbols from /lib64/libm.so.6...done.<br>Loaded symbols for /lib64/libm.so.6<br>Reading symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.<br>Loaded symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1
<br>Reading symbols from /usr/lib64/libsqlite3.so.0...done.<br>Loaded symbols for /usr/lib64/libsqlite3.so.0<br>Reading symbols from /lib64/libuuid.so.1...done.<br>Loaded symbols for /lib64/libuuid.so.1<br>Reading symbols from /lib64/librt.so.1...done.
<br>Loaded symbols for /lib64/librt.so.1<br>Reading symbols from /lib64/libdl.so.2...done.<br>Loaded symbols for /lib64/libdl.so.2<br>Reading symbols from /lib64/libcrypt.so.1...done.<br>Loaded symbols for /lib64/libcrypt.so.1
<br>Reading symbols from /lib64/libpthread.so.0...done.<br>Loaded symbols for /lib64/libpthread.so.0<br>Reading symbols from /lib64/libc.so.6...done.<br>Loaded symbols for /lib64/libc.so.6<br>Reading symbols from /usr/lib64/libstdc++.so.6...done.
<br>Loaded symbols for /usr/lib64/libstdc++.so.6<br>Reading symbols from /lib64/libgcc_s.so.1...done.<br>Loaded symbols for /lib64/libgcc_s.so.1<br>Reading symbols from /lib64/ld-linux-x86-64.so.2...done.<br>Loaded symbols for /lib64/ld-
linux-x86-64.so.2<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_console.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_console.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_syslog.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_syslog.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_xml_rpc.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_sofia.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_sofia.so<br>Reading symbols from /lib64/libssl.so.6...done.<br>Loaded symbols for /lib64/libssl.so.6<br>Reading symbols from /lib64/libcrypto.so.6...done.<br>Loaded symbols for /lib64/libcrypto.so.6
<br>Reading symbols from /lib64/libz.so.1...done.<br>Loaded symbols for /lib64/libz.so.1<br>Reading symbols from /usr/lib64/libgssapi_krb5.so.2...done.<br>Loaded symbols for /usr/lib64/libgssapi_krb5.so.2<br>Reading symbols from /usr/lib64/libkrb5.so.3...done.
<br>Loaded symbols for /usr/lib64/libkrb5.so.3<br>Reading symbols from /lib64/libcom_err.so.2...done.<br>Loaded symbols for /lib64/libcom_err.so.2<br>Reading symbols from /usr/lib64/libk5crypto.so.3...done.<br>Loaded symbols for /usr/lib64/libk5crypto.so.3
<br>Reading symbols from /usr/lib64/libkrb5support.so.0...done.<br>Loaded symbols for /usr/lib64/libkrb5support.so.0<br>Reading symbols from /lib64/libkeyutils.so.1...done.<br>Loaded symbols for /lib64/libkeyutils.so.1<br>
Reading symbols from /lib64/libresolv.so.2...done.<br>Loaded symbols for /lib64/libresolv.so.2<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_commands.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_commands.so
<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_conference.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_conference.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_dptools.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_dptools.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols:<br>DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so]
<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_fifo.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_fifo.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_voicemail.so...done.<br>
Loaded symbols for /usr/local/freeswitch1121/mod/mod_voicemail.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_limit.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_limit.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_dialplan_xml.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_g711.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_g711.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_g729.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_g729.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_l16.so...done.<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_l16.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_say_en.so...done.
<br>Loaded symbols for /usr/local/freeswitch1121/mod/mod_say_en.so<br>Reading symbols from /usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library symbols:<br>DW_FORM_strp pointing outside of .debug_str section [in module /usr/local/freeswitch1121/mod/mod_expr.so]
<br>Core was generated by `./freeswitch'.<br>Program terminated with signal 11, Segmentation fault.<br>#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513<br>
513 if (du->du_refresh && du->du_refresh < next)<br>(gdb) where<br>#0 nua_stack_timer (nua=<value optimized out>, t=<value optimized out>, a=0x0) at nua_stack.c:513<br>#1 0x00002aaaabdb1ae8 in su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8, now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533
<br>#2 0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at su_base_port.c:328<br>#3 0x00002aaaabda7e7b in su_pthread_port_clone_main (varg=<value optimized out>) at su_pthread_port.c:321<br>#4 0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0
<br>#5 0x0000003baaed062d in clone () from /lib64/libc.so.6<br>(gdb)<br><br>After this crash I did a clean rebuild and reinstall - since then I have had no subsequent crashes.<br><br><br><br><div><span class="gmail_quote">
On 11/23/07, <b class="gmail_sendername">Anthony Minessale</b> <<a href="mailto:anthmct@yahoo.com">anthmct@yahoo.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div style="font-family: courier,monaco,monospace,sans-serif; font-size: 12pt;">please disable the crash-protection option in switch.conf.xml<br>and execute the command 'ulimit -c unlimited' from your unix shell before you start freeswitch..
<br><br>here are some other ulimits you may want to turn up for high-volume calls:<br><br>vi u<br>also you can use the unix command "script" to log all the console output to a file so you can capture all of the output for us to examine.
<br><br><br><br><div> </div><div>Anthony Minessale II<br><br><span>FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.freeswitch.org/</a></span>
<br><span>ClueCon
<a href="http://www.cluecon.com/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.cluecon.com/</a></span><br><br>AIM: anthm<br><a href="mailto:MSN:anthony_minessale@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/PAYPAL:<a href="mailto:anthony.minessale@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">anthony.minessale@gmail.com</a><br>IRC: <a href="http://irc.freenode.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
irc.freenode.net</a> #freeswitch</div><div><br>FreeSWITCH Developer Conference<br><a href="mailto:sip:888@conference.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">sip:888@conference.freeswitch.org
</a><br><a href="http://iax:guest@conference.freeswitch.org/888" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">iax:guest@conference.freeswitch.org/888</a><br><a href="mailto:googletalk:conf+888@conference.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
googletalk:conf+888@conference.freeswitch.org</a><br>pstn:213-799-1400</div><div><span class="e" id="q_1166d8f5bca18719_1"><div style="font-family: courier,monaco,monospace,sans-serif; font-size: 12pt;"><br><br><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;">
----- Original Message ----<br>From: Freeswitch Freeswitch <<a href="mailto:freeswitch@2ride.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch@2ride.com</a>><br>To: <a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
freeswitch-users@lists.freeswitch.org</a><br>Sent: Friday, November 23, 2007 10:47:53 AM<br>Subject: Re: [Freeswitch-users] mem leak and core dump<br><br>
Caught signal 11 for unmapped thread!<br><br><div><span class="gmail_quote">On 11/23/07, <b class="gmail_sendername">Freeswitch Freeswitch</b> <<a rel="nofollow" href="mailto:freeswitch@2ride.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
freeswitch@2ride.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br><br>Now I have updated to the latest trunk as of Nov 21 and moved to a fedora 7 system.
<br><br>Still chasing random crashes of FS. First time this happened last night there were around 1000 sessions up.<br>Crit error was logged referring to a signal caught on an unmapped thread. I wasn't able to catch the specific signal in the console before it scrolled off. This morning I had another crash. This time around 200 sessions running. There was no logging reported during the second crash.
<br><br>Calling scenario is about as simple as it gets.<br><br>gw1 -> freeswitch -> gw2 all sip<br>codecs, 729 and 711 <br>transcoding off<br><br>I have been playing with the various rtp knobs in the sofia conf. Sometimes running with rewriting rtp timerstamps, sometimes not. Also tried with soft timer and off. ( can someone point me to an explanation of what the soft timer does? ).
<br><br>Also notable that I compiled the code with debug symbols.<br><br>No core has been generated during crashes. Is there some way to enable core dumps?<br><br>Overall I am very impressed with FS - would love to get the cause of these crashes nailed so I can press forward.
<br><br>Any help is very much appreciated.<br><br>Best regards,<br><br>Tom<br><br><br><br><br><br><div><div><span><span class="gmail_quote">On 11/19/07, <b class="gmail_sendername">Michael Jerris
</b> <<a rel="nofollow" href="mailto:mike@jerris.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
mike@jerris.com</a>> wrote:</span></span></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><span>
<span>On 11/19/07, <b class="gmail_sendername">
Freeswitch Freeswitch</b> <<a rel="nofollow" href="mailto:freeswitch@2ride.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch@2ride.com</a>> wrote:</span><div><span><span class="gmail_quote">
</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello Freeswitch'rs,<br><br>I recently began using the latest freeswitch beta code as a pure B2BUA for the purpose of topology hiding of IP - PSTN calls. <br></blockquote><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
running env:<br>Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686 Intel(R) Xeon(R) CPU 5140 @
2.33GHz GNU/Linux<br>Freeswitch version: FreeSwitch Version 1.0.pre1 (6220M)<br><br>The modules I am using are 711 and 729 codecs, sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.<br><br>I am seeing what appears to be a memory leak. Consumed memory continues to drop linearly based on number of calls.
<br>Also had one core dump.</blockquote></span><div><br>Can you provide details of the call scenarios you are testing, call rates, number of concurrent calls and any other details?<br> </div><span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
What can I do to help troubleshoot these issues?<br><br>Today I plan to stop using the CDR, rpc and logging modules to see if it makes a difference.</blockquote></span><div><br>We have tested quite heavily with mod_console running, that should not be an issue.
<br></div><span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Also considering running on one of my fedora core 7 boxes instead of gentoo, again to see if anything changes.
</blockquote></span><div><br>We have had repeated issues with gentoo due to patches made either to gcc or to libc or the kernel. It is the only os/distro that I highly suggest never using.<br></div><br>Mike<br></div><br>
<br></span></div><span>_______________________________________________<br>Freeswitch-users mailing list<br><a rel="nofollow" href="mailto:Freeswitch-users@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
Freeswitch-users@lists.freeswitch.org
</a><br><a rel="nofollow" href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
</a><br>
UNSUBSCRIBE:
<a rel="nofollow" href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<br><a rel="nofollow" href="http://www.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://www.freeswitch.org</a><br><br></span></blockquote></div><br>
</blockquote></div><br>
</div><br></div></span></div></div><span class="ad"><br>
<hr size="1">Get easy, one-click access to your favorites.
<a href="http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> Make Yahoo! your homepage.</a>
</span></div><br>_______________________________________________<br>Freeswitch-users mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org
</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>UNSUBSCRIBE:
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.freeswitch.org" target="_blank">
http://www.freeswitch.org</a><br><br></blockquote></div><br>