<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16544" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><SPAN class=953053209-25112007><FONT face="Bodoni MT"
color=#0000ff>Hi,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=953053209-25112007><FONT face="Bodoni MT"
color=#0000ff>Try reducing the stack size (-) to about 256K and also double the
file descriptors (-n) to 1024048.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=953053209-25112007><FONT face="Bodoni MT"
color=#0000ff>Ensure the ulimit commands are in the /etc/profile and
bashrc.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=953053209-25112007><FONT face="Bodoni MT"
color=#0000ff>Cheers,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=953053209-25112007><FONT face="Bodoni MT"
color=#0000ff>AH</FONT> </SPAN></DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B>
freeswitch-users-bounces@lists.freeswitch.org
[mailto:freeswitch-users-bounces@lists.freeswitch.org] <B>On Behalf Of
</B>Freeswitch Freeswitch<BR><B>Sent:</B> 24 November 2007 16:48<BR><B>To:</B>
freeswitch-users@lists.freeswitch.org<BR><B>Subject:</B> Re: [Freeswitch-users]
mem leak and core dump<BR></FONT><BR></DIV>
<DIV></DIV>Here are my ulimit settings before I adjusted to meet your
recommendations Anthony.<BR>I had increased file descriptors and decreased stack
size.<BR><BR>t: cpu time
(seconds) unlimited<BR>-f: file
size (blocks) unlimited <BR>-d:
data seg size (kbytes) unlimited<BR>-s: stack size
(kbytes) 1024<BR>-c: core file size
(blocks) 0<BR>-m: resident set size (kbytes) unlimited<BR>-u:
processes
16381<BR>-n: file
descriptors 100000
<BR>-l: locked-in-memory size (kb) 32<BR>-v: address space
(kb) unlimited<BR>-x: file
locks
unlimited<BR>-i: pending
signals
16381<BR>-q: bytes in POSIX msg queues 819200<BR>-N
13:
0 <BR>-N
14:
0<BR><BR>Here are the settings I am running with now.<BR><BR>-t: cpu time
(seconds) unlimited<BR>-f: file
size (blocks) unlimited<BR>-d:
data seg size (kbytes) unlimited <BR>-s: stack size
(kbytes) unlimited<BR>-c: core file
size (blocks) unlimited<BR>-m: resident set size (kbytes)
unlimited<BR>-u:
processes
16381<BR>-n: file
descriptors
999999<BR>-l: locked-in-memory size (kb) 32 <BR>-v: address space
(kb) unlimited<BR>-x: file
locks
unlimited<BR>-i: pending
signals
16381<BR>-q: bytes in POSIX msg queues 819200<BR>-N
13:
0<BR>-N
14:
0 <BR><BR><BR>Here is the stack from my lastest crash.<BR>I noticed some of the
libraries did not seem to be built with debug so I cleaned and rebuilt and
reinstall everything.<BR>Currently running for 15 minutes, over 5000 total
sessions and 300 concurrent. <BR>Finger crossed .......<BR><BR>GNU gdb Red Hat
Linux (6.6-8.fc7rh)<BR>Copyright (C) 2006 Free Software Foundation, Inc.<BR>GDB
is free software, covered by the GNU General Public License, and you
are<BR>welcome to change it and/or distribute copies of it under certain
conditions. <BR>Type "show copying" to see the conditions.<BR>There is
absolutely no warranty for GDB. Type "show warranty" for details.<BR>This
GDB was configured as "x86_64-redhat-linux-gnu"...<BR>Using host libthread_db
library "/lib64/libthread_db.so.1". <BR>Reading symbols from
/lib64/libm.so.6...done.<BR>Loaded symbols for /lib64/libm.so.6<BR>Reading
symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.<BR>Loaded
symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1 <BR>Reading symbols
from /usr/lib64/libsqlite3.so.0...done.<BR>Loaded symbols for
/usr/lib64/libsqlite3.so.0<BR>Reading symbols from
/lib64/libuuid.so.1...done.<BR>Loaded symbols for /lib64/libuuid.so.1<BR>Reading
symbols from /lib64/librt.so.1...done. <BR>Loaded symbols for
/lib64/librt.so.1<BR>Reading symbols from /lib64/libdl.so.2...done.<BR>Loaded
symbols for /lib64/libdl.so.2<BR>Reading symbols from
/lib64/libcrypt.so.1...done.<BR>Loaded symbols for /lib64/libcrypt.so.1
<BR>Reading symbols from /lib64/libpthread.so.0...done.<BR>Loaded symbols for
/lib64/libpthread.so.0<BR>Reading symbols from
/lib64/libc.so.6...done.<BR>Loaded symbols for /lib64/libc.so.6<BR>Reading
symbols from /usr/lib64/libstdc++.so.6...done. <BR>Loaded symbols for
/usr/lib64/libstdc++.so.6<BR>Reading symbols from
/lib64/libgcc_s.so.1...done.<BR>Loaded symbols for
/lib64/libgcc_s.so.1<BR>Reading symbols from
/lib64/ld-linux-x86-64.so.2...done.<BR>Loaded symbols for /lib64/ld-
linux-x86-64.so.2<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_console.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_console.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_syslog.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_syslog.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_xml_rpc.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_sofia.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_sofia.so<BR>Reading symbols from
/lib64/libssl.so.6...done.<BR>Loaded symbols for /lib64/libssl.so.6<BR>Reading
symbols from /lib64/libcrypto.so.6...done.<BR>Loaded symbols for
/lib64/libcrypto.so.6 <BR>Reading symbols from
/lib64/libz.so.1...done.<BR>Loaded symbols for /lib64/libz.so.1<BR>Reading
symbols from /usr/lib64/libgssapi_krb5.so.2...done.<BR>Loaded symbols for
/usr/lib64/libgssapi_krb5.so.2<BR>Reading symbols from
/usr/lib64/libkrb5.so.3...done. <BR>Loaded symbols for
/usr/lib64/libkrb5.so.3<BR>Reading symbols from
/lib64/libcom_err.so.2...done.<BR>Loaded symbols for
/lib64/libcom_err.so.2<BR>Reading symbols from
/usr/lib64/libk5crypto.so.3...done.<BR>Loaded symbols for
/usr/lib64/libk5crypto.so.3 <BR>Reading symbols from
/usr/lib64/libkrb5support.so.0...done.<BR>Loaded symbols for
/usr/lib64/libkrb5support.so.0<BR>Reading symbols from
/lib64/libkeyutils.so.1...done.<BR>Loaded symbols for
/lib64/libkeyutils.so.1<BR>Reading symbols from
/lib64/libresolv.so.2...done.<BR>Loaded symbols for
/lib64/libresolv.so.2<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_commands.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_commands.so <BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_conference.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_conference.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_dptools.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_dptools.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library
symbols:<BR>DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so] <BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_fifo.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_fifo.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_voicemail.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_voicemail.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_limit.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_limit.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_g711.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_g711.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_g729.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_g729.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_l16.so...done.<BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_l16.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_say_en.so...done. <BR>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_say_en.so<BR>Reading symbols from
/usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library
symbols:<BR>DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so] <BR>Core was generated by
`./freeswitch'.<BR>Program terminated with signal 11, Segmentation
fault.<BR>#0 nua_stack_timer (nua=<value optimized out>, t=<value
optimized out>, a=0x0) at
nua_stack.c:513<BR>513 if
(du->du_refresh && du->du_refresh < next)<BR>(gdb)
where<BR>#0 nua_stack_timer (nua=<value optimized out>, t=<value
optimized out>, a=0x0) at nua_stack.c:513<BR>#1 0x00002aaaabdb1ae8 in
su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8,
now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533 <BR>#2
0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at
su_base_port.c:328<BR>#3 0x00002aaaabda7e7b in su_pthread_port_clone_main
(varg=<value optimized out>) at su_pthread_port.c:321<BR>#4
0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0 <BR>#5
0x0000003baaed062d in clone () from /lib64/libc.so.6<BR>(gdb)<BR><BR>After this
crash I did a clean rebuild and reinstall - since then I have had no subsequent
crashes.<BR><BR><BR><BR>
<DIV><SPAN class=gmail_quote>On 11/23/07, <B class=gmail_sendername>Anthony
Minessale</B> <<A href="mailto:anthmct@yahoo.com">anthmct@yahoo.com</A>>
wrote:</SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>
<DIV
style="FONT-SIZE: 12pt; FONT-FAMILY: courier,monaco,monospace,sans-serif">please
disable the crash-protection option in switch.conf.xml<BR>and execute the
command 'ulimit -c unlimited' from your unix shell before you start
freeswitch.. <BR><BR>here are some other ulimits you may want to turn up for
high-volume calls:<BR><BR>vi u<BR>also you can use the unix command "script"
to log all the console output to a file so you can capture all of the output
for us to examine. <BR><BR><BR><BR>
<DIV> </DIV>
<DIV>Anthony Minessale II<BR><BR><SPAN>FreeSWITCH <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://www.freeswitch.org/"
target=_blank>http://www.freeswitch.org/</A></SPAN> <BR><SPAN>ClueCon <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://www.cluecon.com/"
target=_blank>http://www.cluecon.com/</A></SPAN><BR><BR>AIM: anthm<BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:MSN:anthony_minessale@hotmail.com"
target=_blank>MSN:anthony_minessale@hotmail.com</A><BR>GTALK/JABBER/PAYPAL:<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:anthony.minessale@gmail.com"
target=_blank>anthony.minessale@gmail.com</A><BR>IRC: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://irc.freenode.net" target=_blank>irc.freenode.net</A>
#freeswitch</DIV>
<DIV><BR>FreeSWITCH Developer Conference<BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:sip:888@conference.freeswitch.org"
target=_blank>sip:888@conference.freeswitch.org </A><BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://iax:guest@conference.freeswitch.org/888"
target=_blank>iax:guest@conference.freeswitch.org/888</A><BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:googletalk:conf+888@conference.freeswitch.org"
target=_blank>googletalk:conf+888@conference.freeswitch.org</A><BR>pstn:213-799-1400</DIV>
<DIV><SPAN class=e id=q_1166d8f5bca18719_1>
<DIV
style="FONT-SIZE: 12pt; FONT-FAMILY: courier,monaco,monospace,sans-serif"><BR><BR>
<DIV
style="FONT-SIZE: 12pt; FONT-FAMILY: times new roman,new york,times,serif">-----
Original Message ----<BR>From: Freeswitch Freeswitch <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:freeswitch@2ride.com"
target=_blank>freeswitch@2ride.com</A>><BR>To: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:freeswitch-users@lists.freeswitch.org"
target=_blank>freeswitch-users@lists.freeswitch.org</A><BR>Sent: Friday,
November 23, 2007 10:47:53 AM<BR>Subject: Re: [Freeswitch-users] mem leak and
core dump<BR><BR>Caught signal 11 for unmapped thread!<BR><BR>
<DIV><SPAN class=gmail_quote>On 11/23/07, <B class=gmail_sendername>Freeswitch
Freeswitch</B> <<A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:freeswitch@2ride.com" target=_blank rel=nofollow>
freeswitch@2ride.com</A>> wrote:</SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">Hello,<BR><BR>Now
I have updated to the latest trunk as of Nov 21 and moved to a fedora 7
system. <BR><BR>Still chasing random crashes of FS. First time this
happened last night there were around 1000 sessions up.<BR>Crit error was
logged referring to a signal caught on an unmapped thread. I wasn't
able to catch the specific signal in the console before it scrolled
off. This morning I had another crash. This time around 200
sessions running. There was no logging reported during the second
crash. <BR><BR>Calling scenario is about as simple as it gets.<BR><BR>gw1
-> freeswitch -> gw2 all sip<BR>codecs, 729 and 711 <BR>transcoding
off<BR><BR>I have been playing with the various rtp knobs in the sofia
conf. Sometimes running with rewriting rtp timerstamps, sometimes
not. Also tried with soft timer and off. ( can someone point me to an
explanation of what the soft timer does? ). <BR><BR>Also notable that I
compiled the code with debug symbols.<BR><BR>No core has been generated
during crashes. Is there some way to enable core dumps?<BR><BR>Overall
I am very impressed with FS - would love to get the cause of these crashes
nailed so I can press forward. <BR><BR>Any help is very much
appreciated.<BR><BR>Best regards,<BR><BR>Tom<BR><BR><BR><BR><BR><BR>
<DIV>
<DIV><SPAN><SPAN class=gmail_quote>On 11/19/07, <B
class=gmail_sendername>Michael Jerris </B><<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:mike@jerris.com" target=_blank rel=nofollow>
mike@jerris.com</A>> wrote:</SPAN></SPAN></DIV>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV><SPAN><SPAN>On 11/19/07, <B class=gmail_sendername>Freeswitch
Freeswitch</B> <<A
onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:freeswitch@2ride.com" target=_blank
rel=nofollow>freeswitch@2ride.com</A>> wrote:</SPAN>
<DIV><SPAN><SPAN class=gmail_quote></SPAN>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">Hello
Freeswitch'rs,<BR><BR>I recently began using the latest freeswitch beta
code as a pure B2BUA for the purpose of topology hiding of IP - PSTN
calls. <BR></BLOCKQUOTE>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">running
env:<BR>Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686
Intel(R) Xeon(R)
CPU
5140 @ 2.33GHz GNU/Linux<BR>Freeswitch version: FreeSwitch Version
1.0.pre1 (6220M)<BR><BR>The modules I am using are 711 and 729 codecs,
sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.<BR><BR>I am
seeing what appears to be a memory leak. Consumed memory continues
to drop linearly based on number of calls. <BR>Also had one core
dump.</BLOCKQUOTE></SPAN>
<DIV><BR>Can you provide details of the call scenarios you are testing,
call rates, number of concurrent calls and any other
details?<BR> </DIV><SPAN><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">What
can I do to help troubleshoot these issues?<BR><BR>Today I plan to stop
using the CDR, rpc and logging modules to see if it makes a
difference.</BLOCKQUOTE></SPAN>
<DIV><BR>We have tested quite heavily with mod_console running, that
should not be an issue. <BR></DIV><SPAN><BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">Also
considering running on one of my fedora core 7 boxes instead of gentoo,
again to see if anything changes. </BLOCKQUOTE></SPAN>
<DIV><BR>We have had repeated issues with gentoo due to patches made
either to gcc or to libc or the kernel. It is the only os/distro
that I highly suggest never
using.<BR></DIV><BR>Mike<BR></DIV><BR><BR></SPAN></DIV><SPAN>_______________________________________________<BR>Freeswitch-users
mailing list<BR><A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:Freeswitch-users@lists.freeswitch.org" target=_blank
rel=nofollow>Freeswitch-users@lists.freeswitch.org </A><BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users"
target=_blank
rel=nofollow>http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
</A><BR>UNSUBSCRIBE: <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.freeswitch.org/mailman/options/freeswitch-users"
target=_blank
rel=nofollow>http://lists.freeswitch.org/mailman/options/freeswitch-users</A>
<BR><A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://www.freeswitch.org" target=_blank
rel=nofollow>http://www.freeswitch.org</A><BR><BR></SPAN></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></DIV><BR></DIV><BR></DIV></SPAN></DIV></DIV><SPAN
class=ad><BR>
<HR SIZE=1>
Get easy, one-click access to your favorites. <A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs"
target=_blank>Make Yahoo! your homepage.</A>
</SPAN></DIV><BR>_______________________________________________<BR>Freeswitch-users
mailing list<BR><A onclick="return top.js.OpenExtLink(window,event,this)"
href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org
</A><BR><A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users"
target=_blank>http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</A><BR>UNSUBSCRIBE:
<A onclick="return top.js.OpenExtLink(window,event,this)"
href="http://lists.freeswitch.org/mailman/options/freeswitch-users"
target=_blank>http://lists.freeswitch.org/mailman/options/freeswitch-users</A><BR><A
onclick="return top.js.OpenExtLink(window,event,this)"
href="http://www.freeswitch.org"
target=_blank>http://www.freeswitch.org</A><BR><BR></BLOCKQUOTE></DIV><BR></BODY></HTML>