Here's the stack from my latest crash this morning:<br><br>was running with 1400 sessions and the ulimit settings recommended by Anthony<br><br>This was after a svn update last night.<br><br>more ideas?<br><br>Thanks ....
<br><br>Core was generated by `./freeswitch'.<br>Program terminated with signal 11, Segmentation fault.<br>#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601
<br>601 sprintf(p, "%u%s", i[x], x == 3 ? "" : ".");<br>(gdb) where<br>#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601
<br>#1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788, payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at src/switch_rtCore was generated by `./freeswitch'.<br>Program terminated with signal 11, Segmentation fault.
<br>#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601<br>601 sprintf(p, "%u%s", i[x], x == 3 ? "" : ".");
<br>(gdb) where<br>#0 get_addr (buf=0x58a5dab0 "", len=<value optimized out>, in=<value optimized out>) at src/switch_utils.c:601<br>#1 0x00002aaaaab23a7b in rtp_common_read (rtp_session=0x289b788, payload_type=0x2aaab4833680 "", flags=0x2aaab483369c) at src/switch_rtp.c:991
<br>#2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame (rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230<br>#3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>, frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized out>,
<br> stream_id=<value optimized out>) at mod_sofia.c:492<br>#4 0x00002aaaaab1611b in switch_core_session_read_frame (session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at src/switch_core_io.c:113
<br>#5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:168<br>#6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128) at src/switch_ivr_bridge.c:237
<br>#7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128) at src/switch_core_state_machine.c:409<br>#8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value optimized out>, obj=0x0) at src/switch_core_session.c:681
<br>#9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0<br>#10 0x0000003baaed062d in clone () from /lib64/libc.so.6<br>p.c:991<br>#2 0x00002aaaaab24043 in switch_rtp_zerocopy_read_frame (rtp_session=0x289b788, frame=0x2aaab4833648) at src/switch_rtp.c:1230
<br>#3 0x00002aaaabafc317 in sofia_read_frame (session=<value optimized out>, frame=0x58a5de60, timeout=<value optimized out>, flags=<value optimized out>,<br> stream_id=<value optimized out>) at mod_sofia.c:492
<br>#4 0x00002aaaaab1611b in switch_core_session_read_frame (session=0x2aaab482d128, frame=0x58a5de60, timeout=-1, stream_id=0) at src/switch_core_io.c:113<br>#5 0x00002aaaaab251d1 in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:168
<br>#6 0x00002aaaaab257a6 in audio_bridge_on_loopback (session=0x2aaab482d128) at src/switch_ivr_bridge.c:237<br>#7 0x00002aaaaab14420 in switch_core_session_run (session=0x2aaab482d128) at src/switch_core_state_machine.c:409
<br>#8 0x00002aaaaab12485 in switch_core_session_thread (thread=<value optimized out>, obj=0x0) at src/switch_core_session.c:681<br>#9 0x00002aaaaae061c5 in start_thread () from /lib64/libpthread.so.0<br>#10 0x0000003baaed062d in clone () from /lib64/libc.so.6
<br><br><br><div><span class="gmail_quote">On 11/25/07, <b class="gmail_sendername">Abdul Hakeem</b> <<a href="mailto:alhakeem@gmail.com">alhakeem@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Bodoni MT">Hi,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Bodoni MT">Try reducing the stack size (-) to about 256K and also double the
file descriptors (-n) to 1024048.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Bodoni MT">Ensure the ulimit commands are in the /etc/profile and
bashrc.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Bodoni MT">Cheers,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" face="Bodoni MT">AH</font> </span></div><br>
<div dir="ltr" align="left" lang="en-us">
<hr>
<font face="Tahoma" size="2"><b>From:</b>
<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch-users-bounces@lists.freeswitch.org</a>
[mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch-users-bounces@lists.freeswitch.org</a>] <b>On Behalf Of
</b>Freeswitch Freeswitch<br><b>Sent:</b> 24 November 2007 16:48<span class="q"><br><b>To:</b>
<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch-users@lists.freeswitch.org</a><br></span></font><div><span class="e" id="q_116763311b72b9e9_2">
<font face="Tahoma" size="2"><b>Subject:</b> Re: [Freeswitch-users]
mem leak and core dump<br></font></span></div><br></div><div><span class="e" id="q_116763311b72b9e9_4">
<div></div>Here are my ulimit settings before I adjusted to meet your
recommendations Anthony.<br>I had increased file descriptors and decreased stack
size.<br><br>t: cpu time
(seconds) unlimited<br>-f: file
size (blocks) unlimited <br>-d:
data seg size (kbytes) unlimited<br>-s: stack size
(kbytes) 1024<br>-c: core file size
(blocks) 0<br>-m: resident set size (kbytes) unlimited<br>-u:
processes
16381<br>-n: file
descriptors 100000
<br>-l: locked-in-memory size (kb) 32<br>-v: address space
(kb) unlimited<br>-x: file
locks
unlimited<br>-i: pending
signals
16381<br>-q: bytes in POSIX msg queues 819200<br>-N
13:
0 <br>-N
14:
0<br><br>Here are the settings I am running with now.<br><br>-t: cpu time
(seconds) unlimited<br>-f: file
size (blocks) unlimited<br>-d:
data seg size (kbytes) unlimited <br>-s: stack size
(kbytes) unlimited<br>-c: core file
size (blocks) unlimited<br>-m: resident set size (kbytes)
unlimited<br>-u:
processes
16381<br>-n: file
descriptors
999999<br>-l: locked-in-memory size (kb) 32 <br>-v: address space
(kb) unlimited<br>-x: file
locks
unlimited<br>-i: pending
signals
16381<br>-q: bytes in POSIX msg queues 819200<br>-N
13:
0<br>-N
14:
0 <br><br><br>Here is the stack from my lastest crash.<br>I noticed some of the
libraries did not seem to be built with debug so I cleaned and rebuilt and
reinstall everything.<br>Currently running for 15 minutes, over 5000 total
sessions and 300 concurrent. <br>Finger crossed .......<br><br>GNU gdb Red Hat
Linux (6.6-8.fc7rh)<br>Copyright (C) 2006 Free Software Foundation, Inc.<br>GDB
is free software, covered by the GNU General Public License, and you
are<br>welcome to change it and/or distribute copies of it under certain
conditions. <br>Type "show copying" to see the conditions.<br>There is
absolutely no warranty for GDB. Type "show warranty" for details.<br>This
GDB was configured as "x86_64-redhat-linux-gnu"...<br>Using host libthread_db
library "/lib64/libthread_db.so.1". <br>Reading symbols from
/lib64/libm.so.6...done.<br>Loaded symbols for /lib64/libm.so.6<br>Reading
symbols from /usr/local/freeswitch1121/lib/libfreeswitch.so.1...done.<br>Loaded
symbols for /usr/local/freeswitch1121/lib/libfreeswitch.so.1 <br>Reading symbols
from /usr/lib64/libsqlite3.so.0...done.<br>Loaded symbols for
/usr/lib64/libsqlite3.so.0<br>Reading symbols from
/lib64/libuuid.so.1...done.<br>Loaded symbols for /lib64/libuuid.so.1<br>Reading
symbols from /lib64/librt.so.1...done. <br>Loaded symbols for
/lib64/librt.so.1<br>Reading symbols from /lib64/libdl.so.2...done.<br>Loaded
symbols for /lib64/libdl.so.2<br>Reading symbols from
/lib64/libcrypt.so.1...done.<br>Loaded symbols for /lib64/libcrypt.so.1
<br>Reading symbols from /lib64/libpthread.so.0...done.<br>Loaded symbols for
/lib64/libpthread.so.0<br>Reading symbols from
/lib64/libc.so.6...done.<br>Loaded symbols for /lib64/libc.so.6<br>Reading
symbols from /usr/lib64/libstdc++.so.6...done. <br>Loaded symbols for
/usr/lib64/libstdc++.so.6<br>Reading symbols from
/lib64/libgcc_s.so.1...done.<br>Loaded symbols for
/lib64/libgcc_s.so.1<br>Reading symbols from
/lib64/ld-linux-x86-64.so.2...done.<br>Loaded symbols for /lib64/ld-
linux-x86-64.so.2<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_console.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_console.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_syslog.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_syslog.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_xml_rpc.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_xml_rpc.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_sofia.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_sofia.so<br>Reading symbols from
/lib64/libssl.so.6...done.<br>Loaded symbols for /lib64/libssl.so.6<br>Reading
symbols from /lib64/libcrypto.so.6...done.<br>Loaded symbols for
/lib64/libcrypto.so.6 <br>Reading symbols from
/lib64/libz.so.1...done.<br>Loaded symbols for /lib64/libz.so.1<br>Reading
symbols from /usr/lib64/libgssapi_krb5.so.2...done.<br>Loaded symbols for
/usr/lib64/libgssapi_krb5.so.2<br>Reading symbols from
/usr/lib64/libkrb5.so.3...done. <br>Loaded symbols for
/usr/lib64/libkrb5.so.3<br>Reading symbols from
/lib64/libcom_err.so.2...done.<br>Loaded symbols for
/lib64/libcom_err.so.2<br>Reading symbols from
/usr/lib64/libk5crypto.so.3...done.<br>Loaded symbols for
/usr/lib64/libk5crypto.so.3 <br>Reading symbols from
/usr/lib64/libkrb5support.so.0...done.<br>Loaded symbols for
/usr/lib64/libkrb5support.so.0<br>Reading symbols from
/lib64/libkeyutils.so.1...done.<br>Loaded symbols for
/lib64/libkeyutils.so.1<br>Reading symbols from
/lib64/libresolv.so.2...done.<br>Loaded symbols for
/lib64/libresolv.so.2<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_commands.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_commands.so <br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_conference.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_conference.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_dptools.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_dptools.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library
symbols:<br>DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so] <br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_fifo.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_fifo.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_voicemail.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_voicemail.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_limit.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_limit.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_dialplan_xml.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_g711.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_g711.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_g729.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_g729.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_l16.so...done.<br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_l16.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_say_en.so...done. <br>Loaded symbols for
/usr/local/freeswitch1121/mod/mod_say_en.so<br>Reading symbols from
/usr/local/freeswitch1121/mod/mod_expr.so...Error while reading shared library
symbols:<br>DW_FORM_strp pointing outside of .debug_str section [in module
/usr/local/freeswitch1121/mod/mod_expr.so] <br>Core was generated by
`./freeswitch'.<br>Program terminated with signal 11, Segmentation
fault.<br>#0 nua_stack_timer (nua=<value optimized out>, t=<value
optimized out>, a=0x0) at
nua_stack.c:513<br>513 if
(du->du_refresh && du->du_refresh < next)<br>(gdb)
where<br>#0 nua_stack_timer (nua=<value optimized out>, t=<value
optimized out>, a=0x0) at nua_stack.c:513<br>#1 0x00002aaaabdb1ae8 in
su_timer_expire (timers=<value optimized out>, timeout=0x402b40d8,
now={tv_sec = 3404833764, tv_usec = 661017}) at su_timer.c:533 <br>#2
0x00002aaaabdb71a6 in su_base_port_run (self=0x6901f0) at
su_base_port.c:328<br>#3 0x00002aaaabda7e7b in su_pthread_port_clone_main
(varg=<value optimized out>) at su_pthread_port.c:321<br>#4
0x00002aaaaae051c5 in start_thread () from /lib64/libpthread.so.0 <br>#5
0x0000003baaed062d in clone () from /lib64/libc.so.6<br>(gdb)<br><br>After this
crash I did a clean rebuild and reinstall - since then I have had no subsequent
crashes.<br><br><br><br>
<div><span class="gmail_quote">On 11/23/07, <b class="gmail_sendername">Anthony
Minessale</b> <<a href="mailto:anthmct@yahoo.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">anthmct@yahoo.com</a>>
wrote:</span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div style="font-size: 12pt; font-family: courier,monaco,monospace,sans-serif;">please
disable the crash-protection option in switch.conf.xml<br>and execute the
command 'ulimit -c unlimited' from your unix shell before you start
freeswitch.. <br><br>here are some other ulimits you may want to turn up for
high-volume calls:<br><br>vi u<br>also you can use the unix command "script"
to log all the console output to a file so you can capture all of the output
for us to examine. <br><br><br><br>
<div> </div>
<div>Anthony Minessale II<br><br><span>FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.freeswitch.org/</a></span> <br><span>ClueCon <a href="http://www.cluecon.com/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://www.cluecon.com/</a></span><br><br>AIM: anthm<br><a href="mailto:MSN:anthony_minessale@hotmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/PAYPAL:
<a href="mailto:anthony.minessale@gmail.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">anthony.minessale@gmail.com</a><br>IRC: <a href="http://irc.freenode.net" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
irc.freenode.net</a>
#freeswitch</div>
<div><br>FreeSWITCH Developer Conference<br><a href="mailto:sip:888@conference.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">sip:888@conference.freeswitch.org </a><br><a href="http://iax:guest@conference.freeswitch.org/888" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
iax:guest@conference.freeswitch.org/888</a><br><a href="mailto:googletalk:conf+888@conference.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">googletalk:conf+888@conference.freeswitch.org
</a><br>pstn:213-799-1400</div>
<div><span>
<div style="font-size: 12pt; font-family: courier,monaco,monospace,sans-serif;"><br><br>
<div style="font-size: 12pt; font-family: times new roman,new york,times,serif;">-----
Original Message ----<br>From: Freeswitch Freeswitch <<a href="mailto:freeswitch@2ride.com" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch@2ride.com</a>><br>To: <a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
freeswitch-users@lists.freeswitch.org</a><br>Sent: Friday,
November 23, 2007 10:47:53 AM<br>Subject: Re: [Freeswitch-users] mem leak and
core dump<br><br>Caught signal 11 for unmapped thread!<br><br>
<div><span class="gmail_quote">On 11/23/07, <b class="gmail_sendername">Freeswitch
Freeswitch</b> <<a href="mailto:freeswitch@2ride.com" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
freeswitch@2ride.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br><br>Now
I have updated to the latest trunk as of Nov 21 and moved to a fedora 7
system. <br><br>Still chasing random crashes of FS. First time this
happened last night there were around 1000 sessions up.<br>Crit error was
logged referring to a signal caught on an unmapped thread. I wasn't
able to catch the specific signal in the console before it scrolled
off. This morning I had another crash. This time around 200
sessions running. There was no logging reported during the second
crash. <br><br>Calling scenario is about as simple as it gets.<br><br>gw1
-> freeswitch -> gw2 all sip<br>codecs, 729 and 711 <br>transcoding
off<br><br>I have been playing with the various rtp knobs in the sofia
conf. Sometimes running with rewriting rtp timerstamps, sometimes
not. Also tried with soft timer and off. ( can someone point me to an
explanation of what the soft timer does? ). <br><br>Also notable that I
compiled the code with debug symbols.<br><br>No core has been generated
during crashes. Is there some way to enable core dumps?<br><br>Overall
I am very impressed with FS - would love to get the cause of these crashes
nailed so I can press forward. <br><br>Any help is very much
appreciated.<br><br>Best regards,<br><br>Tom<br><br><br><br><br><br>
<div>
<div><span><span class="gmail_quote">On 11/19/07, <b class="gmail_sendername">Michael Jerris </b><<a href="mailto:mike@jerris.com" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
mike@jerris.com</a>> wrote:</span></span></div>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><span><span>On 11/19/07, <b class="gmail_sendername">Freeswitch
Freeswitch</b> <<a href="mailto:freeswitch@2ride.com" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">freeswitch@2ride.com</a>> wrote:</span>
<div><span><span class="gmail_quote"></span>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello
Freeswitch'rs,<br><br>I recently began using the latest freeswitch beta
code as a pure B2BUA for the purpose of topology hiding of IP - PSTN
calls. <br></blockquote>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">running
env:<br>Linux b2bua 2.6.18 #6 SMP Thu Nov 16 10:35:45 EST 2006 i686
Intel(R) Xeon(R)
CPU
5140 @ 2.33GHz GNU/Linux<br>Freeswitch version: FreeSwitch Version
1.0.pre1 (6220M)<br><br>The modules I am using are 711 and 729 codecs,
sip, directory, xml_rpc, cdr, xml_cdr, console and syslog.<br><br>I am
seeing what appears to be a memory leak. Consumed memory continues
to drop linearly based on number of calls. <br>Also had one core
dump.</blockquote></span>
<div><br>Can you provide details of the call scenarios you are testing,
call rates, number of concurrent calls and any other
details?<br> </div><span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">What
can I do to help troubleshoot these issues?<br><br>Today I plan to stop
using the CDR, rpc and logging modules to see if it makes a
difference.</blockquote></span>
<div><br>We have tested quite heavily with mod_console running, that
should not be an issue. <br></div><span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Also
considering running on one of my fedora core 7 boxes instead of gentoo,
again to see if anything changes. </blockquote></span>
<div><br>We have had repeated issues with gentoo due to patches made
either to gcc or to libc or the kernel. It is the only os/distro
that I highly suggest never
using.<br></div><br>Mike<br></div><br><br></span></div><span>_______________________________________________<br>Freeswitch-users
mailing list<br><a href="mailto:Freeswitch-users@lists.freeswitch.org" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">Freeswitch-users@lists.freeswitch.org </a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
</a><br>UNSUBSCRIBE: <a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.freeswitch.org/mailman/options/freeswitch-users
</a>
<br><a href="http://www.freeswitch.org" rel="nofollow" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://www.freeswitch.org</a><br><br></span></blockquote></div><br></blockquote></div><br>
</div><br></div></span></div></div><span><br>
<hr size="1">
Get easy, one-click access to your favorites. <a href="http://us.rd.yahoo.com/evt=51443/*http://www.yahoo.com/r/hs" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">Make Yahoo! your homepage.</a>
</span></div><br>_______________________________________________<br>Freeswitch-users
mailing list<br><a href="mailto:Freeswitch-users@lists.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">Freeswitch-users@lists.freeswitch.org
</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:
<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">
http://www.freeswitch.org</a><br><br></blockquote></div><br></span></div></div>
<br>_______________________________________________<br>Freeswitch-users mailing list<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org
</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>UNSUBSCRIBE:
<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.freeswitch.org" target="_blank">
http://www.freeswitch.org</a><br><br></blockquote></div><br>