<div dir="ltr">last_used is seconds typically unattached handles unused for more than 30 sec are released.<div><br><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jul 23, 2013 at 3:23 PM, Victor Chukalovskiy <span dir="ltr"><<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Yes, these are the only ones I see
printed on our sandbox. Will try if I can get any other ones. I
understand it's time consuming for you to respond, thanks again
for your help.<br>
<br>
Today it got stuck again on production server and I ran db_cache
status:<div class="im"><br>
<br>
db="sofia_reg_<profile_name>"<br>
Type: CORE_DB<br></div>
Last used: 150861<br>
Flags: Locked, Attached(1)<br>
Creator: sofia_glue.c:6380<br>
Last User: src/switch_core_sqldb.c:1816<br>
<br>
.....<br>
<same as above for each profile, we got a few dozens><br>
.....<br>
db="/dev/shm/core.db"<br>
Type: CORE_DB<br>
Last used: 150862<br>
Flags: Locked, Attached(1)<br>
Creator: src/switch_core_sqldb.c:1816<br>
Last User: <br>
<br>
In what units is "last used"? <br>
<br>
VM runs on iSCSI SAN Storage.<div><div class="h5"><br>
<br>
<br>
On 13-07-23 03:01 PM, Anthony Minessale wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">Is that the only ones? how frequent are they?
Those are only for expiration and run very infrequently. You
may wish to study it longer because that should not be enough to
slow everything down. Your server is not on a fake HDD too is
it?
<div>
<br>
</div>
<div>I generally do not help on load testing or virtual server
setups because it can consume a lot of time.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Jul 23, 2013 at 12:05 PM,
Victor Chukalovskiy <span dir="ltr"><<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Anthony, thanks a lot. Here is what I got:<br>
<br>
[ERR] sofia_glue.c:6614 SQL: [select
call_id,sip_user,sip_host,contact,status,rpid,expires,user_agent,server_user,server_host,profile_name,network_ip,0
from sip_registrations where expires > 0 and expires
<= 1374597711]<br>
[ERR] sofia_glue.c:6614 SQL: [select call_id from
sip_shared_appearance_dialogs where hostname='test' and
profile_name='test' and expires <= 1374597711]<br>
<br>
Is there a way to disable those two selects to save a
few DB calls?<br>
Keep in mind that manage-shared-appearance and
manage-presence are already off.<br>
<br>
As for the performance tweaking, I'm going to:<br>
-move ALL sqlite DBs to RAM<br>
-bump max handles. Does it sound reasonable and safe to
go from 50 to 100 handles?<br>
<br>
@Cal, saw your msg, thanks. I was indeed looking at
master while Anthony meant stable.<span><font color="#888888"><br>
<br>
.Victor</font></span>
<div><br>
<br>
On 13-07-22 04:42 PM, Anthony Minessale wrote:<br>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">do you have manage-shared-appearance
off too cos that re-enables presence?
<div><br>
</div>
<div>edit sofia_glue.c on line 6614 and add a
print to see what the sql is that is being
executed.</div>
<div><br>
</div>
<div>switch_log_printf(SWITCH_CHANNEL_LOG,
SWITCH_LOG_ERROR, "SQL: [%s]\n", sql);<br>
</div>
<div><br>
</div>
<div>then make mod_sofia-install</div>
<div><br>
</div>
<div>and relaunch and try to catch a sample of
what it's doing.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Jul 22, 2013 at
3:27 PM, Victor Chukalovskiy <span dir="ltr"><<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Anthony, <br>
<br>
Thank you, this is a very useful command.
I see a bunch of records like:<br>
<br>
db="sofia_reg_<profile_name>"<br>
Type: CORE_DB<br>
Last used: 10<br>
Flags: Unlocked, Detached(0)<br>
Creator: <b>sofia_glue.c:6605</b><br>
Last User: <b>sofia_glue.c:6605</b><br>
<br>
Appears that these DBs are in use and
user/creator is sofia_glue.c:6605. Same as
in error log:
<div><br>
[WARNING] <b>sofia_glue.c:6605</b> Max
handles 50<br>
<br>
</div>
And this is not a cored.db... it's sitting
on the disk drive still. Is there a way to
setup sip_profiles not to create any
sofia_reg_...DBs?<br>
<br>
Re: vmware setup. It's good to know it's
not optimal, however it gives rack space,
administrator's time, and $ savings
compared to dedicated servers.<br>
<br>
Should it be Ok to stick with VM setup
provided enough CPU and RAM is allocated
to each FS guest? Are there reasons not to
run FS on VM other than CPU and RAM
allocation?<br>
<br>
Thank you,<br>
Victor<br>
<br>
<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">Its a large number of
handles to have open all in sofia if
you are not doing anything
<div><br>
</div>
<div>db_cache status</div>
<div><br>
</div>
<div>from cli can show where those
handles come from and what db they
are using.</div>
<div>Clearly the box is being
overloaded past its limits.</div>
<div><br>
</div>
<div>This is why we don't endorse
virtual servers.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Jul
22, 2013 at 2:49 PM, Victor
Chukalovskiy <span dir="ltr"><<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Disabled in all profiles:<br>
<param
name="manage-presence"
value="false"/>
<div>
<div><br>
<br>
On 13-07-22 03:40 PM,
Anthony Minessale wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">Did you
turn of the presence
in the sofia profile,
that is enabled by
default and very
costly. comment out
manage-presence</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Mon, Jul 22, 2013 at
2:32 PM, Victor
Chukalovskiy <span dir="ltr"><<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Michael,
Ken, <br>
<br>
Thank you for
the response.
Firstly, let
me clarify
setup for each
of DBs in use:<br>
<br>
<b>core_db:</b>
sqlite,
/dev/shm/core.db
(RAM) - using
default 50 max
handles<br>
<b>mod_sofia
db</b><b>:</b>
sqlite,
/opt/fs/db
(HDD) <br>
<b>mod_lcr db:</b>
Postgres<br>
<b>mod_limit:</b>
hash backend<br>
<br>
Note: I
believe we
rarely use
mod_sofia
DB's...this is
because we
don't use any
registrations,
acl only. When
I list sofia
per-profile db
files on
production
box, most of
them has not
been edited
for a couple
days. Are my
conclusions
correct here,
or do I miss
something?
Maybe they are
accessed a lot
but only for
reading?<br>
<br>
I understand
recommendations
from Ken and
from Michaels
are along the
lines of
increasing
core db and
sofia db
throughput.
However here
is what
puzzles me:<br>
-the db used
most
extensively is
core_db. It'
already in the
ram.<br>
-we don't have
much load on
sofia dbs (or
am I mistaken
in my logic?)<br>
-error log
started with:
<div><br>
<br>
[CRIT]
switch_time.c:967
Virtual
Migration
Detected!
Syncing Clock<br>
</div>
<div>
[WARNING]
switch_scheduler.c:115
Task was
executed late
by 19 seconds
1 heartbeat
(core)<br>
<br>
</div>
Does it mean
that there
were abnormal
events prior
hitting 50 max
handles limit?<br>
<br>
Thank you,<br>
-Victor
<div>
<div><br>
<br>
On 13-07-22
02:52 PM,
Anthony
Minessale
wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">Check
the
odbcinst.ini
settings from
the wiki or
use native pg
mode.
<div>increase
the max number
of pg
connections on
your server
and turn up FS
to match
in max-db-handles
in
switch.conf.xml</div>
<div><br>
</div>
<div>You are
already
beating it up
pretty good
esp for a fake
server though.
2500 is a
lot.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Mon, Jul 22,
2013 at 1:44
PM, Ken Rice <span dir="ltr"><<a href="mailto:krice@freeswitch.org" target="_blank">krice@freeswitch.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Mount
the entire
freeswitch/db
dir as
tmpfs... most
of these
errors are for<br>
the sofia DB
getting backed
up...<br>
<br>
The coredb is
a completely
different
sqlite db from
the sofia db<br>
<br>
On 7/22/13
12:54 PM,
"Victor
Chukalovskiy"
<<a href="mailto:victor.chukalovskiy@gmail.com" target="_blank">victor.chukalovskiy@gmail.com</a>><br>
wrote:<br>
<div>
<div><br>
> Hello,<br>
><br>
> Need some
help finding
the bottleneck
here. Running
1.2.stable on
a<br>
> decent
multi-core
vmware
platform. No
RTP, SIP
signaling
only. Around<br>
> 2500
sessions and
30 average CPS
load.
Essentially,
it's a
multi-profile<br>
> .xml
config that
uses mod_lcr
extensively
(postgres DB).
Yesterday it<br>
> stopped
processing
calls with the
following log
output:<br>
><br>
>
2013-07-21
06:00:37.642678
[CRIT]
switch_time.c:967
Virtual
Migration<br>
> Detected!
Syncing Clock<br>
>
2013-07-21
06:01:07.141734
[WARNING]
switch_scheduler.c:115
Task was<br>
> executed
late by 19
seconds 1
heartbeat
(core)<br>
>
2013-07-21
06:02:32.402668
[CRIT]
switch_time.c:967
Virtual
Migration<br>
> Detected!
Syncing Clock<br>
>
2013-07-21
06:04:42.499744
[WARNING]
switch_scheduler.c:115
Task was<br>
> executed
late by 115
seconds 1
heartbeat
(core)<br>
>
2013-07-21
06:04:42.499744
[WARNING]
switch_scheduler.c:115
Task was<br>
> executed
late by 74
seconds 2
check_ip
(core)<br>
>
2013-07-21
06:05:15.102683
[CRIT]
switch_time.c:967
Virtual
Migration<br>
> Detected!
Syncing Clock<br>
>
2013-07-21
06:05:44.961004
[WARNING]
switch_scheduler.c:115
Task was<br>
> executed
late by 22
seconds 1
heartbeat
(core)<br>
>
2013-07-21
06:05:44.961004
[WARNING]
switch_scheduler.c:115
Task was<br>
> executed
late by 2
seconds 2
check_ip
(core)<br>
>
2013-07-21
06:05:49.322672
[WARNING]
mod_lcr.c:448
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:50.962987
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:50.962987
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:50.962987
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
................................<br>
>
2013-07-21
06:05:53.962672
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:54.422671
[ERR]
mod_lcr.c:448
Error
connecting<br>
>
2013-07-21
06:05:54.422671
[WARNING]
mod_lcr.c:1779
LCR lookup
failed<br>
> for
14389955082<br>
>
2013-07-21
06:05:54.962671
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:54.962671
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
................................<br>
>
2013-07-21
06:05:55.962758
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:56.062671
[ERR]
sofia_glue.c:6605
Error
connecting<br>
>
2013-07-21
06:05:56.062671
[ERR]
sofia_glue.c:6606
Error Opening
DB<br>
>
2013-07-21
06:05:56.062671
[WARNING]
sofia_glue.c:6605
Max handles 50<br>
> exceeded,
blocking....<br>
>
2013-07-21
06:05:56.062671
[ERR]
sofia_glue.c:6605
Error
connecting<br>
>
2013-07-21
06:05:56.062671
[ERR]
sofia_glue.c:6606
Error Opening
DB<br>
><br>
>
FreeSWITCH
stayed
unusable since
then until it
was restarted.<br>
><br>
> FYI core
DB is already
in the RAM:<br>
> <param
name="core-db-name"
value="/dev/shm/core.db"
/><br>
><br>
><br>
> Thank
you,<br>
> Victor<br>
><br>
>
_________________________________________________________________________<br>
>
Professional
FreeSWITCH
Consulting
Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
>
FreeSWITCH-powered
IP PBX: The
CudaTel
Communication
Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official
FreeSWITCH
Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
>
FreeSWITCH-users
mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
</div>
</div>
<span><font color="#888888">--<br>
Ken<br>
<a href="http://www.FreeSWITCH.org" target="_blank">http://www.FreeSWITCH.org</a><br>
<a href="http://www.ClueCon.com" target="_blank">http://www.ClueCon.com</a><br>
<a href="http://www.OSTAG.org" target="_blank">http://www.OSTAG.org</a><br>
<a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
</font></span>
<div>
<div><br>
<br>
<br>
_________________________________________________________________________<br>
Professional
FreeSWITCH
Consulting
Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered
IP PBX: The
CudaTel
Communication
Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official
FreeSWITCH
Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users
mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Anthony
Minessale II<br>
<br>
FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br>
<br>
AIM: anthm<br>
<a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
<br>
FreeSWITCH
Developer
Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_________________________________________________________________________<br>
Professional
FreeSWITCH
Consulting
Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered
IP PBX: The
CudaTel
Communication
Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official
FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users
mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Anthony Minessale II<br>
<br>
FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br>
<br>
AIM: anthm<br>
<a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
<br>
FreeSWITCH Developer
Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900 </div>
<br>
<fieldset></fieldset>
<br>
<pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH
Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The
CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Anthony Minessale II<br>
<br>
FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br>
<br>
AIM: anthm<br>
<a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
<br>
FreeSWITCH Developer Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900 </div>
<br>
<fieldset></fieldset>
<br>
<pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel
Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Anthony Minessale II<br>
<br>
FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br>
<br>
AIM: anthm<br>
<a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
<br>
FreeSWITCH Developer Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900 </div>
<br>
<fieldset></fieldset>
<br>
<pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Anthony Minessale II<br>
<br>
FreeSWITCH <a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org/</a><br>
ClueCon <a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire" target="_blank">http://twitter.com/FreeSWITCH_wire</a><br>
<br>
AIM: anthm<br>
<a href="mailto:MSN%3Aanthony_minessale@hotmail.com" target="_blank">MSN:anthony_minessale@hotmail.com</a><br>
GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com" target="_blank">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a>
#freeswitch<br>
<br>
FreeSWITCH Developer Conference<br>
<a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org" target="_blank">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire">http://twitter.com/FreeSWITCH_wire</a><br><br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900
</div>