[Freeswitch-users] Strange Performance when using as SBC

peder at networkoblivion.com peder at networkoblivion.com
Mon Feb 2 12:01:36 PST 2009


What is wrong with Intrepid?

Anthony Minessale wrote:
> if you want to use ubuntu, be sure to use hardy and not intrepid.
> 
> On Mon, Feb 2, 2009 at 7:03 AM, Saeed Ahmed <saeedahmad1981 at gmail.com 
> <mailto:saeedahmad1981 at gmail.com>> wrote:
> 
>     Thanks rod for a quick answer,
> 
>     FS is installed on Ubuntu Server.
> 
>     I am planning to replace Nextone SBC with FS, Later I'll also use
>     openZAP to
>     communicate with TDM but this all depends how much calls it can take, or
>     maybe we can also do something in clustering environment ( I am not sure
>     about it). But thanks again and any further help will be highly
>     appreciated.
> 
> 
>     Kind Regards
>     Saeed Ahmed Tariq
> 
> 
> 
>     -----Original Message-----
>     From: freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>     [mailto:freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>] On Behalf Of rod
>     Sent: Monday, February 02, 2009 1:53 PM
>     To: freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>
>     Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
> 
>     Hi Saeed,
> 
>     I just created an account to share my setup on the wiki. I will detail
>     all the steps for a clean install of a debian64 lenny with FS used as a
>     SBC (next step is to try the new LCR module :) )and what I'm doing do
>     stress the server.
> 
>     I wrote nothing at this time so please be patient, I'm waiting for my
>     new hardware so that I will detail as much as possible what I'll do.
> 
>     For beginning I suggest you reading the start page on the wiki,
>     especially these pages:
>        -http://wiki.freeswitch.org/wiki/Getting_Started_Guide
>        -http://wiki.freeswitch.org/wiki/Dialplan_XML
> 
>     maybe you could tell more about the linux distribution you're using so
>     that I can give you some pointers for sipp...
> 
>     regards.
>     rod.
> 
> 
>     Saeed Ahmed wrote:
>      > Hi Rod,
>      >
>      > Could you please share how you configured Sipp & FS to create a test
>      > environment? Especially the dial plan, sofia settings etc...,
>     actually I
>     am
>      > a newbie. I want to test it on a single FS machine.
>      >
>      > Kind Regards
>      > Saeed
>      > -----Original Message-----
>      > From: freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>
>      > [mailto:freeswitch-users-bounces at lists.freeswitch.org
>     <mailto:freeswitch-users-bounces at lists.freeswitch.org>] On Behalf Of rod
>      > Sent: Monday, February 02, 2009 11:00 AM
>      > To: freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>
>      > Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>      >
>      > Hi Ken, Jay,
>      >
>      > thanks for pointing to proxy media, I will test.
>      >
>      > Ken, you are right, I was brain damaged (a stupid mistake) when
>     setting
>      > INFO cause this kind of level could be very verbose. I'm switching to
>      > CRIT or ERR.
>      >
>      > Thanks guys,
>      > rod.
>      >
>      > thanks for
>      >
>      > Ken Rice wrote:
>      >
>      >> If you don't have to transcode, using proxy media mode will
>     still save
>     you
>      >> some CPU time. This is 1/2 way between bypass media and the
>     default media
>      >> interactive mode. The other draw back to this mode is if you are
>     using FS
>      >>
>      > to
>      >
>      >> clean up RTP and DTMF you loose those functions but they are not
>     needed
>     in
>      >> most use cases.
>      >>
>      >> As far as the log level goes, I found that once I had things stable
>      >>
>      > setting
>      >
>      >> the loglevel to helped a good deal... Info is probably a bit too
>     high of
>     a
>      >> loglevel I would probably go for CRIT or ERR (2 or 1
>     respectively) if you
>      >> insist on leaving logging turned on... On a busy system these
>     can and
>     will
>      >> generate a good deal of activity (and disk IO if using mod_logfile)
>      >>
>      >> Ken
>      >>
>      >>
>      >>
>      >>
>      >>> From: rod <kawarod at laposte.net <mailto:kawarod at laposte.net>>
>      >>> Reply-To: <freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>>
>      >>> Date: Mon, 02 Feb 2009 11:36:35 +0400
>      >>> To: <freeswitch-users at lists.freeswitch.org
>     <mailto:freeswitch-users at lists.freeswitch.org>>
>      >>> Subject: Re: [Freeswitch-users] Strange Performance when using
>     as SBC
>      >>>
>      >>> Hi Ken,
>      >>>
>      >>> 1) I'd like to use FS to hide topology, so bypass media is not
>     possible
>      >>> 2) done
>      >>> 3) done
>      >>> 4) not used
>      >>> 5) i'm using this ins switch.xml -> <param name="loglevel"
>      >>> value="info"/>, if you think an other log level is more suitable.
>      >>>
>      >>> Regarding logging, I can see in console and in the
>     freeswitch.log that
>      >>> there is still a lot of NOTICE logging, see below:
>      >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>      >>> switch_core_session_thread() Session 8721
>      >>> (sofia/internal/sipp at 10.10.10.1:5060
>     <http://sipp@10.10.10.1:5060>) Ended
>      >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>      >>> switch_core_session_thread() Close Channel
>      >>> sofia/internal/sipp at 10.10.10.1:5060
>     <http://sipp@10.10.10.1:5060> [CS_HANGUP]
>      >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>      >>> switch_core_session_thread() Session 8722
>      >>> (sofia/external/9998 at 10.10.20.100 <mailto:9998 at 10.10.20.100>) Ended
>      >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>      >>> switch_core_session_thread() Close Channel
>      >>> sofia/external/9998 at 10.10.20.100 <mailto:9998 at 10.10.20.100>
>     [CS_HANGUP]
>      >>> 2009-02-02 08:33:56 [NOTICE] sofia.c:3164
>     sofia_handle_sip_i_state()
>      >>> Channel [sofia/external/9998 at 10.10.20.100
>     <mailto:9998 at 10.10.20.100>] has been answered
>      >>> 2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
>      >>> Changing codec ptime to 30. I bet you have a linksys/sipura =D
>      >>>
>      >>> Do you have any idea where I can switch off this kind of logging. I
>      >>> thought it should be in /dialplan/internal.xml, but I see that in
>      >>> internal.xml -> <param name="debug" value="0"/>
>      >>>
>      >>> thanks a lot for your suggestion.
>      >>>
>      >>> regards,
>      >>> rod
>      >>>
>      >>> Ken Rice wrote:
>      >>>
>      >>>
>      >>>> Dont forget there are several things you can do to increase
>      >>>>
>      > performance...
>      >
>      >>>> 1) where possible use bypass media or media proxy modes
>      >>>> 2) mount freeswitch/db as a ram drive (if you are using
>     voicemail with
>      >>>> the internal FS DBs you'll need a way to make this persistant
>     across
>      >>>> reboots)
>      >>>> 3) see the wiki for setting reasonable ulimits
>      >>>> 4) (this is my oppinion others may vary) dont use mod_cdr_csv
>      >>>> 5) turn off (or reduce logging) in switch.conf.xml
>      >>>>
>      >>>> all of these thing can greatly improve performance.
>      >>>>
>      >>>> On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod at laposte.net
>     <mailto:kawarod at laposte.net>
>      >>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>> wrote:
>      >>>>
>      >>>>     Thanks Anthony,
>      >>>>
>      >>>>     the setup is like this:
>      >>>>
>      >>>>     sipp server ---- FS 1 ---- FS2
>      >>>>
>      >>>>     FS1 is the AMD CPU that has only one extension in dialplan
>     that
>      >>>>     bridges
>      >>>>     9999 to FS2. 9999 is the first extension in FS2 dialplan that
>      >>>>     plays moh,
>      >>>>     FS2 has no CPU pbm.
>      >>>>
>      >>>>     FS1 is maxing out at 60 bridged calls without your option -hp.
>      >>>>
>      >>>>     Using -hp, I'm now able to bridge 200 concurrent calls (a
>     great
>      >>>>     improvement) and the system is still reactive. CPU load is
>     high
>      >>>>     but not
>      >>>>     100% and as the system responds well, I think that doesn't
>     matter.
>      >>>>
>      > The
>      >
>      >>>>     2GB of memory are completely consumed (top command shows
>     700MB for
>      >>>>
>      > FS
>      >
>      >>>>     process).
>      >>>>
>      >>>>     I understand that FS1 server is not the best hardware
>     platform,
>      >>>>     and I'm
>      >>>>     waiting for new 4 cores server for testing.
>      >>>>     I will update those numbers when testing with the new
>     hardware.
>      >>>>
>      >>>>     regards,
>      >>>>     rod.
>      >>>>
>      >>>>     Anthony Minessale wrote:
>      >>>>
>      >>>>
>      >>>>> Which of the 2 machines has the load issue? You said it was
>     one box
>      >>>>> calling the other.
>      >>>>>
>      >>>>> You have 2 major things against you, single CPU and AMD, but you
>      >>>>> should at least be able to get in the vicinity of 800-1000 calls
>      >>>>>
>      >>>>>
>      >>>>     on a
>      >>>>
>      >>>>
>      >>>>> box like that.
>      >>>>>
>      >>>>> Are you calling the default 9999?  It's not really an appropriate
>      >>>>> extension for load testing.
>      >>>>> On the terminating box you should set up a manual extension
>     that is
>      >>>>> the first one in the dial plan
>      >>>>> to play a wav file from preferably a ram disk or /tmp
>      >>>>>
>      >>>>> If you do plan on using this in production accept nothing less
>      >>>>>
>      >>>>>
>      >>>>     than a
>      >>>>
>      >>>>
>      >>>>> multi-core intel machine with at least 4 cores, the more
>     cores the
>      >>>>> better because that parallel processing is where FS gets it's
>      >>>>>
>      >>>>>
>      >>>>     atvantage.
>      >>>>
>      >>>>
>      >>>>> On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod at laposte.net
>     <mailto:kawarod at laposte.net>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>
>      >>>>
>      >>>>
>      >>>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>
>     <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>>> wrote:
>      >>>>>
>      >>>>>     Dear list,
>      >>>>>
>      >>>>>     I've been playing with freeswitch for some time (2 months)
>      >>>>>
>      >>>>>
>      >>>>     and the
>      >>>>
>      >>>>
>      >>>>>     fact
>      >>>>>     is that I'm very pleased with the functionnalities of this
>      >>>>>
>      >>>>>
>      >>>>     software.
>      >>>>
>      >>>>
>      >>>>>     I'd like to use FS as a SBC handling media and I'm doing some
>      >>>>>     tests with
>      >>>>>     sipp to load the machine but I'm unable to bridge more than
>      >>>>>
>      >>>>>
>      >>>>     60 calls
>      >>>>
>      >>>>
>      >>>>>     without seeing the CPU being loaded at 100%. I'm sure
>      >>>>>
>      >>>>>
>      >>>>     something is
>      >>>>
>      >>>>
>      >>>>>     going
>      >>>>>     wrong with my setup but I'm unable to see what.
>      >>>>>
>      >>>>>     The test machine has the following specs:
>      >>>>>       Athlon XP 3500+ with 2GB of memory (I know this is not a
>      >>>>>
>      >>>>>
>      >>>>     high end
>      >>>>
>      >>>>
>      >>>>>     machine :p)
>      >>>>>
>      >>>>>     Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
>      >>>>>     processor       : 0
>      >>>>>     vendor_id       : AuthenticAMD
>      >>>>>     cpu family      : 15
>      >>>>>     model           : 95
>      >>>>>     model name      : AMD Athlon(tm) 64 Processor 3500+
>      >>>>>     stepping        : 2
>      >>>>>     cpu MHz         : 2199.973
>      >>>>>     cache size      : 512 KB
>      >>>>>     fpu             : yes
>      >>>>>     fpu_exception   : yes
>      >>>>>     cpuid level     : 1
>      >>>>>     wp              : yes
>      >>>>>     flags           : fpu vme de pse tsc msr pae mce cx8 apic
>      >>>>>
>      >>>>>
>      >>>>     sep mtrr pge
>      >>>>
>      >>>>
>      >>>>>     mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx
>     mmxext
>      >>>>>     fxsr_opt
>      >>>>>     rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm
>      >>>>>
>      >>>>>
>      >>>>     extapic
>      >>>>
>      >>>>
>      >>>>>     cr8_legacy
>      >>>>>     bogomips        : 4402.97
>      >>>>>     TLB size        : 1024 4K pages
>      >>>>>     clflush size    : 64
>      >>>>>     cache_alignment : 64
>      >>>>>     address sizes   : 40 bits physical, 48 bits virtual
>      >>>>>     power management: ts fid vid ttp tm stc
>      >>>>>
>      >>>>>     I installed FS on a fresh debian 64:
>      >>>>>       Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00
>      >>>>>
>      >>>>>
>      >>>>     UTC 2009
>      >>>>
>      >>>>
>      >>>>>     x86_64 GNU/Linux
>      >>>>>
>      >>>>>     I set the ulimit parameters like those on the website:
>      >>>>>       freeswitch at internal> ...
>      >>>>>     Freeswitch:/opt/free-svn/bin# ulimit -a
>      >>>>>     core file size          (blocks, -c) unlimited
>      >>>>>     data seg size           (kbytes, -d) unlimited
>      >>>>>     scheduling priority             (-e) 0
>      >>>>>     file size               (blocks, -f) unlimited
>      >>>>>     pending signals                 (-i) unlimited
>      >>>>>     max locked memory       (kbytes, -l) unlimited
>      >>>>>     max memory size         (kbytes, -m) unlimited
>      >>>>>     open files                      (-n) 999999
>      >>>>>     pipe size            (512 bytes, -p) 8
>      >>>>>     POSIX message queues     (bytes, -q) unlimited
>      >>>>>     real-time priority              (-r) 0
>      >>>>>     stack size              (kbytes, -s) 244
>      >>>>>     cpu time               (seconds, -t) unlimited
>      >>>>>     max user processes              (-u) unlimited
>      >>>>>     virtual memory          (kbytes, -v) unlimited
>      >>>>>     file locks                      (-x) unlimited
>      >>>>>
>      >>>>>
>      >>>>>     My network setup is the following:
>      >>>>>
>      >>>>>       SIPP machine (10.10.10.1/24)----------------vlan
>     <http://10.10.10.1/24%29----------------vlan>
>      >>>>>
>      >>>>>
>      >>>>     <http://10.10.10.1/24%29----------------vlan>
>      >>>>
>      >>>>
>      >>>>>     <http://10.10.10.1/24%29----------------vlan> 55
>      >>>>>     ----------(10.10.10.254/24 <http://10.10.10.254/24>
>     <http://10.10.10.254/24>
>      >>>>>
>      >>>>>
>      >>>>     <http://10.10.10.254/24>) FS
>      >>>>
>      >>>>
>      >>>>>     (10.10.20.254/24)--------------
>     <http://10.10.20.254/24%29-------------->
>      >>>>>
>      >>>>>
>      >>>>     <http://10.10.20.254/24%29-------------->
>      >>>>
>      >>>>
>      >>>>>     <http://10.10.20.254/24%29--------------> vlan56
>      >>>>>     -------------------(10.10.20.100/24
>     <http://10.10.20.100/24> <http://10.10.20.100/24>
>      >>>>>
>      >>>>>
>      >>>>     <http://10.10.20.100/24>)
>      >>>>
>      >>>>
>      >>>>>     OTHER STOCK FS
>      >>>>>
>      >>>>>
>      >>>>>     I launched sipp with:
>      >>>>>       sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi
>      >>>>>
>      >>>>>
>      >>>>     10.10.10.1 -i
>      >>>>
>      >>>>
>      >>>>>     10.10.10.1 -mp 25000 10.10.10.254:5060
>     <http://10.10.10.254:5060>
>      >>>>>
>      >>>>>
>      >>>>     <http://10.10.10.254:5060> <http://10.10.10.254:5060>
>      >>>>
>      >>>>
>      >>>>>     The dialplan on FS is very simple:
>      >>>>>     <?xml version="1.0" encoding="utf-8"?>
>      >>>>>     <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
>      >>>>>     <include>
>      >>>>>      <context name="default">
>      >>>>>
>      >>>>>        <extension name="hold_music_bridge">
>      >>>>>         <condition field="destination_number"
>     expression="^9999$">
>      >>>>>           <action application="answer"/>
>      >>>>>           <action application="bridge"
>      >>>>>     data="sofia/external/9999 at 10.10.20.100
>     <mailto:9999 at 10.10.20.100>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:9999 at 10.10.20.100 <mailto:9999 at 10.10.20.100>>
>     <mailto:9999 at 10.10.20.100 <mailto:9999 at 10.10.20.100>
>      >>>>     <mailto:9999 at 10.10.20.100 <mailto:9999 at 10.10.20.100>>>"/>
>      >>>>
>      >>>>
>      >>>>>           </condition>
>      >>>>>       </extension>
>      >>>>>       </context>
>      >>>>>
>      >>>>>     </include>
>      >>>>>
>      >>>>>     FreeSWITCH Version 1.0.trunk (11560M) Started.
>      >>>>>     Crash Protection [Disabled]
>      >>>>>     Max Sessions[1000]
>      >>>>>     Session Rate[100]
>      >>>>>     SQL [Enabled]
>      >>>>>
>      >>>>>
>      >>>>>     The test is very simple: sipp dial 9999 that matches in my
>      >>>>>
>      >>>>>
>      >>>>     FS dialplan
>      >>>>
>      >>>>
>      >>>>>     and this is bridged to an other FS machine playing music on
>      >>>>>
>      >>>>>
>      >>>>     hold.
>      >>>>
>      >>>>
>      >>>>>     When I launch "top"  I see after 30 to 40 s that FS
>     consumes all
>      >>>>>     the CPU
>      >>>>>     ressources (with a mean of 50-60 % before), with 80 calls.
>      >>>>>     When I set 70 calls, I have to wait 70-80 s before seeing
>      >>>>>
>      >>>>>
>      >>>>     the same
>      >>>>
>      >>>>
>      >>>>>     issue.
>      >>>>>
>      >>>>>     Presence is set to false on the 2 profile.
>      >>>>>
>      >>>>>     I have the same issue with FS 1.0.2 that' s why I tried FS
>      >>>>>
>      >>>>>
>      >>>>     11560.
>      >>>>
>      >>>>
>      >>>>>     When I use the FS machine as a router to test the packet per
>      >>>>>
>      >>>>>
>      >>>>     second
>      >>>>
>      >>>>
>      >>>>>     performance, I'm reaching 100Mbps with 8000pps in each
>      >>>>>
>      >>>>>
>      >>>>     direction (from
>      >>>>
>      >>>>
>      >>>>>     vlan 55 to vlan56) with less than 12% CPU. So that I
>     don't think
>      >>>>>     there's
>      >>>>>     an issue with the network.
>      >>>>>
>      >>>>>     Here is an "mpstat -P ALL 1" to show you what's happening
>      >>>>>
>      >>>>>
>      >>>>     suddenly
>      >>>>
>      >>>>
>      >>>>>     with
>      >>>>>     70 bridge calls:
>      >>>>>     12:31:26     CPU   %user   %nice    %sys %iowait    %irq
>       %soft
>      >>>>>     %steal   %idle    intr/s
>      >>>>>     12:31:27     all    3,00    0,00    3,00    0,00    1,00
>        4,00
>      >>>>>     0,00   89,00   6241,00
>      >>>>>     12:31:27       0    3,00    0,00    3,00    0,00    1,00
>        4,00
>      >>>>>     0,00   89,00   6241,00
>      >>>>>
>      >>>>>     12:31:27     CPU   %user   %nice    %sys %iowait    %irq
>       %soft
>      >>>>>     %steal   %idle    intr/s
>      >>>>>     12:31:28     all   14,14    0,00   56,57    0,00    2,02
>        5,05
>      >>>>>     0,00   22,22   6035,35
>      >>>>>     12:31:28       0   14,14    0,00   56,57    0,00    2,02
>        5,05
>      >>>>>     0,00   22,22   6035,35
>      >>>>>
>      >>>>>     12:31:28     CPU   %user   %nice    %sys %iowait    %irq
>       %soft
>      >>>>>     %steal   %idle    intr/s
>      >>>>>     12:31:29     all   24,75    0,00   67,33    0,00    0,99
>        6,93
>      >>>>>     0,00    0,00   5483,17
>      >>>>>     12:31:29       0   24,75    0,00   67,33    0,00    0,99
>        6,93
>      >>>>>     0,00    0,00   5483,17
>      >>>>>
>      >>>>>
>      >>>>>     The CPU is going from 89% idle to 0% in less than 2 seconds.
>      >>>>>
>      >>>>>     I know that I don't have to expect too much from this kind of
>      >>>>>     hardware,
>      >>>>>     but it seems strange that the CPU power vanished so suddenly.
>      >>>>>
>      >>>>>     Thanks a lot for the guys that have read this long mail :p
>      >>>>>
>      >>>>>     kind regards,
>      >>>>>     rod
>      >>>>>
>      >>>>>
>      >>>>>     _______________________________________________
>      >>>>>     Freeswitch-users mailing list
>      >>>>>     Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>>
>      >>>>
>      >>>>
>      >>>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>>>
>      >>>>
>      >>>>
>      >>>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>>>>
>      >>>>>
>      >>>>>
>      >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >
>      >>>>
>      >>>>
>      >>>>>     http://www.freeswitch.org
>      >>>>>
>      >>>>>
>      >>>>>
>      >>>>>
>      >>>>> --
>      >>>>> Anthony Minessale II
>      >>>>>
>      >>>>> FreeSWITCH http://www.freeswitch.org/
>      >>>>> ClueCon http://www.cluecon.com/
>      >>>>>
>      >>>>> AIM: anthm
>      >>>>> MSN:anthony_minessale at hotmail.com
>     <mailto:MSN%3Aanthony_minessale at hotmail.com>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:MSN%3Aanthony_minessale at hotmail.com
>     <mailto:MSN%253Aanthony_minessale at hotmail.com>>
>      >>>>
>      >>>>
>      >>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com
>     <mailto:MSN%253Aanthony_minessale at hotmail.com>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:MSN%253Aanthony_minessale at hotmail.com
>     <mailto:MSN%25253Aanthony_minessale at hotmail.com>>>
>      >>>>
>      >>>>
>      >>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com
>     <mailto:PAYPAL%253Aanthony.minessale at gmail.com>>
>      >>>>
>      >>>>
>      >>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com
>     <mailto:PAYPAL%253Aanthony.minessale at gmail.com>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:PAYPAL%253Aanthony.minessale at gmail.com
>     <mailto:PAYPAL%25253Aanthony.minessale at gmail.com>>>
>      >>>>
>      >>>>
>      >>>>> IRC: irc.freenode.net <http://irc.freenode.net>
>     <http://irc.freenode.net>
>      >>>>>
>      >>>>>
>      >>>>     <http://irc.freenode.net> #freeswitch
>      >>>>
>      >>>>
>      >>>>> FreeSWITCH Developer Conference
>      >>>>> sip:888 at conference.freeswitch.org
>     <mailto:sip%3A888 at conference.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:sip%3A888 at conference.freeswitch.org
>     <mailto:sip%253A888 at conference.freeswitch.org>>
>      >>>>
>      >>>>
>      >>>>> <mailto:sip%3A888 at conference.freeswitch.org
>     <mailto:sip%253A888 at conference.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:sip%253A888 at conference.freeswitch.org
>     <mailto:sip%25253A888 at conference.freeswitch.org>>>
>      >>>>
>      >>>>
>      >>>>> iax:guest at conference.freeswitch.org/888
>     <http://iax:guest@conference.freeswitch.org/888>
>      >>>>>
>      >>>>>
>      >>>>     <http://iax:guest@conference.freeswitch.org/888>
>      >>>>
>      >>>>
>      >>>>> <http://iax:guest@conference.freeswitch.org/888>
>      >>>>> googletalk:conf+888 at conference.freeswitch.org
>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org
>     <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org>>
>      >>>>
>      >>>>
>      >>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org
>     <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>    
>     <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org
>     <mailto:googletalk%25253Aconf%25252B888 at conference.freeswitch.org>>>
>      >>>>
>      >>>>
>      >>>>> pstn:213-799-1400
>      >>>>>
>      >>>>>
>      >>>>>
>      >
>     ------------------------------------------------------------------------
>      >
>      >>>>
>      >>>>
>      >>>>> _______________________________________________
>      >>>>> Freeswitch-users mailing list
>      >>>>> Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>>>>
>      >>>>>
>      >>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>>
>      >>>>
>      >>>>
>      >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>>>>
>      >>>>>
>      >>>>>
>      >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >
>      >>>>
>      >>>>
>      >>>>> http://www.freeswitch.org
>      >>>>>
>      >>>>>
>      >>>>>
>      >>>>     _______________________________________________
>      >>>>     Freeswitch-users mailing list
>      >>>>     Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>>
>      >>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>>>
>      >>>>
>      >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >
>      >>>>     http://www.freeswitch.org
>      >>>>
>      >>>>
>      >>>>
>     ------------------------------------------------------------------------
>      >>>>
>      >>>> _______________________________________________
>      >>>> Freeswitch-users mailing list
>      >>>> Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>>>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >>>> http://www.freeswitch.org
>      >>>>
>      >>>>
>      >>>>
>      >>> _______________________________________________
>      >>> Freeswitch-users mailing list
>      >>> Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >>> http://www.freeswitch.org
>      >>>
>      >>>
>      >>
>      >> _______________________________________________
>      >> Freeswitch-users mailing list
>      >> Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >>
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      >> http://www.freeswitch.org
>      >>
>      >>
>      >>
>      >>
>      >
>      > _______________________________________________
>      > Freeswitch-users mailing list
>      > Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      > http://www.freeswitch.org
>      >
>      >
>      > _______________________________________________
>      > Freeswitch-users mailing list
>      > Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>      > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>      >
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>      > http://www.freeswitch.org
>      >
>      >
>      >
> 
>     _______________________________________________
>     Freeswitch-users mailing list
>     Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
> 
> 
>     _______________________________________________
>     Freeswitch-users mailing list
>     Freeswitch-users at lists.freeswitch.org
>     <mailto:Freeswitch-users at lists.freeswitch.org>
>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>     http://www.freeswitch.org
> 
> 
> 
> 
> -- 
> Anthony Minessale II
> 
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> 
> AIM: anthm
> MSN:anthony_minessale at hotmail.com 
> <mailto:MSN%3Aanthony_minessale at hotmail.com>
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com 
> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
> 
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org 
> <mailto:sip%3A888 at conference.freeswitch.org>
> iax:guest at conference.freeswitch.org/888 
> <http://iax:guest@conference.freeswitch.org/888>
> googletalk:conf+888 at conference.freeswitch.org 
> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
> pstn:213-799-1400
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org




More information about the FreeSWITCH-users mailing list