[Freeswitch-users] Strange Performance when using as SBC
Anthony Minessale
anthony.minessale at gmail.com
Mon Feb 2 05:57:43 PST 2009
if you want to use ubuntu, be sure to use hardy and not intrepid.
On Mon, Feb 2, 2009 at 7:03 AM, Saeed Ahmed <saeedahmad1981 at gmail.com>wrote:
> Thanks rod for a quick answer,
>
> FS is installed on Ubuntu Server.
>
> I am planning to replace Nextone SBC with FS, Later I'll also use openZAP
> to
> communicate with TDM but this all depends how much calls it can take, or
> maybe we can also do something in clustering environment ( I am not sure
> about it). But thanks again and any further help will be highly
> appreciated.
>
>
> Kind Regards
> Saeed Ahmed Tariq
>
>
>
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
> Sent: Monday, February 02, 2009 1:53 PM
> To: freeswitch-users at lists.freeswitch.org
> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>
> Hi Saeed,
>
> I just created an account to share my setup on the wiki. I will detail
> all the steps for a clean install of a debian64 lenny with FS used as a
> SBC (next step is to try the new LCR module :) )and what I'm doing do
> stress the server.
>
> I wrote nothing at this time so please be patient, I'm waiting for my
> new hardware so that I will detail as much as possible what I'll do.
>
> For beginning I suggest you reading the start page on the wiki,
> especially these pages:
> -http://wiki.freeswitch.org/wiki/Getting_Started_Guide
> -http://wiki.freeswitch.org/wiki/Dialplan_XML
>
> maybe you could tell more about the linux distribution you're using so
> that I can give you some pointers for sipp...
>
> regards.
> rod.
>
>
> Saeed Ahmed wrote:
> > Hi Rod,
> >
> > Could you please share how you configured Sipp & FS to create a test
> > environment? Especially the dial plan, sofia settings etc..., actually I
> am
> > a newbie. I want to test it on a single FS machine.
> >
> > Kind Regards
> > Saeed
> > -----Original Message-----
> > From: freeswitch-users-bounces at lists.freeswitch.org
> > [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
> > Sent: Monday, February 02, 2009 11:00 AM
> > To: freeswitch-users at lists.freeswitch.org
> > Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
> >
> > Hi Ken, Jay,
> >
> > thanks for pointing to proxy media, I will test.
> >
> > Ken, you are right, I was brain damaged (a stupid mistake) when setting
> > INFO cause this kind of level could be very verbose. I'm switching to
> > CRIT or ERR.
> >
> > Thanks guys,
> > rod.
> >
> > thanks for
> >
> > Ken Rice wrote:
> >
> >> If you don't have to transcode, using proxy media mode will still save
> you
> >> some CPU time. This is 1/2 way between bypass media and the default
> media
> >> interactive mode. The other draw back to this mode is if you are using
> FS
> >>
> > to
> >
> >> clean up RTP and DTMF you loose those functions but they are not needed
> in
> >> most use cases.
> >>
> >> As far as the log level goes, I found that once I had things stable
> >>
> > setting
> >
> >> the loglevel to helped a good deal... Info is probably a bit too high of
> a
> >> loglevel I would probably go for CRIT or ERR (2 or 1 respectively) if
> you
> >> insist on leaving logging turned on... On a busy system these can and
> will
> >> generate a good deal of activity (and disk IO if using mod_logfile)
> >>
> >> Ken
> >>
> >>
> >>
> >>
> >>> From: rod <kawarod at laposte.net>
> >>> Reply-To: <freeswitch-users at lists.freeswitch.org>
> >>> Date: Mon, 02 Feb 2009 11:36:35 +0400
> >>> To: <freeswitch-users at lists.freeswitch.org>
> >>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
> >>>
> >>> Hi Ken,
> >>>
> >>> 1) I'd like to use FS to hide topology, so bypass media is not possible
> >>> 2) done
> >>> 3) done
> >>> 4) not used
> >>> 5) i'm using this ins switch.xml -> <param name="loglevel"
> >>> value="info"/>, if you think an other log level is more suitable.
> >>>
> >>> Regarding logging, I can see in console and in the freeswitch.log that
> >>> there is still a lot of NOTICE logging, see below:
> >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
> >>> switch_core_session_thread() Session 8721
> >>> (sofia/internal/sipp at 10.10.10.1:5060) Ended
> >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
> >>> switch_core_session_thread() Close Channel
> >>> sofia/internal/sipp at 10.10.10.1:5060 [CS_HANGUP]
> >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
> >>> switch_core_session_thread() Session 8722
> >>> (sofia/external/9998 at 10.10.20.100) Ended
> >>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
> >>> switch_core_session_thread() Close Channel
> >>> sofia/external/9998 at 10.10.20.100 [CS_HANGUP]
> >>> 2009-02-02 08:33:56 [NOTICE] sofia.c:3164 sofia_handle_sip_i_state()
> >>> Channel [sofia/external/9998 at 10.10.20.100] has been answered
> >>> 2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
> >>> Changing codec ptime to 30. I bet you have a linksys/sipura =D
> >>>
> >>> Do you have any idea where I can switch off this kind of logging. I
> >>> thought it should be in /dialplan/internal.xml, but I see that in
> >>> internal.xml -> <param name="debug" value="0"/>
> >>>
> >>> thanks a lot for your suggestion.
> >>>
> >>> regards,
> >>> rod
> >>>
> >>> Ken Rice wrote:
> >>>
> >>>
> >>>> Dont forget there are several things you can do to increase
> >>>>
> > performance...
> >
> >>>> 1) where possible use bypass media or media proxy modes
> >>>> 2) mount freeswitch/db as a ram drive (if you are using voicemail with
> >>>> the internal FS DBs you'll need a way to make this persistant across
> >>>> reboots)
> >>>> 3) see the wiki for setting reasonable ulimits
> >>>> 4) (this is my oppinion others may vary) dont use mod_cdr_csv
> >>>> 5) turn off (or reduce logging) in switch.conf.xml
> >>>>
> >>>> all of these thing can greatly improve performance.
> >>>>
> >>>> On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod at laposte.net
> >>>> <mailto:kawarod at laposte.net>> wrote:
> >>>>
> >>>> Thanks Anthony,
> >>>>
> >>>> the setup is like this:
> >>>>
> >>>> sipp server ---- FS 1 ---- FS2
> >>>>
> >>>> FS1 is the AMD CPU that has only one extension in dialplan that
> >>>> bridges
> >>>> 9999 to FS2. 9999 is the first extension in FS2 dialplan that
> >>>> plays moh,
> >>>> FS2 has no CPU pbm.
> >>>>
> >>>> FS1 is maxing out at 60 bridged calls without your option -hp.
> >>>>
> >>>> Using -hp, I'm now able to bridge 200 concurrent calls (a great
> >>>> improvement) and the system is still reactive. CPU load is high
> >>>> but not
> >>>> 100% and as the system responds well, I think that doesn't matter.
> >>>>
> > The
> >
> >>>> 2GB of memory are completely consumed (top command shows 700MB for
> >>>>
> > FS
> >
> >>>> process).
> >>>>
> >>>> I understand that FS1 server is not the best hardware platform,
> >>>> and I'm
> >>>> waiting for new 4 cores server for testing.
> >>>> I will update those numbers when testing with the new hardware.
> >>>>
> >>>> regards,
> >>>> rod.
> >>>>
> >>>> Anthony Minessale wrote:
> >>>>
> >>>>
> >>>>> Which of the 2 machines has the load issue? You said it was one box
> >>>>> calling the other.
> >>>>>
> >>>>> You have 2 major things against you, single CPU and AMD, but you
> >>>>> should at least be able to get in the vicinity of 800-1000 calls
> >>>>>
> >>>>>
> >>>> on a
> >>>>
> >>>>
> >>>>> box like that.
> >>>>>
> >>>>> Are you calling the default 9999? It's not really an appropriate
> >>>>> extension for load testing.
> >>>>> On the terminating box you should set up a manual extension that is
> >>>>> the first one in the dial plan
> >>>>> to play a wav file from preferably a ram disk or /tmp
> >>>>>
> >>>>> If you do plan on using this in production accept nothing less
> >>>>>
> >>>>>
> >>>> than a
> >>>>
> >>>>
> >>>>> multi-core intel machine with at least 4 cores, the more cores the
> >>>>> better because that parallel processing is where FS gets it's
> >>>>>
> >>>>>
> >>>> atvantage.
> >>>>
> >>>>
> >>>>> On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod at laposte.net
> >>>>>
> >>>>>
> >>>> <mailto:kawarod at laposte.net>
> >>>>
> >>>>
> >>>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>> wrote:
> >>>>>
> >>>>> Dear list,
> >>>>>
> >>>>> I've been playing with freeswitch for some time (2 months)
> >>>>>
> >>>>>
> >>>> and the
> >>>>
> >>>>
> >>>>> fact
> >>>>> is that I'm very pleased with the functionnalities of this
> >>>>>
> >>>>>
> >>>> software.
> >>>>
> >>>>
> >>>>> I'd like to use FS as a SBC handling media and I'm doing some
> >>>>> tests with
> >>>>> sipp to load the machine but I'm unable to bridge more than
> >>>>>
> >>>>>
> >>>> 60 calls
> >>>>
> >>>>
> >>>>> without seeing the CPU being loaded at 100%. I'm sure
> >>>>>
> >>>>>
> >>>> something is
> >>>>
> >>>>
> >>>>> going
> >>>>> wrong with my setup but I'm unable to see what.
> >>>>>
> >>>>> The test machine has the following specs:
> >>>>> Athlon XP 3500+ with 2GB of memory (I know this is not a
> >>>>>
> >>>>>
> >>>> high end
> >>>>
> >>>>
> >>>>> machine :p)
> >>>>>
> >>>>> Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
> >>>>> processor : 0
> >>>>> vendor_id : AuthenticAMD
> >>>>> cpu family : 15
> >>>>> model : 95
> >>>>> model name : AMD Athlon(tm) 64 Processor 3500+
> >>>>> stepping : 2
> >>>>> cpu MHz : 2199.973
> >>>>> cache size : 512 KB
> >>>>> fpu : yes
> >>>>> fpu_exception : yes
> >>>>> cpuid level : 1
> >>>>> wp : yes
> >>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic
> >>>>>
> >>>>>
> >>>> sep mtrr pge
> >>>>
> >>>>
> >>>>> mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
> >>>>> fxsr_opt
> >>>>> rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm
> >>>>>
> >>>>>
> >>>> extapic
> >>>>
> >>>>
> >>>>> cr8_legacy
> >>>>> bogomips : 4402.97
> >>>>> TLB size : 1024 4K pages
> >>>>> clflush size : 64
> >>>>> cache_alignment : 64
> >>>>> address sizes : 40 bits physical, 48 bits virtual
> >>>>> power management: ts fid vid ttp tm stc
> >>>>>
> >>>>> I installed FS on a fresh debian 64:
> >>>>> Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00
> >>>>>
> >>>>>
> >>>> UTC 2009
> >>>>
> >>>>
> >>>>> x86_64 GNU/Linux
> >>>>>
> >>>>> I set the ulimit parameters like those on the website:
> >>>>> freeswitch at internal> ...
> >>>>> Freeswitch:/opt/free-svn/bin# ulimit -a
> >>>>> core file size (blocks, -c) unlimited
> >>>>> data seg size (kbytes, -d) unlimited
> >>>>> scheduling priority (-e) 0
> >>>>> file size (blocks, -f) unlimited
> >>>>> pending signals (-i) unlimited
> >>>>> max locked memory (kbytes, -l) unlimited
> >>>>> max memory size (kbytes, -m) unlimited
> >>>>> open files (-n) 999999
> >>>>> pipe size (512 bytes, -p) 8
> >>>>> POSIX message queues (bytes, -q) unlimited
> >>>>> real-time priority (-r) 0
> >>>>> stack size (kbytes, -s) 244
> >>>>> cpu time (seconds, -t) unlimited
> >>>>> max user processes (-u) unlimited
> >>>>> virtual memory (kbytes, -v) unlimited
> >>>>> file locks (-x) unlimited
> >>>>>
> >>>>>
> >>>>> My network setup is the following:
> >>>>>
> >>>>> SIPP machine (10.10.10.1/24)----------------vlan<http://10.10.10.1/24%29----------------vlan>
> >>>>>
> >>>>>
> >>>> <http://10.10.10.1/24%29----------------vlan>
> >>>>
> >>>>
> >>>>> <http://10.10.10.1/24%29----------------vlan> 55
> >>>>> ----------(10.10.10.254/24 <http://10.10.10.254/24>
> >>>>>
> >>>>>
> >>>> <http://10.10.10.254/24>) FS
> >>>>
> >>>>
> >>>>> (10.10.20.254/24)--------------<http://10.10.20.254/24%29-------------->
> >>>>>
> >>>>>
> >>>> <http://10.10.20.254/24%29-------------->
> >>>>
> >>>>
> >>>>> <http://10.10.20.254/24%29--------------> vlan56
> >>>>> -------------------(10.10.20.100/24 <http://10.10.20.100/24>
> >>>>>
> >>>>>
> >>>> <http://10.10.20.100/24>)
> >>>>
> >>>>
> >>>>> OTHER STOCK FS
> >>>>>
> >>>>>
> >>>>> I launched sipp with:
> >>>>> sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi
> >>>>>
> >>>>>
> >>>> 10.10.10.1 -i
> >>>>
> >>>>
> >>>>> 10.10.10.1 -mp 25000 10.10.10.254:5060
> >>>>>
> >>>>>
> >>>> <http://10.10.10.254:5060> <http://10.10.10.254:5060>
> >>>>
> >>>>
> >>>>> The dialplan on FS is very simple:
> >>>>> <?xml version="1.0" encoding="utf-8"?>
> >>>>> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
> >>>>> <include>
> >>>>> <context name="default">
> >>>>>
> >>>>> <extension name="hold_music_bridge">
> >>>>> <condition field="destination_number" expression="^9999$">
> >>>>> <action application="answer"/>
> >>>>> <action application="bridge"
> >>>>> data="sofia/external/9999 at 10.10.20.100
> >>>>>
> >>>>>
> >>>> <mailto:9999 at 10.10.20.100> <mailto:9999 at 10.10.20.100
> >>>> <mailto:9999 at 10.10.20.100>>"/>
> >>>>
> >>>>
> >>>>> </condition>
> >>>>> </extension>
> >>>>> </context>
> >>>>>
> >>>>> </include>
> >>>>>
> >>>>> FreeSWITCH Version 1.0.trunk (11560M) Started.
> >>>>> Crash Protection [Disabled]
> >>>>> Max Sessions[1000]
> >>>>> Session Rate[100]
> >>>>> SQL [Enabled]
> >>>>>
> >>>>>
> >>>>> The test is very simple: sipp dial 9999 that matches in my
> >>>>>
> >>>>>
> >>>> FS dialplan
> >>>>
> >>>>
> >>>>> and this is bridged to an other FS machine playing music on
> >>>>>
> >>>>>
> >>>> hold.
> >>>>
> >>>>
> >>>>> When I launch "top" I see after 30 to 40 s that FS consumes all
> >>>>> the CPU
> >>>>> ressources (with a mean of 50-60 % before), with 80 calls.
> >>>>> When I set 70 calls, I have to wait 70-80 s before seeing
> >>>>>
> >>>>>
> >>>> the same
> >>>>
> >>>>
> >>>>> issue.
> >>>>>
> >>>>> Presence is set to false on the 2 profile.
> >>>>>
> >>>>> I have the same issue with FS 1.0.2 that' s why I tried FS
> >>>>>
> >>>>>
> >>>> 11560.
> >>>>
> >>>>
> >>>>> When I use the FS machine as a router to test the packet per
> >>>>>
> >>>>>
> >>>> second
> >>>>
> >>>>
> >>>>> performance, I'm reaching 100Mbps with 8000pps in each
> >>>>>
> >>>>>
> >>>> direction (from
> >>>>
> >>>>
> >>>>> vlan 55 to vlan56) with less than 12% CPU. So that I don't think
> >>>>> there's
> >>>>> an issue with the network.
> >>>>>
> >>>>> Here is an "mpstat -P ALL 1" to show you what's happening
> >>>>>
> >>>>>
> >>>> suddenly
> >>>>
> >>>>
> >>>>> with
> >>>>> 70 bridge calls:
> >>>>> 12:31:26 CPU %user %nice %sys %iowait %irq %soft
> >>>>> %steal %idle intr/s
> >>>>> 12:31:27 all 3,00 0,00 3,00 0,00 1,00 4,00
> >>>>> 0,00 89,00 6241,00
> >>>>> 12:31:27 0 3,00 0,00 3,00 0,00 1,00 4,00
> >>>>> 0,00 89,00 6241,00
> >>>>>
> >>>>> 12:31:27 CPU %user %nice %sys %iowait %irq %soft
> >>>>> %steal %idle intr/s
> >>>>> 12:31:28 all 14,14 0,00 56,57 0,00 2,02 5,05
> >>>>> 0,00 22,22 6035,35
> >>>>> 12:31:28 0 14,14 0,00 56,57 0,00 2,02 5,05
> >>>>> 0,00 22,22 6035,35
> >>>>>
> >>>>> 12:31:28 CPU %user %nice %sys %iowait %irq %soft
> >>>>> %steal %idle intr/s
> >>>>> 12:31:29 all 24,75 0,00 67,33 0,00 0,99 6,93
> >>>>> 0,00 0,00 5483,17
> >>>>> 12:31:29 0 24,75 0,00 67,33 0,00 0,99 6,93
> >>>>> 0,00 0,00 5483,17
> >>>>>
> >>>>>
> >>>>> The CPU is going from 89% idle to 0% in less than 2 seconds.
> >>>>>
> >>>>> I know that I don't have to expect too much from this kind of
> >>>>> hardware,
> >>>>> but it seems strange that the CPU power vanished so suddenly.
> >>>>>
> >>>>> Thanks a lot for the guys that have read this long mail :p
> >>>>>
> >>>>> kind regards,
> >>>>> rod
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> Freeswitch-users mailing list
> >>>>> Freeswitch-users at lists.freeswitch.org
> >>>>>
> >>>>>
> >>>> <mailto:Freeswitch-users at lists.freeswitch.org>
> >>>>
> >>>>
> >>>>> <mailto:Freeswitch-users at lists.freeswitch.org
> >>>>>
> >>>>>
> >>>> <mailto:Freeswitch-users at lists.freeswitch.org>>
> >>>>
> >>>>
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> >>>>>
> >>>>>
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >
> >>>>
> >>>>
> >>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Anthony Minessale II
> >>>>>
> >>>>> FreeSWITCH http://www.freeswitch.org/
> >>>>> ClueCon http://www.cluecon.com/
> >>>>>
> >>>>> AIM: anthm
> >>>>> MSN:anthony_minessale at hotmail.com<MSN%3Aanthony_minessale at hotmail.com>
> >>>>>
> >>>>>
> >>>> <mailto:MSN%3Aanthony_minessale at hotmail.com<MSN%253Aanthony_minessale at hotmail.com>
> >
> >>>>
> >>>>
> >>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com<MSN%253Aanthony_minessale at hotmail.com>
> >>>>>
> >>>>>
> >>>> <mailto:MSN%253Aanthony_minessale at hotmail.com<MSN%25253Aanthony_minessale at hotmail.com>
> >>
> >>>>
> >>>>
> >>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
> >>>>>
> >>>>>
> >>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com<PAYPAL%253Aanthony.minessale at gmail.com>
> >
> >>>>
> >>>>
> >>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com<PAYPAL%253Aanthony.minessale at gmail.com>
> >>>>>
> >>>>>
> >>>> <mailto:PAYPAL%253Aanthony.minessale at gmail.com<PAYPAL%25253Aanthony.minessale at gmail.com>
> >>
> >>>>
> >>>>
> >>>>> IRC: irc.freenode.net <http://irc.freenode.net>
> >>>>>
> >>>>>
> >>>> <http://irc.freenode.net> #freeswitch
> >>>>
> >>>>
> >>>>> FreeSWITCH Developer Conference
> >>>>> sip:888 at conference.freeswitch.org<sip%3A888 at conference.freeswitch.org>
> >>>>>
> >>>>>
> >>>> <mailto:sip%3A888 at conference.freeswitch.org<sip%253A888 at conference.freeswitch.org>
> >
> >>>>
> >>>>
> >>>>> <mailto:sip%3A888 at conference.freeswitch.org<sip%253A888 at conference.freeswitch.org>
> >>>>>
> >>>>>
> >>>> <mailto:sip%253A888 at conference.freeswitch.org<sip%25253A888 at conference.freeswitch.org>
> >>
> >>>>
> >>>>
> >>>>> iax:guest at conference.freeswitch.org/888
> >>>>>
> >>>>>
> >>>> <http://iax:guest@conference.freeswitch.org/888>
> >>>>
> >>>>
> >>>>> <http://iax:guest@conference.freeswitch.org/888>
> >>>>> googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
> >>>>>
> >>>>>
> >>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org<googletalk%253Aconf%252B888 at conference.freeswitch.org>
> >
> >>>>
> >>>>
> >>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org<googletalk%253Aconf%252B888 at conference.freeswitch.org>
> >>>>>
> >>>>>
> >>>> <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org<googletalk%25253Aconf%25252B888 at conference.freeswitch.org>
> >>
> >>>>
> >>>>
> >>>>> pstn:213-799-1400
> >>>>>
> >>>>>
> >>>>>
> > ------------------------------------------------------------------------
> >
> >>>>
> >>>>
> >>>>> _______________________________________________
> >>>>> Freeswitch-users mailing list
> >>>>> Freeswitch-users at lists.freeswitch.org
> >>>>>
> >>>>>
> >>>> <mailto:Freeswitch-users at lists.freeswitch.org>
> >>>>
> >>>>
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> >>>>>
> >>>>>
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >
> >>>>
> >>>>
> >>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> Freeswitch-users mailing list
> >>>> Freeswitch-users at lists.freeswitch.org
> >>>> <mailto:Freeswitch-users at lists.freeswitch.org>
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>
> >>>>
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >
> >>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------------------
> >>>>
> >>>> _______________________________________________
> >>>> Freeswitch-users mailing list
> >>>> Freeswitch-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>>
> >>> _______________________________________________
> >>> Freeswitch-users mailing list
> >>> Freeswitch-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>>
> >>>
> >>
> >> _______________________________________________
> >> Freeswitch-users mailing list
> >> Freeswitch-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >>
> >>
> >>
> >>
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> > _______________________________________________
> > Freeswitch-users mailing list
> > Freeswitch-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> >
> >
> >
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale at hotmail.com <MSN%3Aanthony_minessale at hotmail.com>
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com<PAYPAL%3Aanthony.minessale at gmail.com>
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org <sip%3A888 at conference.freeswitch.org>
iax:guest at conference.freeswitch.org/888
googletalk:conf+888 at conference.freeswitch.org<googletalk%3Aconf%2B888 at conference.freeswitch.org>
pstn:213-799-1400
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20090202/e58251ab/attachment-0002.html
More information about the FreeSWITCH-users
mailing list