[Freeswitch-users] Strange Performance when using as SBC
Saeed Ahmed
saeedahmad1981 at gmail.com
Tue Feb 3 04:41:03 PST 2009
Hi rod,
It's really amazing! Well described!
Could you please explain a bit why we used Kamailio?
Kind Regards
Saeed Ahmed Tariq
-----Original Message-----
From: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
Sent: Monday, February 02, 2009 3:33 PM
To: freeswitch-users at lists.freeswitch.org
Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
Hi Saeed,
Here is a first draft of what I did to install FS on my server.
Configuration are not present, they'll be in a next release :p
http://wiki.freeswitch.org/wiki/SBC_Setup
My aim is to setup FS as a SBC, I hope this page could be a great
startup point for others. I will update regularly based on what I did.
Saeed, why are you replacing your Nextone, it's said to be one of the
best commercial SBC on the market.
regards.
Saeed Ahmed wrote:
> Thanks rod for a quick answer,
>
> FS is installed on Ubuntu Server.
>
> I am planning to replace Nextone SBC with FS, Later I'll also use openZAP
to
> communicate with TDM but this all depends how much calls it can take, or
> maybe we can also do something in clustering environment ( I am not sure
> about it). But thanks again and any further help will be highly
appreciated.
>
>
> Kind Regards
> Saeed Ahmed Tariq
>
>
>
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
> Sent: Monday, February 02, 2009 1:53 PM
> To: freeswitch-users at lists.freeswitch.org
> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>
> Hi Saeed,
>
> I just created an account to share my setup on the wiki. I will detail
> all the steps for a clean install of a debian64 lenny with FS used as a
> SBC (next step is to try the new LCR module :) )and what I'm doing do
> stress the server.
>
> I wrote nothing at this time so please be patient, I'm waiting for my
> new hardware so that I will detail as much as possible what I'll do.
>
> For beginning I suggest you reading the start page on the wiki,
> especially these pages:
> -http://wiki.freeswitch.org/wiki/Getting_Started_Guide
> -http://wiki.freeswitch.org/wiki/Dialplan_XML
>
> maybe you could tell more about the linux distribution you're using so
> that I can give you some pointers for sipp...
>
> regards.
> rod.
>
>
> Saeed Ahmed wrote:
>
>> Hi Rod,
>>
>> Could you please share how you configured Sipp & FS to create a test
>> environment? Especially the dial plan, sofia settings etc..., actually I
>>
> am
>
>> a newbie. I want to test it on a single FS machine.
>>
>> Kind Regards
>> Saeed
>> -----Original Message-----
>> From: freeswitch-users-bounces at lists.freeswitch.org
>> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
>> Sent: Monday, February 02, 2009 11:00 AM
>> To: freeswitch-users at lists.freeswitch.org
>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>>
>> Hi Ken, Jay,
>>
>> thanks for pointing to proxy media, I will test.
>>
>> Ken, you are right, I was brain damaged (a stupid mistake) when setting
>> INFO cause this kind of level could be very verbose. I'm switching to
>> CRIT or ERR.
>>
>> Thanks guys,
>> rod.
>>
>> thanks for
>>
>> Ken Rice wrote:
>>
>>
>>> If you don't have to transcode, using proxy media mode will still save
>>>
> you
>
>>> some CPU time. This is 1/2 way between bypass media and the default
media
>>> interactive mode. The other draw back to this mode is if you are using
FS
>>>
>>>
>> to
>>
>>
>>> clean up RTP and DTMF you loose those functions but they are not needed
>>>
> in
>
>>> most use cases.
>>>
>>> As far as the log level goes, I found that once I had things stable
>>>
>>>
>> setting
>>
>>
>>> the loglevel to helped a good deal... Info is probably a bit too high of
>>>
> a
>
>>> loglevel I would probably go for CRIT or ERR (2 or 1 respectively) if
you
>>> insist on leaving logging turned on... On a busy system these can and
>>>
> will
>
>>> generate a good deal of activity (and disk IO if using mod_logfile)
>>>
>>> Ken
>>>
>>>
>>>
>>>
>>>
>>>> From: rod <kawarod at laposte.net>
>>>> Reply-To: <freeswitch-users at lists.freeswitch.org>
>>>> Date: Mon, 02 Feb 2009 11:36:35 +0400
>>>> To: <freeswitch-users at lists.freeswitch.org>
>>>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>>>>
>>>> Hi Ken,
>>>>
>>>> 1) I'd like to use FS to hide topology, so bypass media is not possible
>>>> 2) done
>>>> 3) done
>>>> 4) not used
>>>> 5) i'm using this ins switch.xml -> <param name="loglevel"
>>>> value="info"/>, if you think an other log level is more suitable.
>>>>
>>>> Regarding logging, I can see in console and in the freeswitch.log that
>>>> there is still a lot of NOTICE logging, see below:
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>>> switch_core_session_thread() Session 8721
>>>> (sofia/internal/sipp at 10.10.10.1:5060) Ended
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>>> switch_core_session_thread() Close Channel
>>>> sofia/internal/sipp at 10.10.10.1:5060 [CS_HANGUP]
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>>> switch_core_session_thread() Session 8722
>>>> (sofia/external/9998 at 10.10.20.100) Ended
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>>> switch_core_session_thread() Close Channel
>>>> sofia/external/9998 at 10.10.20.100 [CS_HANGUP]
>>>> 2009-02-02 08:33:56 [NOTICE] sofia.c:3164 sofia_handle_sip_i_state()
>>>> Channel [sofia/external/9998 at 10.10.20.100] has been answered
>>>> 2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
>>>> Changing codec ptime to 30. I bet you have a linksys/sipura =D
>>>>
>>>> Do you have any idea where I can switch off this kind of logging. I
>>>> thought it should be in /dialplan/internal.xml, but I see that in
>>>> internal.xml -> <param name="debug" value="0"/>
>>>>
>>>> thanks a lot for your suggestion.
>>>>
>>>> regards,
>>>> rod
>>>>
>>>> Ken Rice wrote:
>>>>
>>>>
>>>>
>>>>> Dont forget there are several things you can do to increase
>>>>>
>>>>>
>> performance...
>>
>>
>>>>> 1) where possible use bypass media or media proxy modes
>>>>> 2) mount freeswitch/db as a ram drive (if you are using voicemail with
>>>>> the internal FS DBs you'll need a way to make this persistant across
>>>>> reboots)
>>>>> 3) see the wiki for setting reasonable ulimits
>>>>> 4) (this is my oppinion others may vary) dont use mod_cdr_csv
>>>>> 5) turn off (or reduce logging) in switch.conf.xml
>>>>>
>>>>> all of these thing can greatly improve performance.
>>>>>
>>>>> On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod at laposte.net
>>>>> <mailto:kawarod at laposte.net>> wrote:
>>>>>
>>>>> Thanks Anthony,
>>>>>
>>>>> the setup is like this:
>>>>>
>>>>> sipp server ---- FS 1 ---- FS2
>>>>>
>>>>> FS1 is the AMD CPU that has only one extension in dialplan that
>>>>> bridges
>>>>> 9999 to FS2. 9999 is the first extension in FS2 dialplan that
>>>>> plays moh,
>>>>> FS2 has no CPU pbm.
>>>>>
>>>>> FS1 is maxing out at 60 bridged calls without your option -hp.
>>>>>
>>>>> Using -hp, I'm now able to bridge 200 concurrent calls (a great
>>>>> improvement) and the system is still reactive. CPU load is high
>>>>> but not
>>>>> 100% and as the system responds well, I think that doesn't matter.
>>>>>
>>>>>
>> The
>>
>>
>>>>> 2GB of memory are completely consumed (top command shows 700MB for
>>>>>
>>>>>
>> FS
>>
>>
>>>>> process).
>>>>>
>>>>> I understand that FS1 server is not the best hardware platform,
>>>>> and I'm
>>>>> waiting for new 4 cores server for testing.
>>>>> I will update those numbers when testing with the new hardware.
>>>>>
>>>>> regards,
>>>>> rod.
>>>>>
>>>>> Anthony Minessale wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Which of the 2 machines has the load issue? You said it was one box
>>>>>> calling the other.
>>>>>>
>>>>>> You have 2 major things against you, single CPU and AMD, but you
>>>>>> should at least be able to get in the vicinity of 800-1000 calls
>>>>>>
>>>>>>
>>>>>>
>>>>> on a
>>>>>
>>>>>
>>>>>
>>>>>> box like that.
>>>>>>
>>>>>> Are you calling the default 9999? It's not really an appropriate
>>>>>> extension for load testing.
>>>>>> On the terminating box you should set up a manual extension that is
>>>>>> the first one in the dial plan
>>>>>> to play a wav file from preferably a ram disk or /tmp
>>>>>>
>>>>>> If you do plan on using this in production accept nothing less
>>>>>>
>>>>>>
>>>>>>
>>>>> than a
>>>>>
>>>>>
>>>>>
>>>>>> multi-core intel machine with at least 4 cores, the more cores the
>>>>>> better because that parallel processing is where FS gets it's
>>>>>>
>>>>>>
>>>>>>
>>>>> atvantage.
>>>>>
>>>>>
>>>>>
>>>>>> On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod at laposte.net
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:kawarod at laposte.net>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>> wrote:
>>>>>>
>>>>>> Dear list,
>>>>>>
>>>>>> I've been playing with freeswitch for some time (2 months)
>>>>>>
>>>>>>
>>>>>>
>>>>> and the
>>>>>
>>>>>
>>>>>
>>>>>> fact
>>>>>> is that I'm very pleased with the functionnalities of this
>>>>>>
>>>>>>
>>>>>>
>>>>> software.
>>>>>
>>>>>
>>>>>
>>>>>> I'd like to use FS as a SBC handling media and I'm doing some
>>>>>> tests with
>>>>>> sipp to load the machine but I'm unable to bridge more than
>>>>>>
>>>>>>
>>>>>>
>>>>> 60 calls
>>>>>
>>>>>
>>>>>
>>>>>> without seeing the CPU being loaded at 100%. I'm sure
>>>>>>
>>>>>>
>>>>>>
>>>>> something is
>>>>>
>>>>>
>>>>>
>>>>>> going
>>>>>> wrong with my setup but I'm unable to see what.
>>>>>>
>>>>>> The test machine has the following specs:
>>>>>> Athlon XP 3500+ with 2GB of memory (I know this is not a
>>>>>>
>>>>>>
>>>>>>
>>>>> high end
>>>>>
>>>>>
>>>>>
>>>>>> machine :p)
>>>>>>
>>>>>> Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
>>>>>> processor : 0
>>>>>> vendor_id : AuthenticAMD
>>>>>> cpu family : 15
>>>>>> model : 95
>>>>>> model name : AMD Athlon(tm) 64 Processor 3500+
>>>>>> stepping : 2
>>>>>> cpu MHz : 2199.973
>>>>>> cache size : 512 KB
>>>>>> fpu : yes
>>>>>> fpu_exception : yes
>>>>>> cpuid level : 1
>>>>>> wp : yes
>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic
>>>>>>
>>>>>>
>>>>>>
>>>>> sep mtrr pge
>>>>>
>>>>>
>>>>>
>>>>>> mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
>>>>>> fxsr_opt
>>>>>> rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm
>>>>>>
>>>>>>
>>>>>>
>>>>> extapic
>>>>>
>>>>>
>>>>>
>>>>>> cr8_legacy
>>>>>> bogomips : 4402.97
>>>>>> TLB size : 1024 4K pages
>>>>>> clflush size : 64
>>>>>> cache_alignment : 64
>>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>>> power management: ts fid vid ttp tm stc
>>>>>>
>>>>>> I installed FS on a fresh debian 64:
>>>>>> Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00
>>>>>>
>>>>>>
>>>>>>
>>>>> UTC 2009
>>>>>
>>>>>
>>>>>
>>>>>> x86_64 GNU/Linux
>>>>>>
>>>>>> I set the ulimit parameters like those on the website:
>>>>>> freeswitch at internal> ...
>>>>>> Freeswitch:/opt/free-svn/bin# ulimit -a
>>>>>> core file size (blocks, -c) unlimited
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) unlimited
>>>>>> max locked memory (kbytes, -l) unlimited
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 999999
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) unlimited
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 244
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) unlimited
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>>
>>>>>> My network setup is the following:
>>>>>>
>>>>>> SIPP machine (10.10.10.1/24)----------------vlan
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://10.10.10.1/24%29----------------vlan>
>>>>>
>>>>>
>>>>>
>>>>>> <http://10.10.10.1/24%29----------------vlan> 55
>>>>>> ----------(10.10.10.254/24 <http://10.10.10.254/24>
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://10.10.10.254/24>) FS
>>>>>
>>>>>
>>>>>
>>>>>> (10.10.20.254/24)--------------
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://10.10.20.254/24%29-------------->
>>>>>
>>>>>
>>>>>
>>>>>> <http://10.10.20.254/24%29--------------> vlan56
>>>>>> -------------------(10.10.20.100/24 <http://10.10.20.100/24>
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://10.10.20.100/24>)
>>>>>
>>>>>
>>>>>
>>>>>> OTHER STOCK FS
>>>>>>
>>>>>>
>>>>>> I launched sipp with:
>>>>>> sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi
>>>>>>
>>>>>>
>>>>>>
>>>>> 10.10.10.1 -i
>>>>>
>>>>>
>>>>>
>>>>>> 10.10.10.1 -mp 25000 10.10.10.254:5060
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://10.10.10.254:5060> <http://10.10.10.254:5060>
>>>>>
>>>>>
>>>>>
>>>>>> The dialplan on FS is very simple:
>>>>>> <?xml version="1.0" encoding="utf-8"?>
>>>>>> <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
>>>>>> <include>
>>>>>> <context name="default">
>>>>>>
>>>>>> <extension name="hold_music_bridge">
>>>>>> <condition field="destination_number" expression="^9999$">
>>>>>> <action application="answer"/>
>>>>>> <action application="bridge"
>>>>>> data="sofia/external/9999 at 10.10.20.100
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:9999 at 10.10.20.100> <mailto:9999 at 10.10.20.100
>>>>> <mailto:9999 at 10.10.20.100>>"/>
>>>>>
>>>>>
>>>>>
>>>>>> </condition>
>>>>>> </extension>
>>>>>> </context>
>>>>>>
>>>>>> </include>
>>>>>>
>>>>>> FreeSWITCH Version 1.0.trunk (11560M) Started.
>>>>>> Crash Protection [Disabled]
>>>>>> Max Sessions[1000]
>>>>>> Session Rate[100]
>>>>>> SQL [Enabled]
>>>>>>
>>>>>>
>>>>>> The test is very simple: sipp dial 9999 that matches in my
>>>>>>
>>>>>>
>>>>>>
>>>>> FS dialplan
>>>>>
>>>>>
>>>>>
>>>>>> and this is bridged to an other FS machine playing music on
>>>>>>
>>>>>>
>>>>>>
>>>>> hold.
>>>>>
>>>>>
>>>>>
>>>>>> When I launch "top" I see after 30 to 40 s that FS consumes all
>>>>>> the CPU
>>>>>> ressources (with a mean of 50-60 % before), with 80 calls.
>>>>>> When I set 70 calls, I have to wait 70-80 s before seeing
>>>>>>
>>>>>>
>>>>>>
>>>>> the same
>>>>>
>>>>>
>>>>>
>>>>>> issue.
>>>>>>
>>>>>> Presence is set to false on the 2 profile.
>>>>>>
>>>>>> I have the same issue with FS 1.0.2 that' s why I tried FS
>>>>>>
>>>>>>
>>>>>>
>>>>> 11560.
>>>>>
>>>>>
>>>>>
>>>>>> When I use the FS machine as a router to test the packet per
>>>>>>
>>>>>>
>>>>>>
>>>>> second
>>>>>
>>>>>
>>>>>
>>>>>> performance, I'm reaching 100Mbps with 8000pps in each
>>>>>>
>>>>>>
>>>>>>
>>>>> direction (from
>>>>>
>>>>>
>>>>>
>>>>>> vlan 55 to vlan56) with less than 12% CPU. So that I don't think
>>>>>> there's
>>>>>> an issue with the network.
>>>>>>
>>>>>> Here is an "mpstat -P ALL 1" to show you what's happening
>>>>>>
>>>>>>
>>>>>>
>>>>> suddenly
>>>>>
>>>>>
>>>>>
>>>>>> with
>>>>>> 70 bridge calls:
>>>>>> 12:31:26 CPU %user %nice %sys %iowait %irq %soft
>>>>>> %steal %idle intr/s
>>>>>> 12:31:27 all 3,00 0,00 3,00 0,00 1,00 4,00
>>>>>> 0,00 89,00 6241,00
>>>>>> 12:31:27 0 3,00 0,00 3,00 0,00 1,00 4,00
>>>>>> 0,00 89,00 6241,00
>>>>>>
>>>>>> 12:31:27 CPU %user %nice %sys %iowait %irq %soft
>>>>>> %steal %idle intr/s
>>>>>> 12:31:28 all 14,14 0,00 56,57 0,00 2,02 5,05
>>>>>> 0,00 22,22 6035,35
>>>>>> 12:31:28 0 14,14 0,00 56,57 0,00 2,02 5,05
>>>>>> 0,00 22,22 6035,35
>>>>>>
>>>>>> 12:31:28 CPU %user %nice %sys %iowait %irq %soft
>>>>>> %steal %idle intr/s
>>>>>> 12:31:29 all 24,75 0,00 67,33 0,00 0,99 6,93
>>>>>> 0,00 0,00 5483,17
>>>>>> 12:31:29 0 24,75 0,00 67,33 0,00 0,99 6,93
>>>>>> 0,00 0,00 5483,17
>>>>>>
>>>>>>
>>>>>> The CPU is going from 89% idle to 0% in less than 2 seconds.
>>>>>>
>>>>>> I know that I don't have to expect too much from this kind of
>>>>>> hardware,
>>>>>> but it seems strange that the CPU power vanished so suddenly.
>>>>>>
>>>>>> Thanks a lot for the guys that have read this long mail :p
>>>>>>
>>>>>> kind regards,
>>>>>> rod
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Freeswitch-users mailing list
>>>>>> Freeswitch-users at lists.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:Freeswitch-users at lists.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:Freeswitch-users at lists.freeswitch.org>>
>>>>>
>>>>>
>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>>
>>>>>
>>>>>
>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> AIM: anthm
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:MSN%253Aanthony_minessale at hotmail.com>>
>>>>>
>>>>>
>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:PAYPAL%253Aanthony.minessale at gmail.com>>
>>>>>
>>>>>
>>>>>
>>>>>> IRC: irc.freenode.net <http://irc.freenode.net>
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://irc.freenode.net> #freeswitch
>>>>>
>>>>>
>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:sip%3A888 at conference.freeswitch.org>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:sip%3A888 at conference.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:sip%253A888 at conference.freeswitch.org>>
>>>>>
>>>>>
>>>>>
>>>>>> iax:guest at conference.freeswitch.org/888
>>>>>>
>>>>>>
>>>>>>
>>>>> <http://iax:guest@conference.freeswitch.org/888>
>>>>>
>>>>>
>>>>>
>>>>>> <http://iax:guest@conference.freeswitch.org/888>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>>>
>>>>>
>>>>>
>>>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org>>
>>>>>
>>>>>
>>>>>
>>>>>> pstn:213-799-1400
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>> ------------------------------------------------------------------------
>>
>>
>>>>>
>>>>>
>>>>>
>>>>>> _______________________________________________
>>>>>> Freeswitch-users mailing list
>>>>>> Freeswitch-users at lists.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>> <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>>
>>>>>
>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>>
>>>>>
>>>>>
>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Freeswitch-users mailing list
>>>>> Freeswitch-users at lists.freeswitch.org
>>>>> <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>>
>>>>>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>
>>
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
> ------------------------------------------------------------------------
>
>>>>> _______________________________________________
>>>>> Freeswitch-users mailing list
>>>>> Freeswitch-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Freeswitch-users mailing list
>>>> Freeswitch-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
More information about the Freeswitch-users
mailing list