[Freeswitch-users] Strange Performance when using as SBC

Saeed Ahmed saeedahmad1981 at gmail.com
Tue Feb 3 04:41:03 PST 2009


Hi rod,

It's really amazing! Well described!

Could you please explain a bit why we used Kamailio?

Kind Regards
Saeed Ahmed Tariq


-----Original Message-----
From: freeswitch-users-bounces at lists.freeswitch.org
[mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
Sent: Monday, February 02, 2009 3:33 PM
To: freeswitch-users at lists.freeswitch.org
Subject: Re: [Freeswitch-users] Strange Performance when using as SBC

Hi Saeed,

Here is a first draft of what I did to install FS on my server. 
Configuration are not present, they'll be in a next release :p
http://wiki.freeswitch.org/wiki/SBC_Setup

My aim is to setup FS as a SBC, I hope this page could be a great 
startup point for others. I will update regularly based on what I did.

Saeed, why are you replacing your Nextone, it's said to be one of the 
best commercial SBC on the market.

regards.

Saeed Ahmed wrote:
> Thanks rod for a quick answer,
>
> FS is installed on Ubuntu Server.
>
> I am planning to replace Nextone SBC with FS, Later I'll also use openZAP
to
> communicate with TDM but this all depends how much calls it can take, or
> maybe we can also do something in clustering environment ( I am not sure
> about it). But thanks again and any further help will be highly
appreciated.
>
>
> Kind Regards
> Saeed Ahmed Tariq
>
>
>
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
> Sent: Monday, February 02, 2009 1:53 PM
> To: freeswitch-users at lists.freeswitch.org
> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>
> Hi Saeed,
>
> I just created an account to share my setup on the wiki. I will detail 
> all the steps for a clean install of a debian64 lenny with FS used as a 
> SBC (next step is to try the new LCR module :) )and what I'm doing do 
> stress the server.
>
> I wrote nothing at this time so please be patient, I'm waiting for my 
> new hardware so that I will detail as much as possible what I'll do.
>
> For beginning I suggest you reading the start page on the wiki, 
> especially these pages:
>     -http://wiki.freeswitch.org/wiki/Getting_Started_Guide
>     -http://wiki.freeswitch.org/wiki/Dialplan_XML
>
> maybe you could tell more about the linux distribution you're using so 
> that I can give you some pointers for sipp...
>
> regards.
> rod.
>
>
> Saeed Ahmed wrote:
>   
>> Hi Rod,
>>
>> Could you please share how you configured Sipp & FS to create a test
>> environment? Especially the dial plan, sofia settings etc..., actually I
>>     
> am
>   
>> a newbie. I want to test it on a single FS machine.  
>>
>> Kind Regards
>> Saeed 
>> -----Original Message-----
>> From: freeswitch-users-bounces at lists.freeswitch.org
>> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of rod
>> Sent: Monday, February 02, 2009 11:00 AM
>> To: freeswitch-users at lists.freeswitch.org
>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>>
>> Hi Ken, Jay,
>>
>> thanks for pointing to proxy media, I will test.
>>
>> Ken, you are right, I was brain damaged (a stupid mistake) when setting 
>> INFO cause this kind of level could be very verbose. I'm switching to 
>> CRIT or ERR.
>>
>> Thanks guys,
>> rod.
>>
>> thanks for
>>
>> Ken Rice wrote:
>>   
>>     
>>> If you don't have to transcode, using proxy media mode will still save
>>>       
> you
>   
>>> some CPU time. This is 1/2 way between bypass media and the default
media
>>> interactive mode. The other draw back to this mode is if you are using
FS
>>>     
>>>       
>> to
>>   
>>     
>>> clean up RTP and DTMF you loose those functions but they are not needed
>>>       
> in
>   
>>> most use cases.
>>>
>>> As far as the log level goes, I found that once I had things stable
>>>     
>>>       
>> setting
>>   
>>     
>>> the loglevel to helped a good deal... Info is probably a bit too high of
>>>       
> a
>   
>>> loglevel I would probably go for CRIT or ERR (2 or 1 respectively) if
you
>>> insist on leaving logging turned on... On a busy system these can and
>>>       
> will
>   
>>> generate a good deal of activity (and disk IO if using mod_logfile)
>>>
>>> Ken
>>>
>>>
>>>   
>>>     
>>>       
>>>> From: rod <kawarod at laposte.net>
>>>> Reply-To: <freeswitch-users at lists.freeswitch.org>
>>>> Date: Mon, 02 Feb 2009 11:36:35 +0400
>>>> To: <freeswitch-users at lists.freeswitch.org>
>>>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>>>>
>>>> Hi Ken,
>>>>
>>>> 1) I'd like to use FS to hide topology, so bypass media is not possible
>>>> 2) done
>>>> 3) done
>>>> 4) not used
>>>> 5) i'm using this ins switch.xml -> <param name="loglevel"
>>>> value="info"/>, if you think an other log level is more suitable.
>>>>
>>>> Regarding logging, I can see in console and in the freeswitch.log that
>>>> there is still a lot of NOTICE logging, see below:
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>>> switch_core_session_thread() Session 8721
>>>> (sofia/internal/sipp at 10.10.10.1:5060) Ended
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>>> switch_core_session_thread() Close Channel
>>>> sofia/internal/sipp at 10.10.10.1:5060 [CS_HANGUP]
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>>> switch_core_session_thread() Session 8722
>>>> (sofia/external/9998 at 10.10.20.100) Ended
>>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>>> switch_core_session_thread() Close Channel
>>>> sofia/external/9998 at 10.10.20.100 [CS_HANGUP]
>>>> 2009-02-02 08:33:56 [NOTICE] sofia.c:3164 sofia_handle_sip_i_state()
>>>> Channel [sofia/external/9998 at 10.10.20.100] has been answered
>>>> 2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
>>>> Changing codec ptime to 30. I bet you have a linksys/sipura =D
>>>>
>>>> Do you have any idea where I can switch off this kind of logging. I
>>>> thought it should be in /dialplan/internal.xml, but I see that in
>>>> internal.xml -> <param name="debug" value="0"/>
>>>>
>>>> thanks a lot for your suggestion.
>>>>
>>>> regards,
>>>> rod
>>>>
>>>> Ken Rice wrote:
>>>>     
>>>>       
>>>>         
>>>>> Dont forget there are several things you can do to increase
>>>>>         
>>>>>           
>> performance...
>>   
>>     
>>>>> 1) where possible use bypass media or media proxy modes
>>>>> 2) mount freeswitch/db as a ram drive (if you are using voicemail with
>>>>> the internal FS DBs you'll need a way to make this persistant across
>>>>> reboots)
>>>>> 3) see the wiki for setting reasonable ulimits
>>>>> 4) (this is my oppinion others may vary) dont use mod_cdr_csv
>>>>> 5) turn off (or reduce logging) in switch.conf.xml
>>>>>
>>>>> all of these thing can greatly improve performance.
>>>>>
>>>>> On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod at laposte.net
>>>>> <mailto:kawarod at laposte.net>> wrote:
>>>>>
>>>>>     Thanks Anthony,
>>>>>
>>>>>     the setup is like this:
>>>>>
>>>>>     sipp server ---- FS 1 ---- FS2
>>>>>
>>>>>     FS1 is the AMD CPU that has only one extension in dialplan that
>>>>>     bridges
>>>>>     9999 to FS2. 9999 is the first extension in FS2 dialplan that
>>>>>     plays moh,
>>>>>     FS2 has no CPU pbm.
>>>>>
>>>>>     FS1 is maxing out at 60 bridged calls without your option -hp.
>>>>>
>>>>>     Using -hp, I'm now able to bridge 200 concurrent calls (a great
>>>>>     improvement) and the system is still reactive. CPU load is high
>>>>>     but not
>>>>>     100% and as the system responds well, I think that doesn't matter.
>>>>>         
>>>>>           
>> The
>>   
>>     
>>>>>     2GB of memory are completely consumed (top command shows 700MB for
>>>>>         
>>>>>           
>> FS
>>   
>>     
>>>>>     process).
>>>>>
>>>>>     I understand that FS1 server is not the best hardware platform,
>>>>>     and I'm
>>>>>     waiting for new 4 cores server for testing.
>>>>>     I will update those numbers when testing with the new hardware.
>>>>>
>>>>>     regards,
>>>>>     rod.
>>>>>
>>>>>     Anthony Minessale wrote:
>>>>>       
>>>>>         
>>>>>           
>>>>>> Which of the 2 machines has the load issue? You said it was one box
>>>>>> calling the other.
>>>>>>
>>>>>> You have 2 major things against you, single CPU and AMD, but you
>>>>>> should at least be able to get in the vicinity of 800-1000 calls
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     on a
>>>>>       
>>>>>         
>>>>>           
>>>>>> box like that.
>>>>>>
>>>>>> Are you calling the default 9999?  It's not really an appropriate
>>>>>> extension for load testing.
>>>>>> On the terminating box you should set up a manual extension that is
>>>>>> the first one in the dial plan
>>>>>> to play a wav file from preferably a ram disk or /tmp
>>>>>>
>>>>>> If you do plan on using this in production accept nothing less
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     than a
>>>>>       
>>>>>         
>>>>>           
>>>>>> multi-core intel machine with at least 4 cores, the more cores the
>>>>>> better because that parallel processing is where FS gets it's
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     atvantage.
>>>>>       
>>>>>         
>>>>>           
>>>>>> On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod at laposte.net
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:kawarod at laposte.net>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>> wrote:
>>>>>>
>>>>>>     Dear list,
>>>>>>
>>>>>>     I've been playing with freeswitch for some time (2 months)
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     and the
>>>>>       
>>>>>         
>>>>>           
>>>>>>     fact
>>>>>>     is that I'm very pleased with the functionnalities of this
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     software.
>>>>>       
>>>>>         
>>>>>           
>>>>>>     I'd like to use FS as a SBC handling media and I'm doing some
>>>>>>     tests with
>>>>>>     sipp to load the machine but I'm unable to bridge more than
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     60 calls
>>>>>       
>>>>>         
>>>>>           
>>>>>>     without seeing the CPU being loaded at 100%. I'm sure
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     something is
>>>>>       
>>>>>         
>>>>>           
>>>>>>     going
>>>>>>     wrong with my setup but I'm unable to see what.
>>>>>>
>>>>>>     The test machine has the following specs:
>>>>>>       Athlon XP 3500+ with 2GB of memory (I know this is not a
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     high end
>>>>>       
>>>>>         
>>>>>           
>>>>>>     machine :p)
>>>>>>
>>>>>>     Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
>>>>>>     processor       : 0
>>>>>>     vendor_id       : AuthenticAMD
>>>>>>     cpu family      : 15
>>>>>>     model           : 95
>>>>>>     model name      : AMD Athlon(tm) 64 Processor 3500+
>>>>>>     stepping        : 2
>>>>>>     cpu MHz         : 2199.973
>>>>>>     cache size      : 512 KB
>>>>>>     fpu             : yes
>>>>>>     fpu_exception   : yes
>>>>>>     cpuid level     : 1
>>>>>>     wp              : yes
>>>>>>     flags           : fpu vme de pse tsc msr pae mce cx8 apic
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     sep mtrr pge
>>>>>       
>>>>>         
>>>>>           
>>>>>>     mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
>>>>>>     fxsr_opt
>>>>>>     rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     extapic
>>>>>       
>>>>>         
>>>>>           
>>>>>>     cr8_legacy
>>>>>>     bogomips        : 4402.97
>>>>>>     TLB size        : 1024 4K pages
>>>>>>     clflush size    : 64
>>>>>>     cache_alignment : 64
>>>>>>     address sizes   : 40 bits physical, 48 bits virtual
>>>>>>     power management: ts fid vid ttp tm stc
>>>>>>
>>>>>>     I installed FS on a fresh debian 64:
>>>>>>       Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     UTC 2009
>>>>>       
>>>>>         
>>>>>           
>>>>>>     x86_64 GNU/Linux
>>>>>>
>>>>>>     I set the ulimit parameters like those on the website:
>>>>>>       freeswitch at internal> ...
>>>>>>     Freeswitch:/opt/free-svn/bin# ulimit -a
>>>>>>     core file size          (blocks, -c) unlimited
>>>>>>     data seg size           (kbytes, -d) unlimited
>>>>>>     scheduling priority             (-e) 0
>>>>>>     file size               (blocks, -f) unlimited
>>>>>>     pending signals                 (-i) unlimited
>>>>>>     max locked memory       (kbytes, -l) unlimited
>>>>>>     max memory size         (kbytes, -m) unlimited
>>>>>>     open files                      (-n) 999999
>>>>>>     pipe size            (512 bytes, -p) 8
>>>>>>     POSIX message queues     (bytes, -q) unlimited
>>>>>>     real-time priority              (-r) 0
>>>>>>     stack size              (kbytes, -s) 244
>>>>>>     cpu time               (seconds, -t) unlimited
>>>>>>     max user processes              (-u) unlimited
>>>>>>     virtual memory          (kbytes, -v) unlimited
>>>>>>     file locks                      (-x) unlimited
>>>>>>
>>>>>>
>>>>>>     My network setup is the following:
>>>>>>
>>>>>>       SIPP machine (10.10.10.1/24)----------------vlan
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://10.10.10.1/24%29----------------vlan>
>>>>>       
>>>>>         
>>>>>           
>>>>>>     <http://10.10.10.1/24%29----------------vlan> 55
>>>>>>     ----------(10.10.10.254/24 <http://10.10.10.254/24>
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://10.10.10.254/24>) FS
>>>>>       
>>>>>         
>>>>>           
>>>>>>     (10.10.20.254/24)--------------
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://10.10.20.254/24%29-------------->
>>>>>       
>>>>>         
>>>>>           
>>>>>>     <http://10.10.20.254/24%29--------------> vlan56
>>>>>>     -------------------(10.10.20.100/24 <http://10.10.20.100/24>
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://10.10.20.100/24>)
>>>>>       
>>>>>         
>>>>>           
>>>>>>     OTHER STOCK FS
>>>>>>
>>>>>>
>>>>>>     I launched sipp with:
>>>>>>       sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     10.10.10.1 -i
>>>>>       
>>>>>         
>>>>>           
>>>>>>     10.10.10.1 -mp 25000 10.10.10.254:5060
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://10.10.10.254:5060> <http://10.10.10.254:5060>
>>>>>       
>>>>>         
>>>>>           
>>>>>>     The dialplan on FS is very simple:
>>>>>>     <?xml version="1.0" encoding="utf-8"?>
>>>>>>     <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
>>>>>>     <include>
>>>>>>      <context name="default">
>>>>>>
>>>>>>        <extension name="hold_music_bridge">
>>>>>>         <condition field="destination_number" expression="^9999$">
>>>>>>           <action application="answer"/>
>>>>>>           <action application="bridge"
>>>>>>     data="sofia/external/9999 at 10.10.20.100
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:9999 at 10.10.20.100> <mailto:9999 at 10.10.20.100
>>>>>     <mailto:9999 at 10.10.20.100>>"/>
>>>>>       
>>>>>         
>>>>>           
>>>>>>           </condition>
>>>>>>       </extension>
>>>>>>       </context>
>>>>>>
>>>>>>     </include>
>>>>>>
>>>>>>     FreeSWITCH Version 1.0.trunk (11560M) Started.
>>>>>>     Crash Protection [Disabled]
>>>>>>     Max Sessions[1000]
>>>>>>     Session Rate[100]
>>>>>>     SQL [Enabled]
>>>>>>
>>>>>>
>>>>>>     The test is very simple: sipp dial 9999 that matches in my
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     FS dialplan
>>>>>       
>>>>>         
>>>>>           
>>>>>>     and this is bridged to an other FS machine playing music on
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     hold.
>>>>>       
>>>>>         
>>>>>           
>>>>>>     When I launch "top"  I see after 30 to 40 s that FS consumes all
>>>>>>     the CPU
>>>>>>     ressources (with a mean of 50-60 % before), with 80 calls.
>>>>>>     When I set 70 calls, I have to wait 70-80 s before seeing
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     the same
>>>>>       
>>>>>         
>>>>>           
>>>>>>     issue.
>>>>>>
>>>>>>     Presence is set to false on the 2 profile.
>>>>>>
>>>>>>     I have the same issue with FS 1.0.2 that' s why I tried FS
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     11560.
>>>>>       
>>>>>         
>>>>>           
>>>>>>     When I use the FS machine as a router to test the packet per
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     second
>>>>>       
>>>>>         
>>>>>           
>>>>>>     performance, I'm reaching 100Mbps with 8000pps in each
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     direction (from
>>>>>       
>>>>>         
>>>>>           
>>>>>>     vlan 55 to vlan56) with less than 12% CPU. So that I don't think
>>>>>>     there's
>>>>>>     an issue with the network.
>>>>>>
>>>>>>     Here is an "mpstat -P ALL 1" to show you what's happening
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     suddenly
>>>>>       
>>>>>         
>>>>>           
>>>>>>     with
>>>>>>     70 bridge calls:
>>>>>>     12:31:26     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>>     %steal   %idle    intr/s
>>>>>>     12:31:27     all    3,00    0,00    3,00    0,00    1,00    4,00
>>>>>>     0,00   89,00   6241,00
>>>>>>     12:31:27       0    3,00    0,00    3,00    0,00    1,00    4,00
>>>>>>     0,00   89,00   6241,00
>>>>>>
>>>>>>     12:31:27     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>>     %steal   %idle    intr/s
>>>>>>     12:31:28     all   14,14    0,00   56,57    0,00    2,02    5,05
>>>>>>     0,00   22,22   6035,35
>>>>>>     12:31:28       0   14,14    0,00   56,57    0,00    2,02    5,05
>>>>>>     0,00   22,22   6035,35
>>>>>>
>>>>>>     12:31:28     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>>     %steal   %idle    intr/s
>>>>>>     12:31:29     all   24,75    0,00   67,33    0,00    0,99    6,93
>>>>>>     0,00    0,00   5483,17
>>>>>>     12:31:29       0   24,75    0,00   67,33    0,00    0,99    6,93
>>>>>>     0,00    0,00   5483,17
>>>>>>
>>>>>>
>>>>>>     The CPU is going from 89% idle to 0% in less than 2 seconds.
>>>>>>
>>>>>>     I know that I don't have to expect too much from this kind of
>>>>>>     hardware,
>>>>>>     but it seems strange that the CPU power vanished so suddenly.
>>>>>>
>>>>>>     Thanks a lot for the guys that have read this long mail :p
>>>>>>
>>>>>>     kind regards,
>>>>>>     rod
>>>>>>
>>>>>>
>>>>>>     _______________________________________________
>>>>>>     Freeswitch-users mailing list
>>>>>>     Freeswitch-users at lists.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>>       
>>>>>         
>>>>>           
>>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>>
>>>>>       
>>>>>         
>>>>>           
>>>>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>    
>>>>>>         
>>>>>>           
>>>>>>             
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>   
>>     
>>>>>       
>>>>>         
>>>>>           
>>>>>>     http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> AIM: anthm
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:MSN%253Aanthony_minessale at hotmail.com>>
>>>>>       
>>>>>         
>>>>>           
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:PAYPAL%253Aanthony.minessale at gmail.com>>
>>>>>       
>>>>>         
>>>>>           
>>>>>> IRC: irc.freenode.net <http://irc.freenode.net>
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://irc.freenode.net> #freeswitch
>>>>>       
>>>>>         
>>>>>           
>>>>>> FreeSWITCH Developer Conference
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:sip%3A888 at conference.freeswitch.org>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <mailto:sip%3A888 at conference.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:sip%253A888 at conference.freeswitch.org>>
>>>>>       
>>>>>         
>>>>>           
>>>>>> iax:guest at conference.freeswitch.org/888
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <http://iax:guest@conference.freeswitch.org/888>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <http://iax:guest@conference.freeswitch.org/888>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>>>       
>>>>>         
>>>>>           
>>>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org>>
>>>>>       
>>>>>         
>>>>>           
>>>>>> pstn:213-799-1400
>>>>>>
>>>>>>         
>>>>>>           
>>>>>>             
>> ------------------------------------------------------------------------
>>   
>>     
>>>>>       
>>>>>         
>>>>>           
>>>>>> _______________________________________________
>>>>>> Freeswitch-users mailing list
>>>>>> Freeswitch-users at lists.freeswitch.org
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>>       
>>>>>         
>>>>>           
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>>         
>>>>>>           
>>>>>>             
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>   
>>     
>>>>>       
>>>>>         
>>>>>           
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>         
>>>>>>           
>>>>>>             
>>>>>     _______________________________________________
>>>>>     Freeswitch-users mailing list
>>>>>     Freeswitch-users at lists.freeswitch.org
>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>>         
>>>>>           
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>   
>>     
>>>>>     http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>           
> ------------------------------------------------------------------------
>   
>>>>> _______________________________________________
>>>>> Freeswitch-users mailing list
>>>>> Freeswitch-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>>           
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>   
>>>>> http://www.freeswitch.org
>>>>>   
>>>>>       
>>>>>         
>>>>>           
>>>> _______________________________________________
>>>> Freeswitch-users mailing list
>>>> Freeswitch-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>     
>>>>       
>>>>         
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>   
>>>     
>>>       
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>   
>>     
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>   

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users at lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org




More information about the Freeswitch-users mailing list