[Freeswitch-users] Freeswitch performance as a redirecting server
Tihomir Culjaga
tculjaga at gmail.com
Wed Aug 26 10:42:19 PDT 2009
Guys you made a monster!!
so, i moved the machine to 64bit CentOS 5.3... recompiled the latest trunk
and did my tests again.
The old tests on 32bit debian 5 on the same hardware shown a CPS rate of 120
as 75 - 80% CPU.... and after some time on that 120 CPS rate the CPU goes to
100% without any chance FS recovers at all.
New tests on 64bit CentOS shown a monster.... 400 CPS rate at 75% CPU....
during the tests FS was really stable and responsive... i placed few calls
that went through as a charm :).
Also, i didn't optimize the machine at all ... as it is after CentOS
install!.... not even db files are on ramdisk.
What did it really happen? .. did you guys change something in the trunk
overnight or it is just moving to CentOS 64bit that boosted drastically?
Here are some details:
ÚnmonÄ12aÄÄÄÄÄÄ[H for help]ÄÄÄHostname=l01sipindir2ÄRefresh= 2secs
ÄÄÄ19:17.48ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
³ CPU
+-------------------------------------------------------------------------+
³
³100%-|
|
³
³ 95%-|
|
³
³ 90%-|
|
³
³ 85%-|
|
³
³ 80%-| | w s w www w s sw s s
s ³
³ 75%-|
|ssssssssssssssssssssswssssssssssssssssssssssssssssssssssssssssssssss
³
³ 70%-|
+sssssssssssssssssssssssssssssssUssssssssUsssUssssssssssUssssssssssss
³
³ 65%-|
|UUUUUsUsUUUUUUUUUUUUUsUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUsUUU
³
³ 60%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 55%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 50%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 45%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 40%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUw
³
³ 35%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs
³
³ 30%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 25%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 20%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 15%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 10%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³ 5%-|
|UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
³
³
+-------------------------------------------------------------------------+
³
³ CPU Utilisation
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄij
³
+-------------------------------------------------+
³
³CPU User% Sys% Wait% Idle|0 |25 |50 |75
100| ³
³ 1 1.0 0.5 0.0 98.5|
> | ³
³ 2 1.5 1.0 0.0 97.5|
> | ³
³
+-------------------------------------------------+
³
³Avg 1.2 0.5 0.0 98.3|
> | ³
³
+-------------------------------------------------+
³
³ Disk I/O ÄÄÄÄÄ(/proc/diskstats)ÄÄÄÄÄÄÄÄall data is Kbytes per
secondÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄij
³DiskName Busy Read WriteKB|0 |25 |50 |75
100| ³
³iss/c0d0 0% 0.0 0.0|
> | ³
³s/c0d0p1 0% 0.0
0.0|>
| ³
³s/c0d0p2 0% 0.0 0.0|
> | ³
³dm-0 0% 0.0 0.0|
> | ³
³dm-1 0% 0.0
0.0|>
| ³
³ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄij
------------------------------ Scenario Screen -------- [1-9]: Change Screen
--
Call-rate(length) Port Total-time Total-calls Remote-host
40.0(0 ms)/0.100s 5060 488.20 s 193555 10.4.4.252:5060(UDP)
402 new calls during 1.002 s period 0 ms scheduler resolution
3 calls (limit 4000) Peak was 53 calls, after 351 s
0 Running, 13216 Paused, 670 Woken up
0 dead call msg (discarded) 0 out-of-call msg
(discarded)
3 open sockets
Messages Retrans Timeout
Unexpected-Msg
INVITE ----------> B-RTD1 193553 0 0
100 <---------- E-RTD1 193553 0 0 0
302 <---------- E-RTD2 193552 0 0 0
ACK ----------> 193552 0
------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic
-----
------------------------------ Scenario Screen -------- [1-9]: Change Screen
--
Call-rate(length) Port Total-time Total-calls Remote-host
40.0(0 ms)/0.100s 5060 489.10 s 193917 10.4.4.252:5060(UDP)
362 new calls during 0.906 s period 1 ms scheduler resolution
2 calls (limit 4000) Peak was 53 calls, after 351 s
0 Running, 13215 Paused, 623 Woken up
0 dead call msg (discarded) 0 out-of-call msg
(discarded)
3 open sockets
Messages Retrans Timeout
Unexpected-Msg
INVITE ----------> B-RTD1 193917 0 0
100 <---------- E-RTD1 193917 0 0 0
302 <---------- E-RTD2 193915 0 0 0
ACK ----------> 193915 0
------------------------------ Test Terminated
--------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen
--
Start Time | 2009-08-26 19:09:34:575
1251306574.575684
Last Reset Time | 2009-08-26 19:17:42:779
1251307062.779468
Current Time | 2009-08-26 19:17:43:685
1251307063.685281
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:905 |
00:08:09:109
Call Rate | 400.000 cps | 396.470
cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 |
0
OutGoing call created | 362 |
193917
Total Call created | |
193917
Current Call | 2
|
-------------------------+---------------------------+--------------------------
Successful call | 363 |
193915
Failed call | 0 |
0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:001 |
00:00:00:000
Response Time 2 | 00:00:00:010 |
00:00:00:008
Call Length | 00:00:00:010 |
00:00:00:008
------------------------------ Test Terminated
--------------------------------
...i didn't beleive to SIPp and i went to FS console issuing status command
to conferm the results.
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 13 seconds, 703 milliseconds, 971
microseconds
183382 session(s) since startup
1 session(s) 410/800
8000 session(s) max
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 15 seconds, 109 milliseconds, 891
microseconds
183944 session(s) since startup
1 session(s) 401/800
8000 session(s) max
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 16 seconds, 139 milliseconds, 412
microseconds
184356 session(s) since startup
2 session(s) 389/800
8000 session(s) max
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 17 seconds, 62 milliseconds, 16
microseconds
184717 session(s) since startup
6 session(s) 410/800
8000 session(s) max
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 35 seconds, 150 milliseconds, 253
microseconds
191959 session(s) since startup
1 session(s) 400/800
8000 session(s) max
freeswitch at l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 36 seconds, 892 milliseconds, 672
microseconds
192657 session(s) since startup
1 session(s) 393/800
8000 session(s) max
On Wed, Aug 26, 2009 at 1:53 PM, Dmitry Kadantsev <kadantsev.d at gmail.com>wrote:
> Hi all,
>
> is there same situation with FS for Windows? I mean 64bit is more
> preferable than 32bit, isn't it?
>
> Any performance test on Win 32/64 were done?
>
> --
> Best regards,
> Dmitry Kadantsev
>
>
>
> On Wed, Aug 26, 2009 at 10:29 AM, Tihomir Culjaga <tculjaga at gmail.com>wrote:
>
>> intanto e il centos che si sta installando :)
>>
>> grazie.
>>
>> T.
>>
>>
>> On Wed, Aug 26, 2009 at 10:25 AM, Giovanni Maruzzelli <
>> gmaruzz at celliax.org> wrote:
>>
>>> netbook remix
>>>
>>>
>>> joking! Server 64bit :-)
>>>
>>> -gm
>>>
>>>
>>>
>>> On Wed, Aug 26, 2009 at 10:08 AM, Tihomir Culjaga<tculjaga at gmail.com>
>>> wrote:
>>> > Hi Giovanny,
>>> >
>>> > regarding ubuntu, did you mean 8.04 server or desktop ?
>>> >
>>> >
>>> > On Tue, Aug 25, 2009 at 3:41 PM, Giovanni Maruzzelli <
>>> gmaruzz at celliax.org>
>>> > wrote:
>>> >>
>>> >> Definitely go for 64 bit OS.
>>> >>
>>> >> If you want to be safe and sure, go for CentOS 5.2 64bit. Is the one
>>> >> used both for development and for heavy duty production.
>>> >>
>>> >> Also Ubuntu 8.04 is good.
>>> >>
>>> >> Other versions/distros are less used by the community.
>>> >>
>>> >> Adding RAM and CPUs helps to scale up.
>>> >>
>>> >> -gm
>>> >>
>>> >>
>>> >>
>>> >> Sincerely,
>>> >>
>>> >> Giovanni Maruzzelli
>>> >> Cell : +39-347-2665618
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Aug 25, 2009 at 3:19 PM, Tihomir Culjaga<tculjaga at gmail.com>
>>> >> wrote:
>>> >> > Hey Giovanni,
>>> >> >
>>> >> > thanks for the tip... indeed the db files were heavily used
>>> regardless
>>> >> > if i
>>> >> > started freeswitch with nosql option (freeswitch -nosql)... FS was
>>> not
>>> >> > writing anything into that files ... instead it was just accessing
>>> >> > it....
>>> >> > This behaviour leads to a waste of 40% CPU time... waiting for other
>>> >> > processes (mainly disk access) to finish!!!
>>> >> >
>>> >> > I moved freeswitch/db/ to a ramdisk and the performance got a boost
>>> to
>>> >> > 140
>>> >> > CPS with a CPU load of 80%. I was keeping the machine for a while
>>> (20 -
>>> >> > 30
>>> >> > minutes) on that rate when i sow CPU suddenly went to 100% and FS
>>> >> > becoming
>>> >> > irresponsive :).
>>> >> >
>>> >> >
>>> >> > What can be wrong?
>>> >> > What are the limits in CPU usage (50%, 60%, 70%, 80%...) we should
>>> not
>>> >> > cross?
>>> >> > What fine tuning do we need in order to asure a long high load run?
>>> >> >
>>> >> >
>>> >> >
>>> >> > Also, I'm running 32-bit OS (debian 5) on a 64 bit CPU... does it
>>> have
>>> >> > sense
>>> >> > to move my OS to 64 bit? ... will FS gain more preformance ?... I
>>> mean
>>> >> > will
>>> >> > FS perofomr drastically better 20%+ ?
>>> >> >
>>> >> >
>>> >> > Tihomir.
>>> >> >
>>> >> >
>>> >> > On Tue, Aug 25, 2009 at 11:00 AM, Giovanni Maruzzelli
>>> >> > <gmaruzz at celliax.org>
>>> >> > wrote:
>>> >> >>
>>> >> >> Maybe your load comes from disk access?
>>> >> >>
>>> >> >> Try putting the sql and log directories on a ramdisk.
>>> >> >>
>>> >> >> OTH,
>>> >> >>
>>> >> >> -giovanni
>>> >> >>
>>> >> >> On Tue, Aug 25, 2009 at 10:54 AM, Tihomir Culjaga<
>>> tculjaga at gmail.com>
>>> >> >> wrote:
>>> >> >> > Hello,
>>> >> >> >
>>> >> >> > i'm trying to use freeswitch as a redirecting server meaning FS
>>> has
>>> >> >> > to
>>> >> >> > receive an INVITE and according to some rules it will redirect
>>> calls
>>> >> >> > to
>>> >> >> > other destinations.
>>> >> >> >
>>> >> >> >
>>> >> >> > CALLING_USER FREESWITCH
>>> >> >> > SOMEWHERE
>>> >> >> >
>>> >> >> > INVITE ------------------------------->
>>> >> >> > <------------------------------ 100 Trying
>>> >> >> > <------------------------------ 302 Moved Temporary
>>> >> >> > ACK ------------------------------->
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> INVITE--------------------------------------------------------------------------------->
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > Well, wverything works well except i have perfromance issues ....
>>> on
>>> >> >> > my
>>> >> >> > HW
>>> >> >> > FS cannot do more than 40 CPS (INVITE answered by 302 Moved
>>> >> >> > Temporary).
>>> >> >> > When
>>> >> >> > i increase the rate, FS starts delaying 302 response. Right at 50
>>> CPS
>>> >> >> > i
>>> >> >> > see
>>> >> >> > "calls" being build up in FS and the delay begining to grow.
>>> >> >> >
>>> >> >> > When i observe the machine, load average is almost nothing (load
>>> >> >> > average:
>>> >> >> > 1.41, 0.61, 0.60) CPU never goes to 100%, and i see only one
>>> thread
>>> >> >> > taking
>>> >> >> > most load... all others are just sitting there with 1-5 % CPU
>>> time.
>>> >> >> > This looks to me as FS handles 302 messages in a single
>>> thread?!?!
>>> >> >> >
>>> >> >> >
>>> >> >> > tculjaga at FS:/usr/local/freeswitch/conf/dialplan$ top -H
>>> >> >> >
>>> >> >> > top - 10:41:37 up 167 days, 20:42, 3 users, load average: 1.41,
>>> >> >> > 0.61,
>>> >> >> > 0.60
>>> >> >> > Tasks: 83 total, 2 running, 81 sleeping, 0 stopped, 0
>>> zombie
>>> >> >> > Cpu(s): 25.3%us, 1.5%sy, 0.0%ni, 30.3%id, 42.7%wa, 0.0%hi,
>>> >> >> > 0.2%si,
>>> >> >> > 0.0%st
>>> >> >> > Mem: 2074520k total, 571244k used, 1503276k free, 259604k
>>> >> >> > buffers
>>> >> >> > Swap: 2650684k total, 3020k used, 2647664k free, 153868k
>>> >> >> > cached
>>> >> >> >
>>> >> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>>> >> >> > COMMAND
>>> >> >> > 4814 root 20 0 34188 20m 3780 S 38 1.0 3:10.29
>>> >> >> > freeswitch
>>> >> >> > 4800 root 20 0 34188 20m 3780 S 6 1.0 0:08.26
>>> >> >> > freeswitch
>>> >> >> > 4798 root 20 0 34188 20m 3780 R 5 1.0 0:24.46
>>> >> >> > freeswitch
>>> >> >> > 4787 root 20 0 34188 20m 3780 S 2 1.0 0:11.24
>>> >> >> > freeswitch
>>> >> >> > 4794 root 20 0 34188 20m 3780 S 1 1.0 0:11.42
>>> >> >> > freeswitch
>>> >> >> > 4803 root 20 0 34188 20m 3780 S 1 1.0 0:11.74
>>> >> >> > freeswitch
>>> >> >> > 4788 root 20 0 34188 20m 3780 S 1 1.0 0:02.96
>>> >> >> > freeswitch
>>> >> >> > 4804 root 20 0 34188 20m 3780 S 1 1.0 0:01.64
>>> >> >> > freeswitch
>>> >> >> > 4807 root 20 0 34188 20m 3780 S 1 1.0 0:01.68
>>> >> >> > freeswitch
>>> >> >> > 4811 root 20 0 34188 20m 3780 S 1 1.0 0:02.50
>>> >> >> > freeswitch
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > cat /proc/cpuinfo
>>> >> >> > processor : 0
>>> >> >> > vendor_id : GenuineIntel
>>> >> >> > cpu family : 6
>>> >> >> > model : 15
>>> >> >> > model name : Intel(R) Xeon(R) CPU 5140 @ 2.33GHz
>>> >> >> > stepping : 6
>>> >> >> > cpu MHz : 2333.560
>>> >> >> > cache size : 4096 KB
>>> >> >> > physical id : 0
>>> >> >> > siblings : 2
>>> >> >> > core id : 0
>>> >> >> > cpu cores : 2
>>> >> >> > apicid : 0
>>> >> >> > initial apicid : 0
>>> >> >> > fdiv_bug : no
>>> >> >> > hlt_bug : no
>>> >> >> > f00f_bug : no
>>> >> >> > coma_bug : no
>>> >> >> > fpu : yes
>>> >> >> > fpu_exception : yes
>>> >> >> > cpuid level : 10
>>> >> >> > wp : yes
>>> >> >> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep
>>> mtrr
>>> >> >> > pge
>>> >> >> > mca
>>> >> >> > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
>>> >> >> > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2
>>> >> >> > ssse3
>>> >> >> > cx16
>>> >> >> > xtpr dca lahf_lm
>>> >> >> > bogomips : 4670.78
>>> >> >> > clflush size : 64
>>> >> >> > power management:
>>> >> >> >
>>> >> >> > processor : 1
>>> >> >> > vendor_id : GenuineIntel
>>> >> >> > cpu family : 6
>>> >> >> > model : 15
>>> >> >> > model name : Intel(R) Xeon(R) CPU 5140 @ 2.33GHz
>>> >> >> > stepping : 6
>>> >> >> > cpu MHz : 2333.560
>>> >> >> > cache size : 4096 KB
>>> >> >> > physical id : 0
>>> >> >> > siblings : 2
>>> >> >> > core id : 1
>>> >> >> > cpu cores : 2
>>> >> >> > apicid : 1
>>> >> >> > initial apicid : 1
>>> >> >> > fdiv_bug : no
>>> >> >> > hlt_bug : no
>>> >> >> > f00f_bug : no
>>> >> >> > coma_bug : no
>>> >> >> > fpu : yes
>>> >> >> > fpu_exception : yes
>>> >> >> > cpuid level : 10
>>> >> >> > wp : yes
>>> >> >> > flags : fpu vme de pse tsc msr pae mce cx8 apic sep
>>> mtrr
>>> >> >> > pge
>>> >> >> > mca
>>> >> >> > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
>>> >> >> > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2
>>> >> >> > ssse3
>>> >> >> > cx16
>>> >> >> > xtpr dca lahf_lm
>>> >> >> > bogomips : 4666.82
>>> >> >> > clflush size : 64
>>> >> >> > power management:
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > uname -a
>>> >> >> > Linux l01sipindir1 2.6.26-1-686 #1 SMP Sat Jan 10 18:29:31 UTC
>>> 2009
>>> >> >> > i686
>>> >> >> > GNU/Linux
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > Of course, i've tuned the machine up
>>> >> >> >
>>> >> >> > ulimit -c unlimited
>>> >> >> > ulimit -d unlimited
>>> >> >> > ulimit -f unlimited
>>> >> >> > ulimit -i unlimited
>>> >> >> > ulimit -n 999999
>>> >> >> > ulimit -q unlimited
>>> >> >> > ulimit -u unlimited
>>> >> >> > ulimit -v unlimited
>>> >> >> > ulimit -x unlimited
>>> >> >> > ulimit -s 240
>>> >> >> > ulimit -l unlimited
>>> >> >> > ulimit -a
>>> >> >> >
>>> >> >> >
>>> >> >> > Started FS with minimum modules but still 40 CPS seems to be the
>>> >> >> > limit.
>>> >> >> >
>>> >> >> >
>>> >> >> > So, is there any way to improve performance?
>>> >> >> >
>>> >> >> >
>>> >> >> > Tihomir.
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > FreeSWITCH-users mailing list
>>> >> >> > FreeSWITCH-users at lists.freeswitch.org
>>> >> >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >> >> >
>>> >> >> > UNSUBSCRIBE:
>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >> >> > http://www.freeswitch.org
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> FreeSWITCH-users mailing list
>>> >> >> FreeSWITCH-users at lists.freeswitch.org
>>> >> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >> >>
>>> >> >> UNSUBSCRIBE:
>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >> >> http://www.freeswitch.org
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > FreeSWITCH-users mailing list
>>> >> > FreeSWITCH-users at lists.freeswitch.org
>>> >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >> > UNSUBSCRIBE:
>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >> > http://www.freeswitch.org
>>> >> >
>>> >> >
>>> >>
>>> >> _______________________________________________
>>> >> FreeSWITCH-users mailing list
>>> >> FreeSWITCH-users at lists.freeswitch.org
>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >> UNSUBSCRIBE:
>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >> http://www.freeswitch.org
>>> >
>>> >
>>> > _______________________________________________
>>> > FreeSWITCH-users mailing list
>>> > FreeSWITCH-users at lists.freeswitch.org
>>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> > UNSUBSCRIBE:
>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> > http://www.freeswitch.org
>>> >
>>> >
>>>
>>> _______________________________________________
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _______________________________________________
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20090826/c8f8501f/attachment-0002.html
More information about the FreeSWITCH-users
mailing list