On Thu, Aug 27, 2009 at 3:42 AM, Tihomir Culjaga <tculjaga@gmail.com> wrote:

Guys you made a monster!!

so, i moved the machine to 64bit CentOS 5.3... recompiled the latest trunk and did my tests again.

The old tests on 32bit debian 5 on the same hardware shown a CPS rate of 120 as 75 - 80% CPU.... and after some time on that 120 CPS rate the CPU goes to 100% without any chance FS recovers at all.
New tests on 64bit CentOS shown a monster.... 400 CPS rate at 75% CPU.... during the tests FS was really stable and responsive... i placed few calls that went through as a charm :).

Also, i didn't optimize the machine at all ... as it is after CentOS install!.... not even db files are on ramdisk.

What did it really happen? .. did you guys change something in the trunk overnight or it is just moving to CentOS 64bit that boosted drastically?

Here are some details:

кnmonФ12aФФФФФФ[H for help]ФФФHostname=l01sipindir2ФRefresh= 2secs ФФФ19:17.48ФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФП
Г CPU +-------------------------------------------------------------------------+                                                  Г
Г100%-| |                                                                                                                         Г
Г 95%-| |                                                                                                                         Г
Г 90%-| |                                                                                                                         Г
Г 85%-| |                                                                                                                         Г
Г 80%-| | w              s w      www w        s   sw    s    s           s                                                     Г
Г 75%-| |ssssssssssssssssssssswssssssssssssssssssssssssssssssssssssssssssssss                                                     Г
Г 70%-| +sssssssssssssssssssssssssssssssUssssssssUsssUssssssssssUssssssssssss                                                     Г
Г 65%-| |UUUUUsUsUUUUUUUUUUUUUsUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUsUUU                                                     Г
Г 60%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                     Г
Г 55%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                     Г
Г 50%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                     Г
Г 45%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                     Г
Г 40%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUw                                                    Г
Г 35%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUs                                                    Г
Г 30%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г 25%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г 20%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г 15%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г 10%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г 5%-| |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU                                                    Г
Г     +-------------------------------------------------------------------------+                                                  Г
Г CPU Utilisation ФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФГ
Г                           +-------------------------------------------------+                                                    Г
ГCPU User% Sys% Wait% Idle|0          |25         |50          |75       100|                                                    Г
Г 1   1.0   0.5   0.0   98.5|                                      >          |                                                    Г
Г 2   1.5   1.0   0.0   97.5|                                              > |                                                    Г
Г                           +-------------------------------------------------+                                                    Г
ГAvg 1.2   0.5   0.0   98.3|                                       >         |                                                    Г
Г                           +-------------------------------------------------+                                                    Г
Г Disk I/O ФФФФФ(/proc/diskstats)ФФФФФФФФall data is Kbytes per secondФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФГ
ГDiskName Busy Read WriteKB|0          |25         |50          |75       100|                                                    Г
Гiss/c0d0   0%    0.0    0.0|                                             >   |                                                    Г
Гs/c0d0p1   0%    0.0    0.0|>                                                |                                                    Г
Гs/c0d0p2   0%    0.0    0.0|                                              > |                                                    Г
Гdm-0       0%    0.0    0.0|                                              > |                                                    Г
Гdm-1       0%    0.0    0.0|>                                                |                                                    Г
ГФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФФГ

------------------------------ Scenario Screen -------- [1-9]: Change Screen --
Call-rate(length)   Port   Total-time Total-calls Remote-host
40.0(0 ms)/0.100s   5060     488.20 s       193555 10.4.4.252:5060(UDP)

402 new calls during 1.002 s period    0 ms scheduler resolution
3 calls (limit 4000)                   Peak was 53 calls, after 351 s
0 Running, 13216 Paused, 670 Woken up
0 dead call msg (discarded)            0 out-of-call msg (discarded)
3 open sockets

                                 Messages Retrans   Timeout   Unexpected-Msg
      INVITE ----------> B-RTD1 193553    0         0
         100 <---------- E-RTD1 193553    0         0         0
         302 <---------- E-RTD2 193552    0         0         0
         ACK ---------->         193552    0
------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic -----

------------------------------ Scenario Screen -------- [1-9]: Change Screen --
Call-rate(length)   Port   Total-time Total-calls Remote-host
40.0(0 ms)/0.100s   5060     489.10 s       193917 10.4.4.252:5060(UDP)

362 new calls during 0.906 s period    1 ms scheduler resolution
2 calls (limit 4000)                   Peak was 53 calls, after 351 s
0 Running, 13215 Paused, 623 Woken up
0 dead call msg (discarded)            0 out-of-call msg (discarded)
3 open sockets

                                 Messages Retrans   Timeout   Unexpected-Msg
      INVITE ----------> B-RTD1 193917    0         0
         100 <---------- E-RTD1 193917    0         0         0
         302 <---------- E-RTD2 193915    0         0         0
         ACK ---------->         193915    0
------------------------------ Test Terminated --------------------------------

----------------------------- Statistics Screen ------- [1-9]: Change Screen --
Start Time             | 2009-08-26   19:09:34:575    1251306574.575684
Last Reset Time        | 2009-08-26   19:17:42:779    1251307062.779468
Current Time           | 2009-08-26   19:17:43:685    1251307063.685281
-------------------------+---------------------------+--------------------------
Counter Name           | Periodic value            | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time           | 00:00:00:905              | 00:08:09:109
Call Rate              | 400.000 cps              | 396.470 cps
-------------------------+---------------------------+--------------------------
Incoming call created |        0                  |        0
OutGoing call created |      362                  |   193917
Total Call created     |                           |   193917
Current Call           |        2                  |
-------------------------+---------------------------+--------------------------
Successful call        |      363                  |   193915
Failed call            |        0                  |        0
-------------------------+---------------------------+--------------------------
Response Time 1        | 00:00:00:001              | 00:00:00:000
Response Time 2        | 00:00:00:010              | 00:00:00:008
Call Length            | 00:00:00:010              | 00:00:00:008
------------------------------ Test Terminated --------------------------------

...i didn't beleive to SIPp and i went to FS console issuing status command to conferm the results.

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 13 seconds, 703 milliseconds, 971 microseconds
183382 session(s) since startup
1 session(s) 410/800
8000 session(s) max

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 15 seconds, 109 milliseconds, 891 microseconds
183944 session(s) since startup
1 session(s) 401/800
8000 session(s) max

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 16 seconds, 139 milliseconds, 412 microseconds
184356 session(s) since startup
2 session(s) 389/800
8000 session(s) max

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 17 seconds, 62 milliseconds, 16 microseconds
184717 session(s) since startup
6 session(s) 410/800
8000 session(s) max

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 35 seconds, 150 milliseconds, 253 microseconds
191959 session(s) since startup
1 session(s) 400/800
8000 session(s) max

freeswitch@l01sipindir2.ot.hr> status
API CALL [status()] output:
UP 0 years, 0 days, 0 hours, 8 minutes, 36 seconds, 892 milliseconds, 672 microseconds
192657 session(s) since startup
1 session(s) 393/800
8000 session(s) max

On Wed, Aug 26, 2009 at 1:53 PM, Dmitry Kadantsev <kadantsev.d@gmail.com> wrote:

Hi all,

is there same situation with FS for Windows? I mean 64bit is more preferable than 32bit, isn't it?

Any performance test on Win 32/64 were done?

--
Best regards,
Dmitry Kadantsev

On Wed, Aug 26, 2009 at 10:29 AM, Tihomir Culjaga <tculjaga@gmail.com> wrote:
intanto e il centos che si sta installando :)

grazie.

T.

On Wed, Aug 26, 2009 at 10:25 AM, Giovanni Maruzzelli <gmaruzz@celliax.org> wrote:

netbook remix

joking! Server 64bit :-)

-gm

On Wed, Aug 26, 2009 at 10:08 AM, Tihomir Culjaga<tculjaga@gmail.com> wrote:
> Hi Giovanny,
>
> regarding ubuntu, did you mean 8.04 server or desktop ?
>
>
> On Tue, Aug 25, 2009 at 3:41 PM, Giovanni Maruzzelli <gmaruzz@celliax.org>
> wrote:
>>
>> Definitely go for 64 bit OS.
>>
>> If you want to be safe and sure, go for CentOS 5.2 64bit. Is the one
>> used both for development and for heavy duty production.
>>
>> Also Ubuntu 8.04 is good.
>>
>> Other versions/distros are less used by the community.
>>
>> Adding RAM and CPUs helps to scale up.
>>
>> -gm
>>
>>
>>
>> Sincerely,
>>
>> Giovanni Maruzzelli
>> Cell : +39-347-2665618
>>
>>
>>
>>
>> On Tue, Aug 25, 2009 at 3:19 PM, Tihomir Culjaga<tculjaga@gmail.com>
>> wrote:
>> > Hey Giovanni,
>> >
>> > thanks for the tip... indeed the db files were heavily used regardless
>> > if i
>> > started freeswitch with nosql option (freeswitch -nosql)... FS was not
>> > writing anything into that files ... instead it was just accessing
>> > it....
>> > This behaviour leads to a waste of 40% CPU time... waiting for other
>> > processes (mainly disk access) to finish!!!
>> >
>> > I moved freeswitch/db/ to a ramdisk and the performance got a boost to
>> > 140
>> > CPS with a CPU load of 80%. I was keeping the machine for a while (20 -
>> > 30
>> > minutes) on that rate when i sow CPU suddenly went to 100% and FS
>> > becoming
>> > irresponsive :).
>> >
>> >
>> > What can be wrong?
>> > What are the limits in CPU usage (50%, 60%, 70%, 80%...) we should not
>> > cross?
>> > What fine tuning do we need in order to asure a long high load run?
>> >
>> >
>> >
>> > Also, I'm running 32-bit OS (debian 5) on a 64 bit CPU... does it have
>> > sense
>> > to move my OS to 64 bit? ... will FS gain more preformance ?... I mean
>> > will
>> > FS perofomr drastically better 20%+ ?
>> >
>> >
>> > Tihomir.
>> >
>> >
>> > On Tue, Aug 25, 2009 at 11:00 AM, Giovanni Maruzzelli
>> > <gmaruzz@celliax.org>
>> > wrote:
>> >>
>> >> Maybe your load comes from disk access?
>> >>
>> >> Try putting the sql and log directories on a ramdisk.
>> >>
>> >> OTH,
>> >>
>> >> -giovanni
>> >>
>> >> On Tue, Aug 25, 2009 at 10:54 AM, Tihomir Culjaga<tculjaga@gmail.com>
>> >> wrote:
>> >> > Hello,
>> >> >
>> >> > i'm trying to use freeswitch as a redirecting server meaning FS has
>> >> > to
>> >> > receive an INVITE and according to some rules it will redirect calls
>> >> > to
>> >> > other destinations.
>> >> >
>> >> >
>> >> > CALLING_USER                FREESWITCH
>> >> > SOMEWHERE
>> >> >
>> >> > INVITE ------------------------------->
>> >> >            <------------------------------ 100 Trying
>> >> >            <------------------------------ 302 Moved Temporary
>> >> > ACK    ------------------------------->
>> >> >
>> >> >
>> >> > INVITE--------------------------------------------------------------------------------->
>> >> >
>> >> >
>> >> >
>> >> > Well, wverything works well except i have perfromance issues .... on
>> >> > my
>> >> > HW
>> >> > FS cannot do more than 40 CPS (INVITE answered by 302 Moved
>> >> > Temporary).
>> >> > When
>> >> > i increase the rate, FS starts delaying 302 response. Right at 50 CPS
>> >> > i
>> >> > see
>> >> > "calls" being build up in FS and the delay begining to grow.
>> >> >
>> >> > When i observe the machine, load average is almost nothing (load
>> >> > average:
>> >> > 1.41, 0.61, 0.60) CPU never goes to 100%, and i see only one thread
>> >> > taking
>> >> > most load... all others are just sitting there with 1-5 % CPU time.
>> >> > This looks to me as FS handles 302 messages in a single thread?!?!
>> >> >
>> >> >
>> >> > tculjaga@FS:/usr/local/freeswitch/conf/dialplan$ top -H
>> >> >
>> >> > top - 10:41:37 up 167 days, 20:42, 3 users, load average: 1.41,
>> >> > 0.61,
>> >> > 0.60
>> >> > Tasks: 83 total,   2 running, 81 sleeping,   0 stopped,   0 zombie
>> >> > Cpu(s): 25.3%us, 1.5%sy, 0.0%ni, 30.3%id, 42.7%wa, 0.0%hi,
>> >> > 0.2%si,
>> >> > 0.0%st
>> >> > Mem:   2074520k total,   571244k used, 1503276k free,   259604k
>> >> > buffers
>> >> > Swap: 2650684k total,     3020k used, 2647664k free,   153868k
>> >> > cached
>> >> >
>> >> > PID USER      PR NI VIRT RES SHR S %CPU %MEM    TIME+
>> >> > COMMAND
>> >> > 4814 root      20   0 34188 20m 3780 S   38 1.0   3:10.29
>> >> > freeswitch
>> >> > 4800 root      20   0 34188 20m 3780 S    6 1.0   0:08.26
>> >> > freeswitch
>> >> > 4798 root      20   0 34188 20m 3780 R    5 1.0   0:24.46
>> >> > freeswitch
>> >> > 4787 root      20   0 34188 20m 3780 S    2 1.0   0:11.24
>> >> > freeswitch
>> >> > 4794 root      20   0 34188 20m 3780 S    1 1.0   0:11.42
>> >> > freeswitch
>> >> > 4803 root      20   0 34188 20m 3780 S    1 1.0   0:11.74
>> >> > freeswitch
>> >> > 4788 root      20   0 34188 20m 3780 S    1 1.0   0:02.96
>> >> > freeswitch
>> >> > 4804 root      20   0 34188 20m 3780 S    1 1.0   0:01.64
>> >> > freeswitch
>> >> > 4807 root      20   0 34188 20m 3780 S    1 1.0   0:01.68
>> >> > freeswitch
>> >> > 4811 root      20   0 34188 20m 3780 S    1 1.0   0:02.50
>> >> > freeswitch
>> >> >
>> >> >
>> >> >
>> >> > cat /proc/cpuinfo
>> >> > processor       : 0
>> >> > vendor_id       : GenuineIntel
>> >> > cpu family      : 6
>> >> > model           : 15
>> >> > model name      : Intel(R) Xeon(R) CPU            5140 @ 2.33GHz
>> >> > stepping        : 6
>> >> > cpu MHz         : 2333.560
>> >> > cache size      : 4096 KB
>> >> > physical id     : 0
>> >> > siblings        : 2
>> >> > core id         : 0
>> >> > cpu cores       : 2
>> >> > apicid          : 0
>> >> > initial apicid : 0
>> >> > fdiv_bug        : no
>> >> > hlt_bug         : no
>> >> > f00f_bug        : no
>> >> > coma_bug        : no
>> >> > fpu             : yes
>> >> > fpu_exception   : yes
>> >> > cpuid level     : 10
>> >> > wp              : yes
>> >> > flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>> >> > pge
>> >> > mca
>> >> > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
>> >> > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2
>> >> > ssse3
>> >> > cx16
>> >> > xtpr dca lahf_lm
>> >> > bogomips        : 4670.78
>> >> > clflush size    : 64
>> >> > power management:
>> >> >
>> >> > processor       : 1
>> >> > vendor_id       : GenuineIntel
>> >> > cpu family      : 6
>> >> > model           : 15
>> >> > model name      : Intel(R) Xeon(R) CPU            5140 @ 2.33GHz
>> >> > stepping        : 6
>> >> > cpu MHz         : 2333.560
>> >> > cache size      : 4096 KB
>> >> > physical id     : 0
>> >> > siblings        : 2
>> >> > core id         : 1
>> >> > cpu cores       : 2
>> >> > apicid          : 1
>> >> > initial apicid : 1
>> >> > fdiv_bug        : no
>> >> > hlt_bug         : no
>> >> > f00f_bug        : no
>> >> > coma_bug        : no
>> >> > fpu             : yes
>> >> > fpu_exception   : yes
>> >> > cpuid level     : 10
>> >> > wp              : yes
>> >> > flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>> >> > pge
>> >> > mca
>> >> > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
>> >> > constant_tsc arch_perfmon pebs bts pni monitor ds_cpl vmx est tm2
>> >> > ssse3
>> >> > cx16
>> >> > xtpr dca lahf_lm
>> >> > bogomips        : 4666.82
>> >> > clflush size    : 64
>> >> > power management:
>> >> >
>> >> >
>> >> >
>> >> > uname -a
>> >> > Linux l01sipindir1 2.6.26-1-686 #1 SMP Sat Jan 10 18:29:31 UTC 2009
>> >> > i686
>> >> > GNU/Linux
>> >> >
>> >> >
>> >> >
>> >> > Of course, i've tuned the machine up
>> >> >
>> >> > ulimit -c unlimited
>> >> > ulimit -d unlimited
>> >> > ulimit -f unlimited
>> >> > ulimit -i unlimited
>> >> > ulimit -n 999999
>> >> > ulimit -q unlimited
>> >> > ulimit -u unlimited
>> >> > ulimit -v unlimited
>> >> > ulimit -x unlimited
>> >> > ulimit -s 240
>> >> > ulimit -l unlimited
>> >> > ulimit -a
>> >> >
>> >> >
>> >> > Started FS with minimum modules but still 40 CPS seems to be the
>> >> > limit.
>> >> >
>> >> >
>> >> > So, is there any way to improve performance?
>> >> >
>> >> >
>> >> > Tihomir.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > FreeSWITCH-users mailing list
>> >> > FreeSWITCH-users@lists.freeswitch.org
>> >> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >> >
>> >> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >> > http://www.freeswitch.org
>> >> >
>> >> >
>> >>
>> >> _______________________________________________
>> >> FreeSWITCH-users mailing list
>> >> FreeSWITCH-users@lists.freeswitch.org
>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >>
>> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >> http://www.freeswitch.org
>> >
>> >
>> > _______________________________________________
>> > FreeSWITCH-users mailing list
>> > FreeSWITCH-users@lists.freeswitch.org
>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> > http://www.freeswitch.org
>> >
>> >
>>
>> _______________________________________________
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users@lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users@lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org