1) But in your case, cyclictest can generate 336000 (42*8000) hardware timer irqs per second VS 8000 from TDM card's.Very interest results.My example creates a lot of unnecessary interruptionsFor VoIP only host more correctly measured socyclictest -t42 -p -d 80 0 -n -i -l 5000 6000orcyclictest -t42 -p -d 80 0 -n -i -l 10000 3000If FS has one TDM cards thencyclictest -t1 -p -d 80 0 -n -i -l 125 240000If two TDM cardscyclictest -t2 -p -d 80 0 -n -i -l 125 240000Where "-t" key equal number of TDM cards2) According to Digium sources your TDM card have 20ms max buffer size(3ms default)USB dongles have at least 20 ms.Can you describe how you have found this information. It is very interest.SergeyOn Fri, Apr 10, 2015 at 5:06 PM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com> wrote:2015-04-10 0:33 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:1) Most(if not all) of modern equipment have buffers. TDM cards too. Even $0.25 microcontrollers have buffers.Firing interrupts at each PCM sample not needed.The problem is whether there is or not the buffer. Is exist in all the equipment.The problem is the amount of buffer, and through any time it overflows. I thing you cannot say buffer size has TDM card and USB modems. :o)According to Digium sources your TDM card have 20ms max buffer size(3ms default)USB dongles have at least 20 ms.The above i write results of testing Digium E1 card in virtual environment. Test that is successful on real host, will fail in virtual environment.I you see source of timertest utility you will see that Digium test timer for 8000 samples for 1000 ms (1s). (Primary utility author Mark Spencer <markster@digium.com>)It is same value I recommend early.But in your case, cyclictest can generate 336000 (42*8000) hardware timer irqs per second VS 8000 from TDM card's.Look:root@debian:~# vmstat 1procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa0 0 0 2348 85428 21156 0 0 134 0 1856 3409 1 2 92 50 0 0 2320 85428 21156 0 0 0 0 106 224 1 0 99 0[similar, skipped]0 0 0 2312 85436 21156 0 0 0 0 110 231 0 0 100 00 0 0 2312 85436 21156 0 0 0 0 109 225 0 2 98 045 0 0 1924 84908 21156 0 0 0 0 102909 146412 0 60 39 0Starting of cyclictest with your parameters ^^^^^.102909 interrupts and 146412 context switches per second.For one TDM card there should be slightly more then 8000 of interrupts.Results:root@debian:~/rt-tests# ./cyclictest -t42 -p 80 -d 0 -n -i 125 -l 240000 -qWARN: stat /dev/cpu_dma_latency failed: No such file or directoryT: 0 ( 4028) P:80 I:125 C: 239821 Min: 0 Act:100534 Avg:38764 Max: 229794T: 1 ( 4029) P:80 I:125 C: 239820 Min: 0 Act:100536 Avg:39646 Max: 233558T: 2 ( 4030) P:80 I:125 C: 240000 Min: 0 Act:100478 Avg:37947 Max: 229143T: 3 ( 4031) P:80 I:125 C: 240000 Min: 0 Act:106202 Avg:38130 Max: 239031[similar,skipped]T:38 ( 4066) P:80 I:125 C: 239821 Min: 1 Act:100292 Avg:41273 Max: 238424T:39 ( 4067) P:80 I:125 C: 239822 Min: 0 Act:100174 Avg:40254 Max: 238881T:40 ( 4068) P:80 I:125 C: 239816 Min: 0 Act:100650 Avg:39910 Max: 239104T:41 ( 4069) P:80 I:125 C: 239817 Min: 0 Act:100526 Avg:38282 Max: 224452root@debian:~/rt-tests#Increased intervals between threads:root@debian:~/rt-tests# ./cyclictest -t42 -p 80 -d 40 -n -i 125 -l 240000 -qWARN: stat /dev/cpu_dma_latency failed: No such file or directoryT: 0 ( 4203) P:80 I:125 C: 240000 Min: 1 Act: 35 Avg: 28 Max: 159T: 1 ( 4204) P:80 I:165 C: 181909 Min: 6 Act: 47 Avg: 29 Max: 170T: 2 ( 4205) P:80 I:205 C: 146417 Min: 6 Act: 15 Avg: 29 Max: 157T: 3 ( 4206) P:80 I:245 C: 122512 Min: 6 Act: 33 Avg: 30 Max: 166[similar,skipped]T:38 ( 4241) P:80 I:1645 C: 18246 Min: 7 Act: 27 Avg: 34 Max: 167T:39 ( 4242) P:80 I:1685 C: 17813 Min: 8 Act: 13 Avg: 34 Max: 164T:40 ( 4243) P:80 I:1725 C: 17398 Min: 7 Act: 22 Avg: 33 Max: 156T:41 ( 4244) P:80 I:1765 C: 17004 Min: 8 Act: 49 Avg: 34 Max: 141root@debian:~/rt-tests#Huge difference, isn't it?2) If you want massive sync audio processing (w/o buffering) you can use hard realtime OS. PC and Linux (even on bare metal) is not suitable for this, at least without RT patched kernel.I am not understanding about you speaking. E1 card can be used on real host without problems.They use buffering. Delays of 125uS is not uncommon in ordinary Linux kernel. BTW, google "system management interrupt".3) AFAIR, 1 thread per call leg, but depends of endpoints. Most of time threads in I/O waiting/sleeping and encoding/decoding state.One thread per FS call, equal one thread cyclictest. In cyclictest most time threads sleep also. This threads measure time interval and sleep again.In Freeswitch all of threads are not waking up every 125uS synchronously. .And TDM cards are not generating tens/hundreds of thousands interrupts per second as cyclictest can do.IMO, to simulate something like 1 TDM E1 card using cyclictest you should use parameters that give you additional ~8000 irqs.root@debian:~# vmstat 1procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa0 0 0 3496 84304 21156 0 0 45 0 999 562 1 1 97 20 0 0 3496 84304 21156 0 0 0 0 105 222 0 0 100 00 0 0 2784 84304 21156 0 0 0 0 4276 8482 2 15 83 00 0 0 2784 84304 21156 0 0 0 0 8705 17146 2 35 63 00 0 0 2784 84304 21156 0 0 0 0 8678 17099 3 33 64 00 0 0 2784 84304 21156 0 0 0 0 8707 17135 2 37 61 0In my case additional 8000 irqs with these parameters:root@debian:~/rt-tests# ./cyclictest -t 30 -p 80 -d 30 -n -i 3000 -l 24000 -qWARN: stat /dev/cpu_dma_latency failed: No such file or directoryT: 0 ( 5862) P:80 I:3000 C: 24000 Min: 7 Act: 9 Avg: 9 Max: 33T: 1 ( 5863) P:80 I:3030 C: 23767 Min: 6 Act: 17 Avg: 8 Max: 49T: 2 ( 5864) P:80 I:3060 C: 23534 Min: 6 Act: 12 Avg: 8 Max: 45[same,skipped]T:26 ( 5888) P:80 I:3780 C: 19051 Min: 7 Act: 8 Avg: 9 Max: 35T:27 ( 5889) P:80 I:3810 C: 18901 Min: 7 Act: 11 Avg: 8 Max: 40T:28 ( 5890) P:80 I:3840 C: 18754 Min: 7 Act: 14 Avg: 8 Max: 39T:29 ( 5891) P:80 I:3870 C: 18608 Min: 7 Act: 14 Avg: 8 Max: 366) In your case, most capable realtime hypervisor with RT_PREEMPT guest would be one of the best solution.May be. According this publication (http://www.linux-kvm.org/wiki/images/0/03/KVM-Forum-2011-RT-KVM.pdf) guest threads can be interrupted not more 112 uS.Simple solution - for TDM card enough ordinary processor without virtualization. :o)This is a best solution.On Thu, Apr 9, 2015 at 8:09 PM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com> wrote:2015-04-08 15:25 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:1) 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,But here we are talking about packet telephony where commonly used frame length is 20000 uS.So, intervals for testing should be 5000-10000-20000uS, not 125uS.Not all FS installation has SIP endpoint. Some can have TDM cards, USB dongle and other analog equipment. Can you say what is length of this frame type?I am wondering about this question. I know that the frame 125ms is enough.Most(if not all) of modern equipment have buffers. TDM cards too. Even $0.25 microcontrollers have buffers.Firing interrupts at each PCM sample not needed.If you want massive sync audio processing (w/o buffering) you can use hard realtime OS. PC and Linux (even on bare metal) is not suitable for this, at least without RT patched kernel.2) Why you decided that all of 42 threads should wake up simultaneously?That totally impossible in real world applications.Here should be tens of uS. Dependent of network card and, possibly, other I/O hardware.Can you say how many FS create threads for 40-50 active calls?AFAIR, 1 thread per call leg, but depends of endpoints. Most of time threads in I/O waiting/sleeping and encoding/decoding state.3) -q switch is missingI am thinking it is optional. Result is sameDynamic output to screen consumes resources. How much, depends on method.(VGA, Serial console, SSH). In my case difference is significiant. So, fair testing should exclude such things, IMO.4) Precise time measurement and realtime processing in VM environment is a challenge.To obtain acceptable results fine tuning of host's and guest's required.WIthout that this numbers ... does not mean anything.Can you learn me how to correctly measure timer source quality in VM?Serious players, as Wind River, use external timer.In your case, most capable realtime hypervisor with RT_PREEMPT guest would be one of the best solution.Dmitry.On Wed, Apr 8, 2015 at 2:07 PM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com> wrote:2015-04-07 10:48 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:...2) i 125 - time us between software timer interrupts for first thread. It time is equal time between samples rate in PSTN networksIn order to maintain our sampling rate we have 125 microseconds (uS) between consecutive samples (1/8,000Hz), which is more...125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,But here we are talking about packet telephony where commonly used frame length is 20000 uS.So, intervals for testing should be 5000-10000-20000uS, not 125uS.3) d 0 - this switch will make time between software timer interrupts for second and following threads is equal timer value for first threadWhy you decided that all of 42 threads should wake up simultaneously?That totally impossible in real world applications.Here should be tens of uS. Dependent of network card and, possibly, other I/O hardware.4) l 240000 - how many timer test must be made. It is adjusted for 30 s overall test5) n - use clock_nanosleep. it is allow take more accurate timer test values on my VM6) p 80 - it is copied from example on Cyclictest page-q switch is missing.And how in your opinion this numbers are related to Freeswitch in VM environment?Relation between Cyclictest switches and FS described above.Relation betwen Cyclictest results and FS simplePrecise time measurement and realtime processing in VM environment is a challenge.To obtain acceptable results fine tuning of host's and guest's required.WIthout that this numbers ... does not mean anything.On Tue, Apr 7, 2015 at 5:38 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com> wrote:Sergey,Why you suggest to use exactly this numbers in cyclictest's switches?And how in your opinion this numbers are related to Freeswitch in VM environment?2015-04-06 15:27 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:Errol can you test VM timer with utility https://rt.wiki.kernel.org/index.php/CyclictestMy VM currently have folowing results[root@fs1 ~]# date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i 125 -l 240000 && date Sun Apr 5 07:01:31 GMT 2015 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574 T: 0 ( 1531) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 16 Max: 1302 T: 1 ( 1532) P:80 I:125 C: 240000 Min: 2 Act: 15 Avg: 14 Max: 2895 T: 2 ( 1533) P:80 I:125 C: 240000 Min: 3 Act: 10 Avg: 12 Max: 2888 T: 3 ( 1534) P:80 I:125 C: 240000 Min: 1 Act: 16 Avg: 13 Max: 1333 T: 4 ( 1535) P:80 I:125 C: 240000 Min: 2 Act: 20 Avg: 14 Max: 2962 T: 5 ( 1536) P:80 I:125 C: 240000 Min: 1 Act: 12 Avg: 14 Max: 2946 T: 6 ( 1537) P:80 I:125 C: 240000 Min: 2 Act: 10 Avg: 13 Max: 1363 T: 7 ( 1538) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 13 Max: 1344 T: 8 ( 1539) P:80 I:125 C: 240000 Min: 3 Act: 12 Avg: 16 Max: 1408 T: 9 ( 1540) P:80 I:125 C: 240000 Min: 1 Act: 8 Avg: 13 Max: 2882 T:10 ( 1541) P:80 I:125 C: 239974 Min: 2 Act: 12 Avg: 13 Max: 2908On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <ewsamuels@gmail.com> wrote:With regards to Xen was this at Linode or inhouse?Just out of curiosity can you tell me the specs of the VM that you tested with and which DO Datacenter? I am using NYC3 so perhaps and I will do some tests as soon as I have some time.Good job and Thanks for sharing!As someone who has been using KVM for a few years and has just migrated about 20 VMs across to Digital Ocean about 6 weeks ago I admit I am a little concerned about this finding.ThanksErrolOn Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:here we go:
https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/
On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
> On Sat, Apr 4, 2015 at 1:03 PM, Regis M <regis.freeswitch.org@tornad.net> wrote:
>> How do you check wavs together in audacity ?
>
> you just open two WAV files, align their starting position, and
> visually check the waves. If there's a problem, you would see that
> after few seconds, the waves are not aligned any more.
>
>> Can you send me 2 samples wav (send... and recorded) so I could see the 20ms
>> diff that you exposed...
>>
>> Is there a way to automate the result checking, so it could help to monitor
>> itsp quality
>
> Here
> https://github.com/openpreserve/scape-xcorrsound/issues/17
> this is a link to a tool which compares two sound files at the wave
> level. You can also see the sound samples in the ticket.
> The problem is, that currently this tool does not detect such slipping frames.
>
> I will also write a detailed article with examples, during the weekend.
>
>
>
>
>
>>
>>
>> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin <ssinyagin@gmail.com>:
>>>
>>> Important is to send the media through two independent carriers - then you
>>> can be sure that your clock is compared against a high quality one.
>>>
>>> On Apr 4, 2015 12:46 PM, "Regis M" <regis.freeswitch.org@tornad.net>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> It's a very interresting test and conclusion.
>>>>
>>>> I will try to build a similar environnement on my freeswitch to check
>>>> some system :)
>>>>
>>>> thanks
>>>>
>>>>
>>>>
>>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin <ssinyagin@gmail.com>:
>>>>>
>>>>> hi,
>>>>>
>>>>> Here are some test results which might be interesting:
>>>>>
>>>>> I've set up a test FreeSWITCH server at digitalocean (they use KVM),
>>>>> and originated test calls to some ITSP accounts. Each call played a
>>>>> test sound file (voice samples from ITU:
>>>>> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050
>>>>> ), and on the receiving end another FreeSWITCH server was recording
>>>>> the received session. The receiving server was on a XEN VM. Both
>>>>> servers talked G711 to their corresponding ITSP accounts.
>>>>>
>>>>> Then I compared the source audio and received audio in Audacity editor.
>>>>>
>>>>> The tests showed that the received voice wave shifts 20ms from the
>>>>> original sound every few seconds, randomly back and forward. This
>>>>> skewing was always present, regardless of ITSP and time of day.
>>>>>
>>>>> Then I moved the sending server to a XEN VM, and the waves appeared
>>>>> almost perfectly synchronized (there was a 5ms skew in one test, and I
>>>>> guess it's related to transcoding buffer, as the sound was obviously
>>>>> transcoded).
>>>>>
>>>>> The effect was always visible with a 30-second sample, and I also made
>>>>> additional tests with 2-minute samples.
>>>>>
>>>>> It would also be interesting to test this from a VmWare VM. If someone
>>>>> wants to run such a test, feel free to contact me directly.
>>>>>
>>>>>
>>>>> cheers,
>>>>> stan
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting@freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users@lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting@freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users@lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting@freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users@lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting@freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users@lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com
Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org