Key "-q" is recommended also. ;o) On Sun, Apr 12, 2015 at 9:01 PM, Sergey Safarov <s.safarov@gmail.com> wrote:
1) But in your case, cyclictest can generate 336000 (42*8000) hardware timer irqs per second VS 8000 from TDM card's.
Very interest results. My example creates a lot of unnecessary interruptions For VoIP only host more correctly measured so cyclictest -t42 -p -d 80 0 -n -i -l 5000 6000 or cyclictest -t42 -p -d 80 0 -n -i -l 10000 3000
If FS has one TDM cards then cyclictest -t1 -p -d 80 0 -n -i -l 125 240000
If two TDM cards cyclictest -t2 -p -d 80 0 -n -i -l 125 240000
Where "-t" key equal number of TDM cards
2) According to Digium sources your TDM card have 20ms max buffer size(3ms default) USB dongles have at least 20 ms.
Can you describe how you have found this information. It is very interest.
Sergey
On Fri, Apr 10, 2015 at 5:06 PM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
2015-04-10 0:33 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:
1) Most(if not all) of modern equipment have buffers. TDM cards too. Even $0.25 microcontrollers have buffers. Firing interrupts at each PCM sample not needed.
The problem is whether there is or not the buffer. Is exist in all the equipment. The problem is the amount of buffer, and through any time it overflows. I thing you cannot say buffer size has TDM card and USB modems. :o)
According to Digium sources your TDM card have 20ms max buffer size(3ms default) USB dongles have at least 20 ms.
The above i write results of testing Digium E1 card in virtual environment. Test that is successful on real host, will fail in virtual environment. I you see source of timertest utility you will see that Digium test timer for 8000 samples for 1000 ms (1s). (Primary utility author Mark Spencer <markster@digium.com>) It is same value I recommend early.
But in your case, cyclictest can generate 336000 (42*8000) hardware timer irqs per second VS 8000 from TDM card's.
Look: root@debian:~# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 2348 85428 21156 0 0 134 0 1856 3409 1 2 92 5 0 0 0 2320 85428 21156 0 0 0 0 106 224 1 0 99 0 [similar, skipped] 0 0 0 2312 85436 21156 0 0 0 0 110 231 0 0 100 0 0 0 0 2312 85436 21156 0 0 0 0 109 225 0 2 98 0 45 0 0 1924 84908 21156 0 0 0 0 102909 146412 0 60 39 0 Starting of cyclictest with your parameters ^^^^^.
102909 interrupts and 146412 context switches per second. For one TDM card there should be slightly more then 8000 of interrupts.
Results:
root@debian:~/rt-tests# ./cyclictest -t42 -p 80 *-d 0* -n -i 125 -l 240000 -q WARN: stat /dev/cpu_dma_latency failed: No such file or directory T: 0 ( 4028) P:80 I:125 C: 239821 Min: 0 Act:100534 Avg:38764 Max: 229794 T: 1 ( 4029) P:80 I:125 C: 239820 Min: 0 Act:100536 Avg:39646 Max: 233558 T: 2 ( 4030) P:80 I:125 C: 240000 Min: 0 Act:100478 Avg:37947 Max: 229143 T: 3 ( 4031) P:80 I:125 C: 240000 Min: 0 Act:106202 Avg:38130 Max: 239031 [similar,skipped] T:38 ( 4066) P:80 I:125 C: 239821 Min: 1 Act:100292 Avg:41273 Max: 238424 T:39 ( 4067) P:80 I:125 C: 239822 Min: 0 Act:100174 Avg:40254 Max: 238881 T:40 ( 4068) P:80 I:125 C: 239816 Min: 0 Act:100650 Avg:39910 Max: 239104 T:41 ( 4069) P:80 I:125 C: 239817 Min: 0 Act:100526 Avg:38282 Max: 224452 root@debian:~/rt-tests#
Increased intervals between threads:
root@debian:~/rt-tests# ./cyclictest -t42 -p 80 *-d 40* -n -i 125 -l 240000 -q WARN: stat /dev/cpu_dma_latency failed: No such file or directory T: 0 ( 4203) P:80 I:125 C: 240000 Min: 1 Act: 35 Avg: 28 Max: 159 T: 1 ( 4204) P:80 I:165 C: 181909 Min: 6 Act: 47 Avg: 29 Max: 170 T: 2 ( 4205) P:80 I:205 C: 146417 Min: 6 Act: 15 Avg: 29 Max: 157 T: 3 ( 4206) P:80 I:245 C: 122512 Min: 6 Act: 33 Avg: 30 Max: 166 [similar,skipped] T:38 ( 4241) P:80 I:1645 C: 18246 Min: 7 Act: 27 Avg: 34 Max: 167 T:39 ( 4242) P:80 I:1685 C: 17813 Min: 8 Act: 13 Avg: 34 Max: 164 T:40 ( 4243) P:80 I:1725 C: 17398 Min: 7 Act: 22 Avg: 33 Max: 156 T:41 ( 4244) P:80 I:1765 C: 17004 Min: 8 Act: 49 Avg: 34 Max: 141 root@debian:~/rt-tests#
Huge difference, isn't it?
2) If you want massive sync audio processing (w/o buffering) you can use
hard realtime OS. PC and Linux (even on bare metal) is not suitable for this, at least without RT patched kernel.
I am not understanding about you speaking. E1 card can be used on real host without problems.
They use buffering. Delays of 125uS is not uncommon in ordinary Linux kernel. BTW, google "system management interrupt".
3) AFAIR, 1 thread per call leg, but depends of endpoints. Most of time
threads in I/O waiting/sleeping and encoding/decoding state.
One thread per FS call, equal one thread cyclictest. In cyclictest most time threads sleep also. This threads measure time interval and sleep again.
In Freeswitch all of threads are not waking up every 125uS synchronously. .And TDM cards are not generating tens/hundreds of thousands interrupts per second as cyclictest can do.
IMO, to simulate something like 1 TDM E1 card using cyclictest you should use parameters that give you additional ~8000 irqs.
root@debian:~# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 3496 84304 21156 0 0 45 0 999 562 1 1 97 2 0 0 0 3496 84304 21156 0 0 0 0 105 222 0 0 100 0 0 0 0 2784 84304 21156 0 0 0 0 4276 8482 2 15 83 0 0 0 0 2784 84304 21156 0 0 0 0 8705 17146 2 35 63 0 0 0 0 2784 84304 21156 0 0 0 0 8678 17099 3 33 64 0 0 0 0 2784 84304 21156 0 0 0 0 8707 17135 2 37 61 0
In my case additional 8000 irqs with these parameters:
root@debian:~/rt-tests# ./cyclictest -t 30 -p 80 -d 30 -n -i 3000 -l 24000 -q WARN: stat /dev/cpu_dma_latency failed: No such file or directory T: 0 ( 5862) P:80 I:3000 C: 24000 Min: 7 Act: 9 Avg: 9 Max: 33 T: 1 ( 5863) P:80 I:3030 C: 23767 Min: 6 Act: 17 Avg: 8 Max: 49 T: 2 ( 5864) P:80 I:3060 C: 23534 Min: 6 Act: 12 Avg: 8 Max: 45 [same,skipped] T:26 ( 5888) P:80 I:3780 C: 19051 Min: 7 Act: 8 Avg: 9 Max: 35 T:27 ( 5889) P:80 I:3810 C: 18901 Min: 7 Act: 11 Avg: 8 Max: 40 T:28 ( 5890) P:80 I:3840 C: 18754 Min: 7 Act: 14 Avg: 8 Max: 39 T:29 ( 5891) P:80 I:3870 C: 18608 Min: 7 Act: 14 Avg: 8 Max: 36
6) In your case, most capable realtime hypervisor with RT_PREEMPT guest would be one of the best solution.
May be. According this publication ( http://www.linux-kvm.org/wiki/images/0/03/KVM-Forum-2011-RT-KVM.pdf) guest threads can be interrupted not more 112 uS.
Simple solution - for TDM card enough ordinary processor without virtualization. :o)
This is a best solution.
On Thu, Apr 9, 2015 at 8:09 PM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
2015-04-08 15:25 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:
1) 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz, But here we are talking about packet telephony where commonly used frame length is 20000 uS. So, intervals for testing should be 5000-10000-20000uS, not 125uS.
Not all FS installation has SIP endpoint. Some can have TDM cards, USB dongle and other analog equipment. Can you say what is length of this frame type? I am wondering about this question. I know that the frame 125ms is enough.
Most(if not all) of modern equipment have buffers. TDM cards too. Even $0.25 microcontrollers have buffers. Firing interrupts at each PCM sample not needed. If you want massive sync audio processing (w/o buffering) you can use hard realtime OS. PC and Linux (even on bare metal) is not suitable for this, at least without RT patched kernel.
2) Why you decided that all of 42 threads should wake up simultaneously? That totally impossible in real world applications. Here should be tens of uS. Dependent of network card and, possibly, other I/O hardware.
Can you say how many FS create threads for 40-50 active calls?
AFAIR, 1 thread per call leg, but depends of endpoints. Most of time threads in I/O waiting/sleeping and encoding/decoding state.
3) -q switch is missing I am thinking it is optional. Result is same
Dynamic output to screen consumes resources. How much, depends on method.(VGA, Serial console, SSH). In my case difference is significiant. So, fair testing should exclude such things, IMO.
4) Precise time measurement and realtime processing in VM environment is a challenge. To obtain acceptable results fine tuning of host's and guest's required. WIthout that this numbers ... does not mean anything.
Can you learn me how to correctly measure timer source quality in VM?
Serious players, as Wind River, use external timer.
In your case, most capable realtime hypervisor with RT_PREEMPT guest would be one of the best solution.
Dmitry.
On Wed, Apr 8, 2015 at 2:07 PM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
2015-04-07 10:48 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>:
> ... > 2) i 125 - time us between software timer interrupts for first > thread. It time is equal time between samples rate in PSTN networks > In order to maintain our sampling rate we have 125 microseconds (uS) > between consecutive samples (1/8,000Hz), which is more > ... >
125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz, But here we are talking about packet telephony where commonly used frame length is 20000 uS. So, intervals for testing should be 5000-10000-20000uS, not 125uS.
> 3) d 0 - this switch will make time between software timer > interrupts for second and following threads is equal timer value for first > thread >
Why you decided that all of 42 threads should wake up simultaneously? That totally impossible in real world applications. Here should be tens of uS. Dependent of network card and, possibly, other I/O hardware.
> 4) l 240000 - how many timer test must be made. It is adjusted for > 30 s overall test > 5) n - use clock_nanosleep. it is allow take more accurate timer > test values on my VM > 6) p 80 - it is copied from example on Cyclictest page >
-q switch is missing.
> And how in your opinion this numbers are related to Freeswitch in VM > environment? > Relation between Cyclictest switches and FS described above. > Relation betwen Cyclictest results and FS simple >
Precise time measurement and realtime processing in VM environment is a challenge. To obtain acceptable results fine tuning of host's and guest's required. WIthout that this numbers ... does not mean anything.
On Tue, Apr 7, 2015 at 5:38 AM, Dmitry Lysenko < > dvl36.ripe.nick@gmail.com> wrote: > >> Sergey, >> Why you suggest to use exactly this numbers in cyclictest's >> switches? >> And how in your opinion this numbers are related to Freeswitch in >> VM environment? >> >> P.S. >> https://mindlinux.wordpress.com/2013/10/25/using-and-understanding-the-real-... >> >> 2015-04-06 15:27 GMT+03:00 Sergey Safarov <s.safarov@gmail.com>: >> >>> Errol can you test VM timer with utility >>> https://rt.wiki.kernel.org/index.php/Cyclictest >>> >>> My VM currently have folowing results >>> >>> [root@fs1 ~]# date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i 125 -l 240000 && date >>> Sun Apr 5 07:01:31 GMT 2015 >>> # /dev/cpu_dma_latency set to 0us >>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574 >>> >>> T: 0 ( 1531) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 16 Max: 1302 >>> T: 1 ( 1532) P:80 I:125 C: 240000 Min: 2 Act: 15 Avg: 14 Max: 2895 >>> T: 2 ( 1533) P:80 I:125 C: 240000 Min: 3 Act: 10 Avg: 12 Max: 2888 >>> T: 3 ( 1534) P:80 I:125 C: 240000 Min: 1 Act: 16 Avg: 13 Max: 1333 >>> T: 4 ( 1535) P:80 I:125 C: 240000 Min: 2 Act: 20 Avg: 14 Max: 2962 >>> T: 5 ( 1536) P:80 I:125 C: 240000 Min: 1 Act: 12 Avg: 14 Max: 2946 >>> T: 6 ( 1537) P:80 I:125 C: 240000 Min: 2 Act: 10 Avg: 13 Max: 1363 >>> T: 7 ( 1538) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 13 Max: 1344 >>> T: 8 ( 1539) P:80 I:125 C: 240000 Min: 3 Act: 12 Avg: 16 Max: 1408 >>> T: 9 ( 1540) P:80 I:125 C: 240000 Min: 1 Act: 8 Avg: 13 Max: 2882 >>> T:10 ( 1541) P:80 I:125 C: 239974 Min: 2 Act: 12 Avg: 13 Max: 2908 >>> >>> >>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels < >>> ewsamuels@gmail.com> wrote: >>> >>>> Good job and Thanks for sharing! >>>> >>>> KVM at digitalocean.com, Debian 7 64bit, FreeSWITCH 1.4.15: bad >>>>> precision >>>> >>>> >>>> As someone who has been using KVM for a few years and has just >>>> migrated about 20 VMs across to Digital Ocean about 6 weeks ago I admit I >>>> am a little concerned about this finding. >>>> >>>> Just out of curiosity can you tell me the specs of the VM that >>>> you tested with and which DO Datacenter? I am using NYC3 so perhaps and I >>>> will do some tests as soon as I have some time. >>>> >>>> With regards to Xen was this at Linode or inhouse? >>>> >>>> Thanks >>>> >>>> Errol >>>> >>>> >>>> >>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin < >>>> ssinyagin@gmail.com> wrote: >>>> >>>>> here we go: >>>>> >>>>> https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virt... >>>>> >>>>> >>>>> >>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin < >>>>> ssinyagin@gmail.com> wrote: >>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M < >>>>> regis.freeswitch.org@tornad.net> wrote: >>>>> >> How do you check wavs together in audacity ? >>>>> > >>>>> > you just open two WAV files, align their starting position, and >>>>> > visually check the waves. If there's a problem, you would see >>>>> that >>>>> > after few seconds, the waves are not aligned any more. >>>>> > >>>>> >> Can you send me 2 samples wav (send... and recorded) so I >>>>> could see the 20ms >>>>> >> diff that you exposed... >>>>> >> >>>>> >> Is there a way to automate the result checking, so it could >>>>> help to monitor >>>>> >> itsp quality >>>>> > >>>>> > Here >>>>> > https://github.com/openpreserve/scape-xcorrsound/issues/17 >>>>> > this is a link to a tool which compares two sound files at the >>>>> wave >>>>> > level. You can also see the sound samples in the ticket. >>>>> > The problem is, that currently this tool does not detect such >>>>> slipping frames. >>>>> > >>>>> > I will also write a detailed article with examples, during the >>>>> weekend. >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> >> >>>>> >> >>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin < >>>>> ssinyagin@gmail.com>: >>>>> >>> >>>>> >>> Important is to send the media through two independent >>>>> carriers - then you >>>>> >>> can be sure that your clock is compared against a high >>>>> quality one. >>>>> >>> >>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M" < >>>>> regis.freeswitch.org@tornad.net> >>>>> >>> wrote: >>>>> >>>> >>>>> >>>> Hi, >>>>> >>>> >>>>> >>>> It's a very interresting test and conclusion. >>>>> >>>> >>>>> >>>> I will try to build a similar environnement on my >>>>> freeswitch to check >>>>> >>>> some system :) >>>>> >>>> >>>>> >>>> thanks >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin < >>>>> ssinyagin@gmail.com>: >>>>> >>>>> >>>>> >>>>> hi, >>>>> >>>>> >>>>> >>>>> Here are some test results which might be interesting: >>>>> >>>>> >>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they >>>>> use KVM), >>>>> >>>>> and originated test calls to some ITSP accounts. Each call >>>>> played a >>>>> >>>>> test sound file (voice samples from ITU: >>>>> >>>>> >>>>> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050 >>>>> >>>>> ), and on the receiving end another FreeSWITCH server was >>>>> recording >>>>> >>>>> the received session. The receiving server was on a XEN >>>>> VM. Both >>>>> >>>>> servers talked G711 to their corresponding ITSP accounts. >>>>> >>>>> >>>>> >>>>> Then I compared the source audio and received audio in >>>>> Audacity editor. >>>>> >>>>> >>>>> >>>>> The tests showed that the received voice wave shifts 20ms >>>>> from the >>>>> >>>>> original sound every few seconds, randomly back and >>>>> forward. This >>>>> >>>>> skewing was always present, regardless of ITSP and time of >>>>> day. >>>>> >>>>> >>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves >>>>> appeared >>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one >>>>> test, and I >>>>> >>>>> guess it's related to transcoding buffer, as the sound was >>>>> obviously >>>>> >>>>> transcoded). >>>>> >>>>> >>>>> >>>>> The effect was always visible with a 30-second sample, and >>>>> I also made >>>>> >>>>> additional tests with 2-minute samples. >>>>> >>>>> >>>>> >>>>> It would also be interesting to test this from a VmWare >>>>> VM. If someone >>>>> >>>>> wants to run such a test, feel free to contact me directly. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> cheers, >>>>> >>>>> stan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> >>>>> Professional FreeSWITCH Consulting Services: >>>>> >>>>> consulting@freeswitch.org >>>>> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> >>>>> >>>>> Official FreeSWITCH Sites >>>>> >>>>> http://www.freeswitch.org >>>>> >>>>> http://confluence.freeswitch.org >>>>> >>>>> http://www.cluecon.com >>>>> >>>>> >>>>> >>>>> FreeSWITCH-users mailing list >>>>> >>>>> FreeSWITCH-users@lists.freeswitch.org >>>>> >>>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> >>>>> http://www.freeswitch.org >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> _________________________________________________________________________ >>>>> >>>> Professional FreeSWITCH Consulting Services: >>>>> >>>> consulting@freeswitch.org >>>>> >>>> http://www.freeswitchsolutions.com >>>>> >>>> >>>>> >>>> Official FreeSWITCH Sites >>>>> >>>> http://www.freeswitch.org >>>>> >>>> http://confluence.freeswitch.org >>>>> >>>> http://www.cluecon.com >>>>> >>>> >>>>> >>>> FreeSWITCH-users mailing list >>>>> >>>> FreeSWITCH-users@lists.freeswitch.org >>>>> >>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> >>>> http://www.freeswitch.org >>>>> >>> >>>>> >>> >>>>> >>> >>>>> _________________________________________________________________________ >>>>> >>> Professional FreeSWITCH Consulting Services: >>>>> >>> consulting@freeswitch.org >>>>> >>> http://www.freeswitchsolutions.com >>>>> >>> >>>>> >>> Official FreeSWITCH Sites >>>>> >>> http://www.freeswitch.org >>>>> >>> http://confluence.freeswitch.org >>>>> >>> http://www.cluecon.com >>>>> >>> >>>>> >>> FreeSWITCH-users mailing list >>>>> >>> FreeSWITCH-users@lists.freeswitch.org >>>>> >>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> >>> http://www.freeswitch.org >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> _________________________________________________________________________ >>>>> >> Professional FreeSWITCH Consulting Services: >>>>> >> consulting@freeswitch.org >>>>> >> http://www.freeswitchsolutions.com >>>>> >> >>>>> >> Official FreeSWITCH Sites >>>>> >> http://www.freeswitch.org >>>>> >> http://confluence.freeswitch.org >>>>> >> http://www.cluecon.com >>>>> >> >>>>> >> FreeSWITCH-users mailing list >>>>> >> FreeSWITCH-users@lists.freeswitch.org >>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> >> http://www.freeswitch.org >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://confluence.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>>> >>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://confluence.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org