[Freeswitch-users] KVM vs. XEN virtualization

Dmitry Lysenko dvl36.ripe.nick at gmail.com
Fri Apr 10 18:06:34 MSD 2015 

2015-04-10 0:33 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:

> 1) Most(if not all) of modern equipment have buffers. TDM cards too. Even
> $0.25 microcontrollers have buffers.
> Firing interrupts at each PCM sample not needed.
>
> The problem is whether there is or not the buffer. Is exist in all the
> equipment.
> The problem is the amount of buffer, and through any time it overflows. I
> thing you cannot say buffer size has TDM card and USB modems. :o)
>

According to Digium sources your TDM card have 20ms max buffer size(3ms
default)
USB dongles have at least 20 ms.


> The above i write results of testing Digium E1 card in virtual
> environment. Test that is successful on real host, will fail in virtual
> environment.
> I you see source of timertest utility you will see that Digium test timer
> for 8000 samples for 1000 ms (1s). (Primary utility author Mark Spencer <
> markster at digium.com>)
> It is same value I recommend early.
>

But in your case, cyclictest can generate 336000 (42*8000) hardware timer
irqs per second VS 8000 from TDM card's.

Look:
root at debian:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
wa
 0  0      0   2348  85428  21156    0    0   134     0 1856 3409  1  2 92
 5
 0  0      0   2320  85428  21156    0    0     0     0  106  224  1  0 99
 0
[similar, skipped]
 0  0      0   2312  85436  21156    0    0     0     0  110  231  0  0 100
 0
 0  0      0   2312  85436  21156    0    0     0     0  109  225  0  2 98
 0
45  0      0   1924  84908  21156    0    0     0     0 102909 146412  0 60
39  0
Starting of cyclictest with your parameters ^^^^^.

102909 interrupts and 146412 context switches per second.
For one TDM card there should be slightly more then 8000 of interrupts.

Results:

root at debian:~/rt-tests# ./cyclictest -t42 -p 80 *-d 0* -n -i 125 -l 240000
-q
WARN: stat /dev/cpu_dma_latency failed: No such file or directory
T: 0 ( 4028) P:80 I:125 C: 239821 Min:      0 Act:100534 Avg:38764 Max:
 229794
T: 1 ( 4029) P:80 I:125 C: 239820 Min:      0 Act:100536 Avg:39646 Max:
 233558
T: 2 ( 4030) P:80 I:125 C: 240000 Min:      0 Act:100478 Avg:37947 Max:
 229143
T: 3 ( 4031) P:80 I:125 C: 240000 Min:      0 Act:106202 Avg:38130 Max:
 239031
[similar,skipped]
T:38 ( 4066) P:80 I:125 C: 239821 Min:      1 Act:100292 Avg:41273 Max:
 238424
T:39 ( 4067) P:80 I:125 C: 239822 Min:      0 Act:100174 Avg:40254 Max:
 238881
T:40 ( 4068) P:80 I:125 C: 239816 Min:      0 Act:100650 Avg:39910 Max:
 239104
T:41 ( 4069) P:80 I:125 C: 239817 Min:      0 Act:100526 Avg:38282 Max:
 224452
root at debian:~/rt-tests#

Increased intervals between threads:

root at debian:~/rt-tests# ./cyclictest -t42 -p 80 *-d 40* -n -i 125 -l 240000
-q
WARN: stat /dev/cpu_dma_latency failed: No such file or directory
T: 0 ( 4203) P:80 I:125 C: 240000 Min:      1 Act:   35 Avg:   28 Max:
159
T: 1 ( 4204) P:80 I:165 C: 181909 Min:      6 Act:   47 Avg:   29 Max:
170
T: 2 ( 4205) P:80 I:205 C: 146417 Min:      6 Act:   15 Avg:   29 Max:
157
T: 3 ( 4206) P:80 I:245 C: 122512 Min:      6 Act:   33 Avg:   30 Max:
166
[similar,skipped]
T:38 ( 4241) P:80 I:1645 C:  18246 Min:      7 Act:   27 Avg:   34 Max:
167
T:39 ( 4242) P:80 I:1685 C:  17813 Min:      8 Act:   13 Avg:   34 Max:
164
T:40 ( 4243) P:80 I:1725 C:  17398 Min:      7 Act:   22 Avg:   33 Max:
156
T:41 ( 4244) P:80 I:1765 C:  17004 Min:      8 Act:   49 Avg:   34 Max:
141
root at debian:~/rt-tests#

Huge difference, isn't it?

2) If you want massive sync audio processing (w/o buffering) you can use
> hard realtime OS. PC and Linux (even on bare metal) is not suitable for
> this, at least without RT patched kernel.
>
> I am not understanding about you speaking. E1 card can be used on real
> host without problems.
>

They use buffering. Delays of 125uS is not uncommon in ordinary Linux
kernel. BTW, google "system management interrupt".

3) AFAIR, 1 thread per call leg, but depends of endpoints. Most of time
> threads in I/O waiting/sleeping and encoding/decoding state.
>
> One thread per FS call, equal one thread cyclictest. In cyclictest most
> time threads sleep also. This threads measure time interval and sleep again.
>

In Freeswitch all of threads are not waking up every 125uS synchronously.
.And TDM cards are not generating tens/hundreds of thousands interrupts per
second as cyclictest can do.

IMO, to simulate something like 1 TDM E1 card using cyclictest you should
use parameters that give you additional ~8000 irqs.

root at debian:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
wa
 0  0      0   3496  84304  21156    0    0    45     0  999  562  1  1 97
 2
 0  0      0   3496  84304  21156    0    0     0     0  105  222  0  0 100
 0
 0  0      0   2784  84304  21156    0    0     0     0 4276 8482  2 15 83
 0
 0  0      0   2784  84304  21156    0    0     0     0 8705 17146  2 35 63
 0
 0  0      0   2784  84304  21156    0    0     0     0 8678 17099  3 33 64
 0
 0  0      0   2784  84304  21156    0    0     0     0 8707 17135  2 37 61
 0

In my case additional 8000 irqs with these parameters:

root at debian:~/rt-tests# ./cyclictest  -t 30 -p 80 -d 30 -n -i 3000 -l 24000
-q
WARN: stat /dev/cpu_dma_latency failed: No such file or directory
T: 0 ( 5862) P:80 I:3000 C:  24000 Min:      7 Act:    9 Avg:    9 Max:
 33
T: 1 ( 5863) P:80 I:3030 C:  23767 Min:      6 Act:   17 Avg:    8 Max:
 49
T: 2 ( 5864) P:80 I:3060 C:  23534 Min:      6 Act:   12 Avg:    8 Max:
 45
[same,skipped]
T:26 ( 5888) P:80 I:3780 C:  19051 Min:      7 Act:    8 Avg:    9 Max:
 35
T:27 ( 5889) P:80 I:3810 C:  18901 Min:      7 Act:   11 Avg:    8 Max:
 40
T:28 ( 5890) P:80 I:3840 C:  18754 Min:      7 Act:   14 Avg:    8 Max:
 39
T:29 ( 5891) P:80 I:3870 C:  18608 Min:      7 Act:   14 Avg:    8 Max:
 36


> 6) In your case, most capable realtime hypervisor with RT_PREEMPT guest
> would be one of the best solution.
>
> May be. According this publication (
> http://www.linux-kvm.org/wiki/images/0/03/KVM-Forum-2011-RT-KVM.pdf)
> guest threads can be interrupted not more 112 uS.
>
> Simple solution - for TDM card enough ordinary processor without
> virtualization. :o)
>

This is a best solution.


> On Thu, Apr 9, 2015 at 8:09 PM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com>
> wrote:
>
>>
>>
>> 2015-04-08 15:25 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>>
>>> 1) 125uS is the interval between FrameSync of ONE(!) PCM sample at
>>> 8000hz,
>>> But here we are talking about packet telephony where commonly used frame
>>> length is 20000 uS.
>>> So, intervals for testing should be 5000-10000-20000uS, not 125uS.
>>>
>>> Not all FS installation has SIP endpoint. Some can have TDM cards, USB
>>> dongle and other analog equipment. Can you say what is length of this frame
>>> type?
>>> I am wondering about this question. I know that the frame 125ms is
>>> enough.
>>>
>>
>> Most(if not all) of modern equipment have buffers. TDM cards too. Even
>> $0.25 microcontrollers have buffers.
>> Firing interrupts at each PCM sample not needed.
>> If you want massive sync audio processing (w/o buffering) you can use
>> hard realtime OS. PC and Linux (even on bare metal) is not suitable for
>> this, at least without RT patched kernel.
>>
>>
>>> 2) Why you decided that all of 42 threads should wake up simultaneously?
>>> That totally impossible in real world applications.
>>> Here should be tens of uS. Dependent of network card and, possibly,
>>> other I/O hardware.
>>>
>>> Can you say how many FS create threads for 40-50 active calls?
>>>
>>
>>
>> AFAIR, 1 thread per call leg, but depends of endpoints. Most of time
>> threads in I/O waiting/sleeping and encoding/decoding state.
>>
>>
>>> 3) -q switch is missing
>>> I am thinking it is optional. Result is same
>>>
>>
>> Dynamic output to screen consumes resources. How much, depends on
>> method.(VGA, Serial console, SSH). In my case difference is significiant.
>> So, fair testing should exclude such things, IMO.
>>
>>
>>> 4) Precise time measurement and realtime processing in VM environment
>>>  is a challenge.
>>> To obtain acceptable results fine tuning of host's and guest's  required.
>>> WIthout that this numbers ... does not mean anything.
>>>
>>> Can you learn me how to correctly measure timer source quality in VM?
>>>
>>
>> Serious players, as Wind River, use external timer.
>>
>> In your case, most capable realtime hypervisor with RT_PREEMPT guest
>> would be one of the best solution.
>>
>> Dmitry.
>>
>>
>>> On Wed, Apr 8, 2015 at 2:07 PM, Dmitry Lysenko <
>>> dvl36.ripe.nick at gmail.com> wrote:
>>>
>>>> 2015-04-07 10:48 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>>>>
>>>>> ...
>>>>> 2) i 125 - time us between software timer interrupts for first thread.
>>>>> It time is equal time between samples rate in PSTN networks
>>>>> In order to maintain our sampling rate we have 125 microseconds (uS)
>>>>> between consecutive samples (1/8,000Hz), which is more
>>>>>
>>>> ...
>>>>>
>>>>
>>>> 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,
>>>> But here we are talking about packet telephony where commonly used
>>>> frame length is 20000 uS.
>>>> So, intervals for testing should be 5000-10000-20000uS, not 125uS.
>>>>
>>>>
>>>>> 3) d 0 - this switch will make time between software timer interrupts
>>>>> for second and following threads is equal timer value for first thread
>>>>>
>>>>
>>>> Why you decided that all of 42 threads should wake up simultaneously?
>>>> That totally impossible in real world applications.
>>>> Here should be tens of uS. Dependent of network card and, possibly,
>>>> other I/O hardware.
>>>>
>>>>
>>>>> 4) l 240000 - how many timer test must be made. It is adjusted for 30
>>>>> s overall test
>>>>> 5) n - use clock_nanosleep. it is allow take more accurate timer test
>>>>> values on my VM
>>>>> 6) p 80 - it is copied from example on Cyclictest page
>>>>>
>>>>
>>>> -q switch is missing.
>>>>
>>>>
>>>>> And how in your opinion this numbers are related to Freeswitch in VM
>>>>> environment?
>>>>> Relation between Cyclictest switches and FS described above.
>>>>> Relation betwen Cyclictest results and FS simple
>>>>>
>>>>
>>>> Precise time measurement and realtime processing in VM environment  is
>>>> a challenge.
>>>> To obtain acceptable results fine tuning of host's and guest's
>>>>  required.
>>>> WIthout that this numbers ... does not mean anything.
>>>>
>>>>
>>>> On Tue, Apr 7, 2015 at 5:38 AM, Dmitry Lysenko <
>>>>> dvl36.ripe.nick at gmail.com> wrote:
>>>>>
>>>>>> Sergey,
>>>>>> Why you suggest to use exactly this numbers in cyclictest's switches?
>>>>>> And how in your opinion this numbers are related to Freeswitch in VM
>>>>>> environment?
>>>>>>
>>>>>> P.S.
>>>>>> https://mindlinux.wordpress.com/2013/10/25/using-and-understanding-the-real-time-cyclictest-benchmark-frank-rowand-sony/
>>>>>>
>>>>>> 2015-04-06 15:27 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>>>>>>
>>>>>>> Errol can you test VM timer with utility
>>>>>>> https://rt.wiki.kernel.org/index.php/Cyclictest
>>>>>>>
>>>>>>> My VM currently have folowing results
>>>>>>>
>>>>>>> [root at fs1 ~]#  date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i 125 -l 240000 && date
>>>>>>> Sun Apr  5 07:01:31 GMT 2015
>>>>>>> # /dev/cpu_dma_latency set to 0us
>>>>>>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574
>>>>>>>
>>>>>>> T: 0 ( 1531) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   16 Max:    1302
>>>>>>> T: 1 ( 1532) P:80 I:125 C: 240000 Min:      2 Act:   15 Avg:   14 Max:    2895
>>>>>>> T: 2 ( 1533) P:80 I:125 C: 240000 Min:      3 Act:   10 Avg:   12 Max:    2888
>>>>>>> T: 3 ( 1534) P:80 I:125 C: 240000 Min:      1 Act:   16 Avg:   13 Max:    1333
>>>>>>> T: 4 ( 1535) P:80 I:125 C: 240000 Min:      2 Act:   20 Avg:   14 Max:    2962
>>>>>>> T: 5 ( 1536) P:80 I:125 C: 240000 Min:      1 Act:   12 Avg:   14 Max:    2946
>>>>>>> T: 6 ( 1537) P:80 I:125 C: 240000 Min:      2 Act:   10 Avg:   13 Max:    1363
>>>>>>> T: 7 ( 1538) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   13 Max:    1344
>>>>>>> T: 8 ( 1539) P:80 I:125 C: 240000 Min:      3 Act:   12 Avg:   16 Max:    1408
>>>>>>> T: 9 ( 1540) P:80 I:125 C: 240000 Min:      1 Act:    8 Avg:   13 Max:    2882
>>>>>>> T:10 ( 1541) P:80 I:125 C: 239974 Min:      2 Act:   12 Avg:   13 Max:    2908
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <ewsamuels at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Good job and Thanks for sharing!
>>>>>>>>
>>>>>>>> KVM at digitalocean.com, Debian 7 64bit, FreeSWITCH 1.4.15: bad
>>>>>>>>> precision
>>>>>>>>
>>>>>>>>
>>>>>>>> As someone who has been using KVM for a few years and has just
>>>>>>>> migrated about 20 VMs across to Digital Ocean about 6 weeks ago I admit I
>>>>>>>> am a little concerned about this finding.
>>>>>>>>
>>>>>>>> Just out of curiosity can you tell me the specs of the VM that you
>>>>>>>> tested with and which DO Datacenter?  I am using NYC3 so perhaps and I will
>>>>>>>> do some tests as soon as I have some time.
>>>>>>>>
>>>>>>>> With regards to Xen was this at Linode or inhouse?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Errol
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin <
>>>>>>>> ssinyagin at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> here we go:
>>>>>>>>>
>>>>>>>>> https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin <
>>>>>>>>> ssinyagin at gmail.com> wrote:
>>>>>>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M <
>>>>>>>>> regis.freeswitch.org at tornad.net> wrote:
>>>>>>>>> >> How do you check wavs together in audacity ?
>>>>>>>>> >
>>>>>>>>> > you just open two WAV files, align their starting position, and
>>>>>>>>> > visually check the waves. If there's a problem, you would see
>>>>>>>>> that
>>>>>>>>> > after few seconds, the waves are not aligned any more.
>>>>>>>>> >
>>>>>>>>> >> Can you send me 2 samples wav (send... and recorded) so I could
>>>>>>>>> see the 20ms
>>>>>>>>> >> diff that you exposed...
>>>>>>>>> >>
>>>>>>>>> >> Is there a way to automate the result checking, so it could
>>>>>>>>> help to monitor
>>>>>>>>> >> itsp quality
>>>>>>>>> >
>>>>>>>>> > Here
>>>>>>>>> > https://github.com/openpreserve/scape-xcorrsound/issues/17
>>>>>>>>> > this is a link to a tool which compares two sound files at the
>>>>>>>>> wave
>>>>>>>>> > level. You can also see the sound samples in the ticket.
>>>>>>>>> > The problem is, that currently this tool does not detect such
>>>>>>>>> slipping frames.
>>>>>>>>> >
>>>>>>>>> > I will also write a detailed article with examples, during the
>>>>>>>>> weekend.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin <
>>>>>>>>> ssinyagin at gmail.com>:
>>>>>>>>> >>>
>>>>>>>>> >>> Important is to send the media through two independent
>>>>>>>>> carriers - then you
>>>>>>>>> >>> can be sure that your clock is compared against a high quality
>>>>>>>>> one.
>>>>>>>>> >>>
>>>>>>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M" <
>>>>>>>>> regis.freeswitch.org at tornad.net>
>>>>>>>>> >>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Hi,
>>>>>>>>> >>>>
>>>>>>>>> >>>> It's a very interresting test and conclusion.
>>>>>>>>> >>>>
>>>>>>>>> >>>> I will try to build a similar environnement on my freeswitch
>>>>>>>>> to check
>>>>>>>>> >>>> some system :)
>>>>>>>>> >>>>
>>>>>>>>> >>>> thanks
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin <
>>>>>>>>> ssinyagin at gmail.com>:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> hi,
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Here are some test results which might be interesting:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they
>>>>>>>>> use KVM),
>>>>>>>>> >>>>> and originated test calls to some ITSP accounts. Each call
>>>>>>>>> played a
>>>>>>>>> >>>>> test sound file (voice samples from ITU:
>>>>>>>>> >>>>>
>>>>>>>>> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050
>>>>>>>>> >>>>> ), and on the receiving end another FreeSWITCH server was
>>>>>>>>> recording
>>>>>>>>> >>>>> the received session. The receiving server was on a XEN VM.
>>>>>>>>> Both
>>>>>>>>> >>>>> servers talked G711 to their corresponding ITSP accounts.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Then I compared the source audio and received audio in
>>>>>>>>> Audacity editor.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> The tests showed that the received voice wave shifts 20ms
>>>>>>>>> from the
>>>>>>>>> >>>>> original sound every few seconds, randomly back and forward.
>>>>>>>>> This
>>>>>>>>> >>>>> skewing was always present, regardless of ITSP and time of
>>>>>>>>> day.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves
>>>>>>>>> appeared
>>>>>>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one
>>>>>>>>> test, and I
>>>>>>>>> >>>>> guess it's related to transcoding buffer, as the sound was
>>>>>>>>> obviously
>>>>>>>>> >>>>> transcoded).
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> The effect was always visible with a 30-second sample, and I
>>>>>>>>> also made
>>>>>>>>> >>>>> additional tests with 2-minute samples.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> It would also be interesting to test this from a VmWare VM.
>>>>>>>>> If someone
>>>>>>>>> >>>>> wants to run such a test, feel free to contact me directly.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> cheers,
>>>>>>>>> >>>>> stan
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> >>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> >>>>> consulting at freeswitch.org
>>>>>>>>> >>>>> http://www.freeswitchsolutions.com
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Official FreeSWITCH Sites
>>>>>>>>> >>>>> http://www.freeswitch.org
>>>>>>>>> >>>>> http://confluence.freeswitch.org
>>>>>>>>> >>>>> http://www.cluecon.com
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> FreeSWITCH-users mailing list
>>>>>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> >>>>>
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> >>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> >>>>> http://www.freeswitch.org
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> >>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> >>>> consulting at freeswitch.org
>>>>>>>>> >>>> http://www.freeswitchsolutions.com
>>>>>>>>> >>>>
>>>>>>>>> >>>> Official FreeSWITCH Sites
>>>>>>>>> >>>> http://www.freeswitch.org
>>>>>>>>> >>>> http://confluence.freeswitch.org
>>>>>>>>> >>>> http://www.cluecon.com
>>>>>>>>> >>>>
>>>>>>>>> >>>> FreeSWITCH-users mailing list
>>>>>>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> >>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> >>>> http://www.freeswitch.org
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> >>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> >>> consulting at freeswitch.org
>>>>>>>>> >>> http://www.freeswitchsolutions.com
>>>>>>>>> >>>
>>>>>>>>> >>> Official FreeSWITCH Sites
>>>>>>>>> >>> http://www.freeswitch.org
>>>>>>>>> >>> http://confluence.freeswitch.org
>>>>>>>>> >>> http://www.cluecon.com
>>>>>>>>> >>>
>>>>>>>>> >>> FreeSWITCH-users mailing list
>>>>>>>>> >>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> >>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> >>> http://www.freeswitch.org
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> >> Professional FreeSWITCH Consulting Services:
>>>>>>>>> >> consulting at freeswitch.org
>>>>>>>>> >> http://www.freeswitchsolutions.com
>>>>>>>>> >>
>>>>>>>>> >> Official FreeSWITCH Sites
>>>>>>>>> >> http://www.freeswitch.org
>>>>>>>>> >> http://confluence.freeswitch.org
>>>>>>>>> >> http://www.cluecon.com
>>>>>>>>> >>
>>>>>>>>> >> FreeSWITCH-users mailing list
>>>>>>>>> >> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> >> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> >> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150410/6b153092/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list