[Freeswitch-users] KVM vs. XEN virtualization

Sergey Safarov s.safarov at gmail.com
Fri Apr 10 01:33:43 MSD 2015 

1) Most(if not all) of modern equipment have buffers. TDM cards too. Even
$0.25 microcontrollers have buffers.
Firing interrupts at each PCM sample not needed.

The problem is whether there is or not the buffer. Is exist in all the
equipment.
The problem is the amount of buffer, and through any time it overflows. I
thing you cannot say buffer size has TDM card and USB modems. :o)
The above i write results of testing Digium E1 card in virtual environment.
Test that is successful on real host, will fail in virtual environment.
I you see source of timertest utility you will see that Digium test timer
for 8000 samples for 1000 ms (1s). (Primary utility author Mark Spencer <
markster at digium.com>)
It is same value I recommend early.

2) If you want massive sync audio processing (w/o buffering) you can use
hard realtime OS. PC and Linux (even on bare metal) is not suitable for
this, at least without RT patched kernel.

I am not understanding about you speaking. E1 card can be used on real host
without problems.

3) AFAIR, 1 thread per call leg, but depends of endpoints. Most of time
threads in I/O waiting/sleeping and encoding/decoding state.

One thread per FS call, equal one thread cyclictest. In cyclictest most
time threads sleep also. This threads measure time interval and sleep again.

4) Dynamic output to screen consumes resources. How much, depends on
method.(VGA, Serial console, SSH). In my case difference is significiant.
So, fair testing should exclude such things, IMO.

You rightly pointed out it

5) Serious players, as Wind River, use external timer.

Wind River solutions for highly specialized market. Not for simple user.

6) In your case, most capable realtime hypervisor with RT_PREEMPT guest
would be one of the best solution.

May be. According this publication (
http://www.linux-kvm.org/wiki/images/0/03/KVM-Forum-2011-RT-KVM.pdf) guest
threads can be interrupted not more 112 uS.

Simple solution - for TDM card enough ordinary processor without
virtualization. :o)



On Thu, Apr 9, 2015 at 8:09 PM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com>
wrote:

>
>
> 2015-04-08 15:25 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>
>> 1) 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,
>> But here we are talking about packet telephony where commonly used frame
>> length is 20000 uS.
>> So, intervals for testing should be 5000-10000-20000uS, not 125uS.
>>
>> Not all FS installation has SIP endpoint. Some can have TDM cards, USB
>> dongle and other analog equipment. Can you say what is length of this frame
>> type?
>> I am wondering about this question. I know that the frame 125ms is enough.
>>
>
> Most(if not all) of modern equipment have buffers. TDM cards too. Even
> $0.25 microcontrollers have buffers.
> Firing interrupts at each PCM sample not needed.
> If you want massive sync audio processing (w/o buffering) you can use hard
> realtime OS. PC and Linux (even on bare metal) is not suitable for this, at
> least without RT patched kernel.
>
>
>> 2) Why you decided that all of 42 threads should wake up simultaneously?
>> That totally impossible in real world applications.
>> Here should be tens of uS. Dependent of network card and, possibly, other
>> I/O hardware.
>>
>> Can you say how many FS create threads for 40-50 active calls?
>>
>
>
> AFAIR, 1 thread per call leg, but depends of endpoints. Most of time
> threads in I/O waiting/sleeping and encoding/decoding state.
>
>
>> 3) -q switch is missing
>> I am thinking it is optional. Result is same
>>
>
> Dynamic output to screen consumes resources. How much, depends on
> method.(VGA, Serial console, SSH). In my case difference is significiant.
> So, fair testing should exclude such things, IMO.
>
>
>> 4) Precise time measurement and realtime processing in VM environment
>>  is a challenge.
>> To obtain acceptable results fine tuning of host's and guest's  required.
>> WIthout that this numbers ... does not mean anything.
>>
>> Can you learn me how to correctly measure timer source quality in VM?
>>
>
> Serious players, as Wind River, use external timer.
>
> In your case, most capable realtime hypervisor with RT_PREEMPT guest would
> be one of the best solution.
>
> Dmitry.
>
>
>> On Wed, Apr 8, 2015 at 2:07 PM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com
>> > wrote:
>>
>>> 2015-04-07 10:48 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>>>
>>>> ...
>>>> 2) i 125 - time us between software timer interrupts for first thread.
>>>> It time is equal time between samples rate in PSTN networks
>>>> In order to maintain our sampling rate we have 125 microseconds (uS)
>>>> between consecutive samples (1/8,000Hz), which is more
>>>>
>>> ...
>>>>
>>>
>>> 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,
>>> But here we are talking about packet telephony where commonly used frame
>>> length is 20000 uS.
>>> So, intervals for testing should be 5000-10000-20000uS, not 125uS.
>>>
>>>
>>>> 3) d 0 - this switch will make time between software timer interrupts
>>>> for second and following threads is equal timer value for first thread
>>>>
>>>
>>> Why you decided that all of 42 threads should wake up simultaneously?
>>> That totally impossible in real world applications.
>>> Here should be tens of uS. Dependent of network card and, possibly,
>>> other I/O hardware.
>>>
>>>
>>>> 4) l 240000 - how many timer test must be made. It is adjusted for 30 s
>>>> overall test
>>>> 5) n - use clock_nanosleep. it is allow take more accurate timer test
>>>> values on my VM
>>>> 6) p 80 - it is copied from example on Cyclictest page
>>>>
>>>
>>> -q switch is missing.
>>>
>>>
>>>> And how in your opinion this numbers are related to Freeswitch in VM
>>>> environment?
>>>> Relation between Cyclictest switches and FS described above.
>>>> Relation betwen Cyclictest results and FS simple
>>>>
>>>
>>> Precise time measurement and realtime processing in VM environment  is a
>>> challenge.
>>> To obtain acceptable results fine tuning of host's and guest's  required.
>>> WIthout that this numbers ... does not mean anything.
>>>
>>>
>>> On Tue, Apr 7, 2015 at 5:38 AM, Dmitry Lysenko <
>>>> dvl36.ripe.nick at gmail.com> wrote:
>>>>
>>>>> Sergey,
>>>>> Why you suggest to use exactly this numbers in cyclictest's switches?
>>>>> And how in your opinion this numbers are related to Freeswitch in VM
>>>>> environment?
>>>>>
>>>>> P.S.
>>>>> https://mindlinux.wordpress.com/2013/10/25/using-and-understanding-the-real-time-cyclictest-benchmark-frank-rowand-sony/
>>>>>
>>>>> 2015-04-06 15:27 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
>>>>>
>>>>>> Errol can you test VM timer with utility
>>>>>> https://rt.wiki.kernel.org/index.php/Cyclictest
>>>>>>
>>>>>> My VM currently have folowing results
>>>>>>
>>>>>> [root at fs1 ~]#  date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i 125 -l 240000 && date
>>>>>> Sun Apr  5 07:01:31 GMT 2015
>>>>>> # /dev/cpu_dma_latency set to 0us
>>>>>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574
>>>>>>
>>>>>> T: 0 ( 1531) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   16 Max:    1302
>>>>>> T: 1 ( 1532) P:80 I:125 C: 240000 Min:      2 Act:   15 Avg:   14 Max:    2895
>>>>>> T: 2 ( 1533) P:80 I:125 C: 240000 Min:      3 Act:   10 Avg:   12 Max:    2888
>>>>>> T: 3 ( 1534) P:80 I:125 C: 240000 Min:      1 Act:   16 Avg:   13 Max:    1333
>>>>>> T: 4 ( 1535) P:80 I:125 C: 240000 Min:      2 Act:   20 Avg:   14 Max:    2962
>>>>>> T: 5 ( 1536) P:80 I:125 C: 240000 Min:      1 Act:   12 Avg:   14 Max:    2946
>>>>>> T: 6 ( 1537) P:80 I:125 C: 240000 Min:      2 Act:   10 Avg:   13 Max:    1363
>>>>>> T: 7 ( 1538) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   13 Max:    1344
>>>>>> T: 8 ( 1539) P:80 I:125 C: 240000 Min:      3 Act:   12 Avg:   16 Max:    1408
>>>>>> T: 9 ( 1540) P:80 I:125 C: 240000 Min:      1 Act:    8 Avg:   13 Max:    2882
>>>>>> T:10 ( 1541) P:80 I:125 C: 239974 Min:      2 Act:   12 Avg:   13 Max:    2908
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <ewsamuels at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Good job and Thanks for sharing!
>>>>>>>
>>>>>>> KVM at digitalocean.com, Debian 7 64bit, FreeSWITCH 1.4.15: bad
>>>>>>>> precision
>>>>>>>
>>>>>>>
>>>>>>> As someone who has been using KVM for a few years and has just
>>>>>>> migrated about 20 VMs across to Digital Ocean about 6 weeks ago I admit I
>>>>>>> am a little concerned about this finding.
>>>>>>>
>>>>>>> Just out of curiosity can you tell me the specs of the VM that you
>>>>>>> tested with and which DO Datacenter?  I am using NYC3 so perhaps and I will
>>>>>>> do some tests as soon as I have some time.
>>>>>>>
>>>>>>> With regards to Xen was this at Linode or inhouse?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Errol
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin <
>>>>>>> ssinyagin at gmail.com> wrote:
>>>>>>>
>>>>>>>> here we go:
>>>>>>>>
>>>>>>>> https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin <
>>>>>>>> ssinyagin at gmail.com> wrote:
>>>>>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M <
>>>>>>>> regis.freeswitch.org at tornad.net> wrote:
>>>>>>>> >> How do you check wavs together in audacity ?
>>>>>>>> >
>>>>>>>> > you just open two WAV files, align their starting position, and
>>>>>>>> > visually check the waves. If there's a problem, you would see that
>>>>>>>> > after few seconds, the waves are not aligned any more.
>>>>>>>> >
>>>>>>>> >> Can you send me 2 samples wav (send... and recorded) so I could
>>>>>>>> see the 20ms
>>>>>>>> >> diff that you exposed...
>>>>>>>> >>
>>>>>>>> >> Is there a way to automate the result checking, so it could help
>>>>>>>> to monitor
>>>>>>>> >> itsp quality
>>>>>>>> >
>>>>>>>> > Here
>>>>>>>> > https://github.com/openpreserve/scape-xcorrsound/issues/17
>>>>>>>> > this is a link to a tool which compares two sound files at the
>>>>>>>> wave
>>>>>>>> > level. You can also see the sound samples in the ticket.
>>>>>>>> > The problem is, that currently this tool does not detect such
>>>>>>>> slipping frames.
>>>>>>>> >
>>>>>>>> > I will also write a detailed article with examples, during the
>>>>>>>> weekend.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin <
>>>>>>>> ssinyagin at gmail.com>:
>>>>>>>> >>>
>>>>>>>> >>> Important is to send the media through two independent carriers
>>>>>>>> - then you
>>>>>>>> >>> can be sure that your clock is compared against a high quality
>>>>>>>> one.
>>>>>>>> >>>
>>>>>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M" <
>>>>>>>> regis.freeswitch.org at tornad.net>
>>>>>>>> >>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Hi,
>>>>>>>> >>>>
>>>>>>>> >>>> It's a very interresting test and conclusion.
>>>>>>>> >>>>
>>>>>>>> >>>> I will try to build a similar environnement on my freeswitch
>>>>>>>> to check
>>>>>>>> >>>> some system :)
>>>>>>>> >>>>
>>>>>>>> >>>> thanks
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin <
>>>>>>>> ssinyagin at gmail.com>:
>>>>>>>> >>>>>
>>>>>>>> >>>>> hi,
>>>>>>>> >>>>>
>>>>>>>> >>>>> Here are some test results which might be interesting:
>>>>>>>> >>>>>
>>>>>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they
>>>>>>>> use KVM),
>>>>>>>> >>>>> and originated test calls to some ITSP accounts. Each call
>>>>>>>> played a
>>>>>>>> >>>>> test sound file (voice samples from ITU:
>>>>>>>> >>>>>
>>>>>>>> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050
>>>>>>>> >>>>> ), and on the receiving end another FreeSWITCH server was
>>>>>>>> recording
>>>>>>>> >>>>> the received session. The receiving server was on a XEN VM.
>>>>>>>> Both
>>>>>>>> >>>>> servers talked G711 to their corresponding ITSP accounts.
>>>>>>>> >>>>>
>>>>>>>> >>>>> Then I compared the source audio and received audio in
>>>>>>>> Audacity editor.
>>>>>>>> >>>>>
>>>>>>>> >>>>> The tests showed that the received voice wave shifts 20ms
>>>>>>>> from the
>>>>>>>> >>>>> original sound every few seconds, randomly back and forward.
>>>>>>>> This
>>>>>>>> >>>>> skewing was always present, regardless of ITSP and time of
>>>>>>>> day.
>>>>>>>> >>>>>
>>>>>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves
>>>>>>>> appeared
>>>>>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one
>>>>>>>> test, and I
>>>>>>>> >>>>> guess it's related to transcoding buffer, as the sound was
>>>>>>>> obviously
>>>>>>>> >>>>> transcoded).
>>>>>>>> >>>>>
>>>>>>>> >>>>> The effect was always visible with a 30-second sample, and I
>>>>>>>> also made
>>>>>>>> >>>>> additional tests with 2-minute samples.
>>>>>>>> >>>>>
>>>>>>>> >>>>> It would also be interesting to test this from a VmWare VM.
>>>>>>>> If someone
>>>>>>>> >>>>> wants to run such a test, feel free to contact me directly.
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> cheers,
>>>>>>>> >>>>> stan
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> >>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> >>>>> consulting at freeswitch.org
>>>>>>>> >>>>> http://www.freeswitchsolutions.com
>>>>>>>> >>>>>
>>>>>>>> >>>>> Official FreeSWITCH Sites
>>>>>>>> >>>>> http://www.freeswitch.org
>>>>>>>> >>>>> http://confluence.freeswitch.org
>>>>>>>> >>>>> http://www.cluecon.com
>>>>>>>> >>>>>
>>>>>>>> >>>>> FreeSWITCH-users mailing list
>>>>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> >>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> >>>>> http://www.freeswitch.org
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> >>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> >>>> consulting at freeswitch.org
>>>>>>>> >>>> http://www.freeswitchsolutions.com
>>>>>>>> >>>>
>>>>>>>> >>>> Official FreeSWITCH Sites
>>>>>>>> >>>> http://www.freeswitch.org
>>>>>>>> >>>> http://confluence.freeswitch.org
>>>>>>>> >>>> http://www.cluecon.com
>>>>>>>> >>>>
>>>>>>>> >>>> FreeSWITCH-users mailing list
>>>>>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> >>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> >>>> http://www.freeswitch.org
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> >>> Professional FreeSWITCH Consulting Services:
>>>>>>>> >>> consulting at freeswitch.org
>>>>>>>> >>> http://www.freeswitchsolutions.com
>>>>>>>> >>>
>>>>>>>> >>> Official FreeSWITCH Sites
>>>>>>>> >>> http://www.freeswitch.org
>>>>>>>> >>> http://confluence.freeswitch.org
>>>>>>>> >>> http://www.cluecon.com
>>>>>>>> >>>
>>>>>>>> >>> FreeSWITCH-users mailing list
>>>>>>>> >>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> >>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> >>> http://www.freeswitch.org
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> _________________________________________________________________________
>>>>>>>> >> Professional FreeSWITCH Consulting Services:
>>>>>>>> >> consulting at freeswitch.org
>>>>>>>> >> http://www.freeswitchsolutions.com
>>>>>>>> >>
>>>>>>>> >> Official FreeSWITCH Sites
>>>>>>>> >> http://www.freeswitch.org
>>>>>>>> >> http://confluence.freeswitch.org
>>>>>>>> >> http://www.cluecon.com
>>>>>>>> >>
>>>>>>>> >> FreeSWITCH-users mailing list
>>>>>>>> >> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> >> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> >> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150410/ad8a8340/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list