[Freeswitch-users] KVM vs. XEN virtualization
Sergey Safarov
s.safarov at gmail.com
Thu Apr 9 14:37:03 MSD 2015
Hi Giovanni
1) Usb dongles goes to 20ms, and skype too (I wrote it :) ).
I would be pretty sure TDM too.
I has virtualized Digium card and has flowing result
[root at localhost tools]# lspci -k | tail -n 11
03:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev
02)
Subsystem: VMware PVSCSI SCSI Controller
Kernel driver in use: vmw_pvscsi
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
Subsystem: VMware VMXNET3 Ethernet Controller
Kernel driver in use: vmxnet3
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
Subsystem: VMware VMXNET3 Ethernet Controller
Kernel driver in use: vmxnet3
1b:00.0 Network controller: Digium, Inc. Wildcard TE133 single-span
T1/E1/J1 card (PCI Express) (rev 02)
Kernel driver in use: wcte13xp
[root at localhost tools]#
[root at localhost ~]# /usr/sbin/dahdi_scan
[1]
active=yes
alarms=OK
description=Wildcard TE131/TE133 Card 0
name=WCT13x/0
manufacturer=Digium
devicetype=Wildcard TE131/TE133 (VPMOCT032)
location=PCI Bus 27 Slot 01
basechan=1
totchans=31
irq=0
type=digital-E1
syncsrc=1
lbo=0 db (CSU)/0-133 feet (DSX-1)
coding_opts=AMI,HDB3
framing_opts=CCS,CRC4
coding=HDB3
framing=CCS/CRC4
[root at localhost tools]#
/tmp/dahdi-linux-complete-2.10.1+2.10.1/tools/timertest
Opened timer...
Set timer duration to 8000 samples (1000 ms)
Waiting...
Timer Expired (1000 ms)!
Timer Expired (2000 ms)!
Timer Expired (3000 ms)!
Timer Expired (4000 ms)!
Timer Expired (5000 ms)!
[root at localhost rt-tests]# ./cyclictest -t1 -p 80 -d 0 -n -i 125 -l 240000
-q
# /dev/cpu_dma_latency set to 0us
T: 0 ( 1469) P:80 I:125 C: 240000 Min: 3 Act: 22 Avg: 22 Max:
965
2) Not all of those threads will wake up at same time... You probably have
very little threads active with 50 calls, lets say less than 5 if you dont
transcode
My test VM has two cores and only two cyclictest threads may run
simultaneously. Other treads is seeping like FS treads. ;o)
On Wed, Apr 8, 2015 at 7:43 PM, Giovanni Maruzzelli <gmaruzz at gmail.com>
wrote:
> sent from my mobile,
> Giovanni Maruzzelli
> cell: +39 347 266 56 18
> On Apr 8, 2015 2:26 PM, "Sergey Safarov" <s.safarov at gmail.com> wrote:
> >
> > 1) 125uS is the interval between FrameSync of ONE(!) PCM sample at
> 8000hz,
> > But here we are talking about packet telephony where commonly used frame
> length is 20000 uS.
> > So, intervals for testing should be 5000-10000-20000uS, not 125uS.
> >
> > Not all FS installation has SIP endpoint. Some can have TDM cards, USB
> dongle and other analog equipment. Can you say what is length of this frame
> type?
>
> Usb dongles goes to 20ms, and skype too (I wrote it :) ).
>
> I would be pretty sure TDM too.
>
> > I am wondering about this question. I know that the frame 125ms is
> enough.
> >
> > 2) Why you decided that all of 42 threads should wake up simultaneously?
> > That totally impossible in real world applications.
> > Here should be tens of uS. Dependent of network card and, possibly,
> other I/O hardware.
> >
> > Can you say how many FS create threads for 40-50 active calls?
>
> Not all of those threads will wake up at same time... You probably have
> very little threads active with 50 calls, lets say less than 5 if you dont
> transcode
>
> >
> > 3) -q switch is missing
> > I am thinking it is optional. Result is same
> >
> > 4) Precise time measurement and realtime processing in VM environment
> is a challenge.
> > To obtain acceptable results fine tuning of host's and guest's required.
> > WIthout that this numbers ... does not mean anything.
> >
> > Can you learn me how to correctly measure timer source quality in VM?
>
> I would say its by definition variable and impredictable, and depends on
> what else other VMs are doing millisecond by millisecond, and their
> interaction.
> I would test the application, and if percwived quality is not satisfying,
> just throw more resources at it.
>
> >
> > Sergey
> >
> >
> > On Wed, Apr 8, 2015 at 2:07 PM, Dmitry Lysenko <
> dvl36.ripe.nick at gmail.com> wrote:
> >>
> >> 2015-04-07 10:48 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
> >>>
> >>> ...
> >>> 2) i 125 - time us between software timer interrupts for first thread.
> It time is equal time between samples rate in PSTN networks
> >>> In order to maintain our sampling rate we have 125 microseconds (uS)
> between consecutive samples (1/8,000Hz), which is more
> >>>
> >>> ...
> >>
> >>
> >> 125uS is the interval between FrameSync of ONE(!) PCM sample at 8000hz,
> >> But here we are talking about packet telephony where commonly used
> frame length is 20000 uS.
> >> So, intervals for testing should be 5000-10000-20000uS, not 125uS.
> >>
> >>>
> >>> 3) d 0 - this switch will make time between software timer interrupts
> for second and following threads is equal timer value for first thread
> >>
> >>
> >> Why you decided that all of 42 threads should wake up simultaneously?
> >> That totally impossible in real world applications.
> >> Here should be tens of uS. Dependent of network card and, possibly,
> other I/O hardware.
> >>
> >>>
> >>> 4) l 240000 - how many timer test must be made. It is adjusted for 30
> s overall test
> >>> 5) n - use clock_nanosleep. it is allow take more accurate timer test
> values on my VM
> >>> 6) p 80 - it is copied from example on Cyclictest page
> >>
> >>
> >> -q switch is missing.
> >>
> >>>
> >>> And how in your opinion this numbers are related to Freeswitch in VM
> environment?
> >>> Relation between Cyclictest switches and FS described above.
> >>> Relation betwen Cyclictest results and FS simple
> >>
> >>
> >> Precise time measurement and realtime processing in VM environment is
> a challenge.
> >> To obtain acceptable results fine tuning of host's and guest's
> required.
> >> WIthout that this numbers ... does not mean anything.
> >>
> >>
> >>> On Tue, Apr 7, 2015 at 5:38 AM, Dmitry Lysenko <
> dvl36.ripe.nick at gmail.com> wrote:
> >>>>
> >>>> Sergey,
> >>>> Why you suggest to use exactly this numbers in cyclictest's switches?
> >>>> And how in your opinion this numbers are related to Freeswitch in VM
> environment?
> >>>>
> >>>> P.S.
> https://mindlinux.wordpress.com/2013/10/25/using-and-understanding-the-real-time-cyclictest-benchmark-frank-rowand-sony/
> >>>>
> >>>> 2015-04-06 15:27 GMT+03:00 Sergey Safarov <s.safarov at gmail.com>:
> >>>>>
> >>>>> Errol can you test VM timer with utility
> https://rt.wiki.kernel.org/index.php/Cyclictest
> >>>>>
> >>>>> My VM currently have folowing results
> >>>>>
> >>>>> [root at fs1 ~]# date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n
> -i 125 -l 240000 && date
> >>>>> Sun Apr 5 07:01:31 GMT 2015
> >>>>> # /dev/cpu_dma_latency set to 0us
> >>>>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574
> >>>>>
> >>>>> T: 0 ( 1531) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 16
> Max: 1302
> >>>>> T: 1 ( 1532) P:80 I:125 C: 240000 Min: 2 Act: 15 Avg: 14
> Max: 2895
> >>>>> T: 2 ( 1533) P:80 I:125 C: 240000 Min: 3 Act: 10 Avg: 12
> Max: 2888
> >>>>> T: 3 ( 1534) P:80 I:125 C: 240000 Min: 1 Act: 16 Avg: 13
> Max: 1333
> >>>>> T: 4 ( 1535) P:80 I:125 C: 240000 Min: 2 Act: 20 Avg: 14
> Max: 2962
> >>>>> T: 5 ( 1536) P:80 I:125 C: 240000 Min: 1 Act: 12 Avg: 14
> Max: 2946
> >>>>> T: 6 ( 1537) P:80 I:125 C: 240000 Min: 2 Act: 10 Avg: 13
> Max: 1363
> >>>>> T: 7 ( 1538) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 13
> Max: 1344
> >>>>> T: 8 ( 1539) P:80 I:125 C: 240000 Min: 3 Act: 12 Avg: 16
> Max: 1408
> >>>>> T: 9 ( 1540) P:80 I:125 C: 240000 Min: 1 Act: 8 Avg: 13
> Max: 2882
> >>>>> T:10 ( 1541) P:80 I:125 C: 239974 Min: 2 Act: 12 Avg: 13
> Max: 2908
> >>>>>
> >>>>>
> >>>>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <ewsamuels at gmail.com>
> wrote:
> >>>>>>
> >>>>>> Good job and Thanks for sharing!
> >>>>>>
> >>>>>>> KVM at digitalocean.com, Debian 7 64bit, FreeSWITCH 1.4.15: bad
> precision
> >>>>>>
> >>>>>>
> >>>>>> As someone who has been using KVM for a few years and has just
> migrated about 20 VMs across to Digital Ocean about 6 weeks ago I admit I
> am a little concerned about this finding.
> >>>>>>
> >>>>>> Just out of curiosity can you tell me the specs of the VM that you
> tested with and which DO Datacenter? I am using NYC3 so perhaps and I will
> do some tests as soon as I have some time.
> >>>>>>
> >>>>>> With regards to Xen was this at Linode or inhouse?
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> Errol
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin <
> ssinyagin at gmail.com> wrote:
> >>>>>>>
> >>>>>>> here we go:
> >>>>>>>
> https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin <
> ssinyagin at gmail.com> wrote:
> >>>>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M <
> regis.freeswitch.org at tornad.net> wrote:
> >>>>>>> >> How do you check wavs together in audacity ?
> >>>>>>> >
> >>>>>>> > you just open two WAV files, align their starting position, and
> >>>>>>> > visually check the waves. If there's a problem, you would see
> that
> >>>>>>> > after few seconds, the waves are not aligned any more.
> >>>>>>> >
> >>>>>>> >> Can you send me 2 samples wav (send... and recorded) so I could
> see the 20ms
> >>>>>>> >> diff that you exposed...
> >>>>>>> >>
> >>>>>>> >> Is there a way to automate the result checking, so it could
> help to monitor
> >>>>>>> >> itsp quality
> >>>>>>> >
> >>>>>>> > Here
> >>>>>>> > https://github.com/openpreserve/scape-xcorrsound/issues/17
> >>>>>>> > this is a link to a tool which compares two sound files at the
> wave
> >>>>>>> > level. You can also see the sound samples in the ticket.
> >>>>>>> > The problem is, that currently this tool does not detect such
> slipping frames.
> >>>>>>> >
> >>>>>>> > I will also write a detailed article with examples, during the
> weekend.
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin <
> ssinyagin at gmail.com>:
> >>>>>>> >>>
> >>>>>>> >>> Important is to send the media through two independent
> carriers - then you
> >>>>>>> >>> can be sure that your clock is compared against a high quality
> one.
> >>>>>>> >>>
> >>>>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M" <
> regis.freeswitch.org at tornad.net>
> >>>>>>> >>> wrote:
> >>>>>>> >>>>
> >>>>>>> >>>> Hi,
> >>>>>>> >>>>
> >>>>>>> >>>> It's a very interresting test and conclusion.
> >>>>>>> >>>>
> >>>>>>> >>>> I will try to build a similar environnement on my freeswitch
> to check
> >>>>>>> >>>> some system :)
> >>>>>>> >>>>
> >>>>>>> >>>> thanks
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin <
> ssinyagin at gmail.com>:
> >>>>>>> >>>>>
> >>>>>>> >>>>> hi,
> >>>>>>> >>>>>
> >>>>>>> >>>>> Here are some test results which might be interesting:
> >>>>>>> >>>>>
> >>>>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they
> use KVM),
> >>>>>>> >>>>> and originated test calls to some ITSP accounts. Each call
> played a
> >>>>>>> >>>>> test sound file (voice samples from ITU:
> >>>>>>> >>>>>
> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050
> >>>>>>> >>>>> ), and on the receiving end another FreeSWITCH server was
> recording
> >>>>>>> >>>>> the received session. The receiving server was on a XEN VM.
> Both
> >>>>>>> >>>>> servers talked G711 to their corresponding ITSP accounts.
> >>>>>>> >>>>>
> >>>>>>> >>>>> Then I compared the source audio and received audio in
> Audacity editor.
> >>>>>>> >>>>>
> >>>>>>> >>>>> The tests showed that the received voice wave shifts 20ms
> from the
> >>>>>>> >>>>> original sound every few seconds, randomly back and forward.
> This
> >>>>>>> >>>>> skewing was always present, regardless of ITSP and time of
> day.
> >>>>>>> >>>>>
> >>>>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves
> appeared
> >>>>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one
> test, and I
> >>>>>>> >>>>> guess it's related to transcoding buffer, as the sound was
> obviously
> >>>>>>> >>>>> transcoded).
> >>>>>>> >>>>>
> >>>>>>> >>>>> The effect was always visible with a 30-second sample, and I
> also made
> >>>>>>> >>>>> additional tests with 2-minute samples.
> >>>>>>> >>>>>
> >>>>>>> >>>>> It would also be interesting to test this from a VmWare VM.
> If someone
> >>>>>>> >>>>> wants to run such a test, feel free to contact me directly.
> >>>>>>> >>>>>
> >>>>>>> >>>>>
> >>>>>>> >>>>> cheers,
> >>>>>>> >>>>> stan
> >>>>>>> >>>>>
> >>>>>>> >>>>>
> >>>>>>> >>>>>
> _________________________________________________________________________
> >>>>>>> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>>>> >>>>> consulting at freeswitch.org
> >>>>>>> >>>>> http://www.freeswitchsolutions.com
> >>>>>>> >>>>>
> >>>>>>> >>>>> Official FreeSWITCH Sites
> >>>>>>> >>>>> http://www.freeswitch.org
> >>>>>>> >>>>> http://confluence.freeswitch.org
> >>>>>>> >>>>> http://www.cluecon.com
> >>>>>>> >>>>>
> >>>>>>> >>>>> FreeSWITCH-users mailing list
> >>>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>>> >>>>>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>>> >>>>> http://www.freeswitch.org
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>>
> >>>>>>> >>>>
> _________________________________________________________________________
> >>>>>>> >>>> Professional FreeSWITCH Consulting Services:
> >>>>>>> >>>> consulting at freeswitch.org
> >>>>>>> >>>> http://www.freeswitchsolutions.com
> >>>>>>> >>>>
> >>>>>>> >>>> Official FreeSWITCH Sites
> >>>>>>> >>>> http://www.freeswitch.org
> >>>>>>> >>>> http://confluence.freeswitch.org
> >>>>>>> >>>> http://www.cluecon.com
> >>>>>>> >>>>
> >>>>>>> >>>> FreeSWITCH-users mailing list
> >>>>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>>> >>>> http://www.freeswitch.org
> >>>>>>> >>>
> >>>>>>> >>>
> >>>>>>> >>>
> _________________________________________________________________________
> >>>>>>> >>> Professional FreeSWITCH Consulting Services:
> >>>>>>> >>> consulting at freeswitch.org
> >>>>>>> >>> http://www.freeswitchsolutions.com
> >>>>>>> >>>
> >>>>>>> >>> Official FreeSWITCH Sites
> >>>>>>> >>> http://www.freeswitch.org
> >>>>>>> >>> http://confluence.freeswitch.org
> >>>>>>> >>> http://www.cluecon.com
> >>>>>>> >>>
> >>>>>>> >>> FreeSWITCH-users mailing list
> >>>>>>> >>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>>> >>> http://www.freeswitch.org
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >>
> >>>>>>> >>
> _________________________________________________________________________
> >>>>>>> >> Professional FreeSWITCH Consulting Services:
> >>>>>>> >> consulting at freeswitch.org
> >>>>>>> >> http://www.freeswitchsolutions.com
> >>>>>>> >>
> >>>>>>> >> Official FreeSWITCH Sites
> >>>>>>> >> http://www.freeswitch.org
> >>>>>>> >> http://confluence.freeswitch.org
> >>>>>>> >> http://www.cluecon.com
> >>>>>>> >>
> >>>>>>> >> FreeSWITCH-users mailing list
> >>>>>>> >> FreeSWITCH-users at lists.freeswitch.org
> >>>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>>> >> http://www.freeswitch.org
> >>>>>>>
> >>>>>>>
> _________________________________________________________________________
> >>>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>>> consulting at freeswitch.org
> >>>>>>> http://www.freeswitchsolutions.com
> >>>>>>>
> >>>>>>> Official FreeSWITCH Sites
> >>>>>>> http://www.freeswitch.org
> >>>>>>> http://confluence.freeswitch.org
> >>>>>>> http://www.cluecon.com
> >>>>>>>
> >>>>>>> FreeSWITCH-users mailing list
> >>>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>>> http://www.freeswitch.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> _________________________________________________________________________
> >>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> consulting at freeswitch.org
> >>>>>> http://www.freeswitchsolutions.com
> >>>>>>
> >>>>>> Official FreeSWITCH Sites
> >>>>>> http://www.freeswitch.org
> >>>>>> http://confluence.freeswitch.org
> >>>>>> http://www.cluecon.com
> >>>>>>
> >>>>>> FreeSWITCH-users mailing list
> >>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://confluence.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>>
> >>>>
> _________________________________________________________________________
> >>>> Professional FreeSWITCH Consulting Services:
> >>>> consulting at freeswitch.org
> >>>> http://www.freeswitchsolutions.com
> >>>>
> >>>> Official FreeSWITCH Sites
> >>>> http://www.freeswitch.org
> >>>> http://confluence.freeswitch.org
> >>>> http://www.cluecon.com
> >>>>
> >>>> FreeSWITCH-users mailing list
> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> >>>
> >>>
> >>>
> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org
> >>> http://www.freeswitchsolutions.com
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://confluence.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >>
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://confluence.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150409/09b2b9ac/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list