[Freeswitch-users] KVM vs. XEN virtualization

Brian West brian at freeswitch.org
Mon Apr 6 20:05:54 MSD 2015


I've never had issues running under KVM on Proxmox, Looking back at this
thread was there a specific issue that you're trying to solve that put you
on to this path of testing?

On Mon, Apr 6, 2015 at 10:51 AM, Stanislav Sinyagin <ssinyagin at gmail.com>
wrote:

> yes, and we're trying to find the one that works with minimal impact
> on quality  :)
>
> I'm thinking to move my test server (the one on receiving side) from
> Xen VM to a physical ARM machine at https://www.scaleway.com/  in
> France. It would record audio and raw UDP packets and provide them for
> downloading to everyone.
>
> Do we know about any timing issues on ARM platform?
>
> if someone sponsors some additional DID numbers, that will be appreciated.
>
>
>
>
>
>
> On Mon, Apr 6, 2015 at 5:30 PM, Anthony Minessale
> <anthony.minessale at gmail.com> wrote:
> > Depending on the host and guest OS the timing will be a product of
> several
> > thing.
> > If some of the features FS tries to detect are emulated, it can be a
> > problem.
> >
> > The primary thing FS looks for is a kernel with timerfd support.
> > This should be the most accurate.  If the guest OS supports timerfd but
> its
> > using emulated hardware, it might not work as well as it should.
> >
> > So on something like openvz, you might have support for timerfd in your
> > debian 7/8 instance but the host kernel has no support at all.
> > So in theory kvm would be preferred since it actually boots the guest
> kernel
> > and you have a chance for it to hit the real timer hardware.
> >
> > If no timerfd is present, it reverts to the single thread ms precision
> > thread with conditional broadcast for timing.
> > You can also manually configure a combo of one timerfd timer and shared
> > broadcast.
> >
> > This param in switch.conf.xml helps to tune a few behavior you might
> want to
> > compare results on.
> >
> > enable-softtimer-timerfd
> >
> > true|fd-per-timer  --- Use one fd for each timer instance (default)
> >
> > false                    --- Don't use timerfd at all
> >
> > broadcast            --- Use one timerfd and share it with conditional
> > broadcasts
> >
> > This is one reason we do not support virtualization as its too variable
> to
> > provide definitive policies to make things work every time.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Apr 6, 2015 at 10:17 AM, Stanislav Sinyagin <ssinyagin at gmail.com
> >
> > wrote:
> >>
> >> I didn't yet have time to learn this utility - how do we interpret the
> >> results?
> >>
> >> On Apr 6, 2015 4:39 PM, "Errol Samuels" <ewsamuels at gmail.com> wrote:
> >>>
> >>> Ok Sergey,
> >>>
> >>> Here it goes:
> >>>
> >>> Test 1
> >>> ====
> >>> VM: Digital Ocean KVM at AMS3 / 2 Cores / 2GB Ram / 40GB SSD
> >>> OS: Debian Wheezy 7.8 64 bit / Linux 3.2.0-4-amd64 x86_64
> >>>
> >>> policy: fifo: loadavg: 6.22 1.53 0.58 1/164 31050          :  195 Max:
> >>> 42869
> >>> T:12 (31021) P:80 I:125 C: 238830 Min:      2 Act:   18 Avg:  196 Max:
> >>> 42846
> >>> T: 0 (31009) P:80 I:125 C: 239000 Min:      2 Act:   12 Avg:  190 Max:
> >>> 42845
> >>> T: 1 (31010) P:80 I:125 C: 239002 Min:      2 Act:   18 Avg:  196 Max:
> >>> 42866
> >>> T: 2 (31011) P:80 I:125 C: 239004 Min:      3 Act:   10 Avg:  191 Max:
> >>> 42890
> >>> T: 3 (31012) P:80 I:125 C: 239004 Min:      3 Act:    8 Avg:  197 Max:
> >>> 42876
> >>> T: 4 (31013) P:80 I:125 C: 239005 Min:      2 Act:   27 Avg:  194 Max:
> >>> 42846
> >>> T: 5 (31014) P:80 I:125 C: 239008 Min:      2 Act:   26 Avg:  196 Max:
> >>> 42872
> >>> T: 6 (31015) P:80 I:125 C: 239013 Min:      2 Act:   12 Avg:  196 Max:
> >>> 42843
> >>> T: 7 (31016) P:80 I:125 C: 239013 Min:      2 Act:   19 Avg:  194 Max:
> >>> 42848
> >>> T: 8 (31017) P:80 I:125 C: 239013 Min:      2 Act:   14 Avg:  190 Max:
> >>> 42856
> >>> T: 9 (31018) P:80 I:125 C: 239014 Min:      2 Act:   14 Avg:  197 Max:
> >>> 42865
> >>> T:10 (31019) P:80 I:125 C: 239016 Min:      2 Act:    9 Avg:  197 Max:
> >>> 42868
> >>>
> >>>
> >>> Test 2
> >>> ====
> >>> VM: Linode (Xen) London / 8 Cores (E5-2670 0 @ 2.60GHz) / 4GB Ram
> >>> OS: Ubuntu 12.04.4 LTS / Linux 3.18.5-x86_64-linode52 x86_64
> >>>
> >>> policy: fifo: loadavg: 6.50 1.56 0.55 15/246 30752             97 Max:
> >>> 1030
> >>> T:16 (30726) P:80 I:125 C: 239853 Min:      5 Act:   77 Avg:   97 Max:
> >>> 1066
> >>> T: 0 (30710) P:80 I:125 C: 239946 Min:      5 Act:   63 Avg:   98 Max:
> >>> 1245
> >>> T: 1 (30711) P:80 I:125 C: 239946 Min:      5 Act:   67 Avg:   98 Max:
> >>> 1091
> >>> T: 2 (30712) P:80 I:125 C: 239945 Min:      5 Act:  119 Avg:   97 Max:
> >>> 1351
> >>> T: 3 (30713) P:80 I:125 C: 239945 Min:      5 Act:   99 Avg:   98 Max:
> >>> 932
> >>> T: 4 (30714) P:80 I:125 C: 239945 Min:      5 Act:   35 Avg:   98 Max:
> >>> 1413
> >>> T: 5 (30715) P:80 I:125 C: 239945 Min:      5 Act:   35 Avg:   98 Max:
> >>> 1407
> >>> T: 6 (30716) P:80 I:125 C: 239944 Min:      5 Act:  110 Avg:   97 Max:
> >>> 1173
> >>> T: 7 (30717) P:80 I:125 C: 239947 Min:      5 Act:   85 Avg:   99 Max:
> >>> 1144
> >>> T: 8 (30718) P:80 I:125 C: 239948 Min:      4 Act:   69 Avg:   99 Max:
> >>> 916
> >>> T: 9 (30719) P:80 I:125 C: 239947 Min:      4 Act:   40 Avg:   97 Max:
> >>> 916
> >>> T:10 (30720) P:80 I:125 C: 239948 Min:      5 Act:   61 Avg:   97 Max:
> >>> 1024
> >>> T:11 (30721) P:80 I:125 C: 239946 Min:      6 Act:   84 Avg:   98 Max:
> >>> 2082
> >>> T:12 (30722) P:80 I:125 C: 239947 Min:      4 Act:   40 Avg:   98 Max:
> >>> 2142
> >>> T:13 (30723) P:80 I:125 C: 239946 Min:      5 Act:   49 Avg:   98 Max:
> >>> 1986
> >>> T:14 (30724) P:80 I:125 C: 239946 Min:      4 Act:  113 Avg:   99 Max:
> >>> 2061
> >>>
> >>> This is with system defaults.
> >>>
> >>> regards,
> >>>
> >>> Errol
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Apr 6, 2015 at 1:27 PM, Sergey Safarov <s.safarov at gmail.com>
> >>> wrote:
> >>>>
> >>>> Errol can you test VM timer with utility
> >>>> https://rt.wiki.kernel.org/index.php/Cyclictest
> >>>>
> >>>> My VM currently have folowing results
> >>>>
> >>>> [root at fs1 ~]#  date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i
> >>>> 125 -l 240000 && date
> >>>> Sun Apr  5 07:01:31 GMT 2015
> >>>> # /dev/cpu_dma_latency set to 0us
> >>>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574
> >>>>
> >>>> T: 0 ( 1531) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   16 Max:
> >>>> 1302
> >>>> T: 1 ( 1532) P:80 I:125 C: 240000 Min:      2 Act:   15 Avg:   14 Max:
> >>>> 2895
> >>>> T: 2 ( 1533) P:80 I:125 C: 240000 Min:      3 Act:   10 Avg:   12 Max:
> >>>> 2888
> >>>> T: 3 ( 1534) P:80 I:125 C: 240000 Min:      1 Act:   16 Avg:   13 Max:
> >>>> 1333
> >>>> T: 4 ( 1535) P:80 I:125 C: 240000 Min:      2 Act:   20 Avg:   14 Max:
> >>>> 2962
> >>>> T: 5 ( 1536) P:80 I:125 C: 240000 Min:      1 Act:   12 Avg:   14 Max:
> >>>> 2946
> >>>> T: 6 ( 1537) P:80 I:125 C: 240000 Min:      2 Act:   10 Avg:   13 Max:
> >>>> 1363
> >>>> T: 7 ( 1538) P:80 I:125 C: 240000 Min:      2 Act:   12 Avg:   13 Max:
> >>>> 1344
> >>>> T: 8 ( 1539) P:80 I:125 C: 240000 Min:      3 Act:   12 Avg:   16 Max:
> >>>> 1408
> >>>> T: 9 ( 1540) P:80 I:125 C: 240000 Min:      1 Act:    8 Avg:   13 Max:
> >>>> 2882
> >>>> T:10 ( 1541) P:80 I:125 C: 239974 Min:      2 Act:   12 Avg:   13 Max:
> >>>> 2908
> >>>>
> >>>>
> >>>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <ewsamuels at gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Good job and Thanks for sharing!
> >>>>>
> >>>>>> KVM at digitalocean.com, Debian 7 64bit, FreeSWITCH 1.4.15: bad
> >>>>>> precision
> >>>>>
> >>>>>
> >>>>> As someone who has been using KVM for a few years and has just
> migrated
> >>>>> about 20 VMs across to Digital Ocean about 6 weeks ago I admit I am
> a little
> >>>>> concerned about this finding.
> >>>>>
> >>>>> Just out of curiosity can you tell me the specs of the VM that you
> >>>>> tested with and which DO Datacenter?  I am using NYC3 so perhaps and
> I will
> >>>>> do some tests as soon as I have some time.
> >>>>>
> >>>>> With regards to Xen was this at Linode or inhouse?
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Errol
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin
> >>>>> <ssinyagin at gmail.com> wrote:
> >>>>>>
> >>>>>> here we go:
> >>>>>>
> >>>>>>
> https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin
> >>>>>> <ssinyagin at gmail.com> wrote:
> >>>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M
> >>>>>> > <regis.freeswitch.org at tornad.net> wrote:
> >>>>>> >> How do you check wavs together in audacity ?
> >>>>>> >
> >>>>>> > you just open two WAV files, align their starting position, and
> >>>>>> > visually check the waves. If there's a problem, you would see that
> >>>>>> > after few seconds, the waves are not aligned any more.
> >>>>>> >
> >>>>>> >> Can you send me 2 samples wav (send... and recorded) so I could
> see
> >>>>>> >> the 20ms
> >>>>>> >> diff that you exposed...
> >>>>>> >>
> >>>>>> >> Is there a way to automate the result checking, so it could help
> to
> >>>>>> >> monitor
> >>>>>> >> itsp quality
> >>>>>> >
> >>>>>> > Here
> >>>>>> > https://github.com/openpreserve/scape-xcorrsound/issues/17
> >>>>>> > this is a link to a tool which compares two sound files at the
> wave
> >>>>>> > level. You can also see the sound samples in the ticket.
> >>>>>> > The problem is, that currently this tool does not detect such
> >>>>>> > slipping frames.
> >>>>>> >
> >>>>>> > I will also write a detailed article with examples, during the
> >>>>>> > weekend.
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> >
> >>>>>> >>
> >>>>>> >>
> >>>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin
> >>>>>> >> <ssinyagin at gmail.com>:
> >>>>>> >>>
> >>>>>> >>> Important is to send the media through two independent carriers
> -
> >>>>>> >>> then you
> >>>>>> >>> can be sure that your clock is compared against a high quality
> >>>>>> >>> one.
> >>>>>> >>>
> >>>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M"
> >>>>>> >>> <regis.freeswitch.org at tornad.net>
> >>>>>> >>> wrote:
> >>>>>> >>>>
> >>>>>> >>>> Hi,
> >>>>>> >>>>
> >>>>>> >>>> It's a very interresting test and conclusion.
> >>>>>> >>>>
> >>>>>> >>>> I will try to build a similar environnement on my freeswitch to
> >>>>>> >>>> check
> >>>>>> >>>> some system :)
> >>>>>> >>>>
> >>>>>> >>>> thanks
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin
> >>>>>> >>>> <ssinyagin at gmail.com>:
> >>>>>> >>>>>
> >>>>>> >>>>> hi,
> >>>>>> >>>>>
> >>>>>> >>>>> Here are some test results which might be interesting:
> >>>>>> >>>>>
> >>>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they use
> >>>>>> >>>>> KVM),
> >>>>>> >>>>> and originated test calls to some ITSP accounts. Each call
> >>>>>> >>>>> played a
> >>>>>> >>>>> test sound file (voice samples from ITU:
> >>>>>> >>>>>
> >>>>>> >>>>>
> http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050
> >>>>>> >>>>> ), and on the receiving end another FreeSWITCH server was
> >>>>>> >>>>> recording
> >>>>>> >>>>> the received session. The receiving server was on a XEN VM.
> Both
> >>>>>> >>>>> servers talked G711 to their corresponding ITSP accounts.
> >>>>>> >>>>>
> >>>>>> >>>>> Then I compared the source audio and received audio in
> Audacity
> >>>>>> >>>>> editor.
> >>>>>> >>>>>
> >>>>>> >>>>> The tests showed that the received voice wave shifts 20ms from
> >>>>>> >>>>> the
> >>>>>> >>>>> original sound every few seconds, randomly back and forward.
> >>>>>> >>>>> This
> >>>>>> >>>>> skewing was always present, regardless of ITSP and time of
> day.
> >>>>>> >>>>>
> >>>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves
> >>>>>> >>>>> appeared
> >>>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one
> test,
> >>>>>> >>>>> and I
> >>>>>> >>>>> guess it's related to transcoding buffer, as the sound was
> >>>>>> >>>>> obviously
> >>>>>> >>>>> transcoded).
> >>>>>> >>>>>
> >>>>>> >>>>> The effect was always visible with a 30-second sample, and I
> >>>>>> >>>>> also made
> >>>>>> >>>>> additional tests with 2-minute samples.
> >>>>>> >>>>>
> >>>>>> >>>>> It would also be interesting to test this from a VmWare VM. If
> >>>>>> >>>>> someone
> >>>>>> >>>>> wants to run such a test, feel free to contact me directly.
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>> cheers,
> >>>>>> >>>>> stan
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>>
> _________________________________________________________________________
> >>>>>> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> >>>>> consulting at freeswitch.org
> >>>>>> >>>>> http://www.freeswitchsolutions.com
> >>>>>> >>>>>
> >>>>>> >>>>> Official FreeSWITCH Sites
> >>>>>> >>>>> http://www.freeswitch.org
> >>>>>> >>>>> http://confluence.freeswitch.org
> >>>>>> >>>>> http://www.cluecon.com
> >>>>>> >>>>>
> >>>>>> >>>>> FreeSWITCH-users mailing list
> >>>>>> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> >>>>>
> >>>>>> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> >>>>> http://www.freeswitch.org
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>
> _________________________________________________________________________
> >>>>>> >>>> Professional FreeSWITCH Consulting Services:
> >>>>>> >>>> consulting at freeswitch.org
> >>>>>> >>>> http://www.freeswitchsolutions.com
> >>>>>> >>>>
> >>>>>> >>>> Official FreeSWITCH Sites
> >>>>>> >>>> http://www.freeswitch.org
> >>>>>> >>>> http://confluence.freeswitch.org
> >>>>>> >>>> http://www.cluecon.com
> >>>>>> >>>>
> >>>>>> >>>> FreeSWITCH-users mailing list
> >>>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> >>>>
> >>>>>> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> >>>> http://www.freeswitch.org
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>>
> _________________________________________________________________________
> >>>>>> >>> Professional FreeSWITCH Consulting Services:
> >>>>>> >>> consulting at freeswitch.org
> >>>>>> >>> http://www.freeswitchsolutions.com
> >>>>>> >>>
> >>>>>> >>> Official FreeSWITCH Sites
> >>>>>> >>> http://www.freeswitch.org
> >>>>>> >>> http://confluence.freeswitch.org
> >>>>>> >>> http://www.cluecon.com
> >>>>>> >>>
> >>>>>> >>> FreeSWITCH-users mailing list
> >>>>>> >>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> >>>
> >>>>>> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> >>> http://www.freeswitch.org
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> _________________________________________________________________________
> >>>>>> >> Professional FreeSWITCH Consulting Services:
> >>>>>> >> consulting at freeswitch.org
> >>>>>> >> http://www.freeswitchsolutions.com
> >>>>>> >>
> >>>>>> >> Official FreeSWITCH Sites
> >>>>>> >> http://www.freeswitch.org
> >>>>>> >> http://confluence.freeswitch.org
> >>>>>> >> http://www.cluecon.com
> >>>>>> >>
> >>>>>> >> FreeSWITCH-users mailing list
> >>>>>> >> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> >>
> >>>>>> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> >> http://www.freeswitch.org
> >>>>>>
> >>>>>>
> >>>>>>
> _________________________________________________________________________
> >>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> consulting at freeswitch.org
> >>>>>> http://www.freeswitchsolutions.com
> >>>>>>
> >>>>>> Official FreeSWITCH Sites
> >>>>>> http://www.freeswitch.org
> >>>>>> http://confluence.freeswitch.org
> >>>>>> http://www.cluecon.com
> >>>>>>
> >>>>>> FreeSWITCH-users mailing list
> >>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>>
> >>>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> http://www.freeswitch.org
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://confluence.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>
> >>>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> _________________________________________________________________________
> >>>> Professional FreeSWITCH Consulting Services:
> >>>> consulting at freeswitch.org
> >>>> http://www.freeswitchsolutions.com
> >>>>
> >>>> Official FreeSWITCH Sites
> >>>> http://www.freeswitch.org
> >>>> http://confluence.freeswitch.org
> >>>> http://www.cluecon.com
> >>>>
> >>>> FreeSWITCH-users mailing list
> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> >>>
> >>>
> >>>
> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org
> >>> http://www.freeswitchsolutions.com
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://confluence.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >>
> >>
> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://confluence.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:
> http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
> >
> >
> >
> > --
> > Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
> >
> > ☞ http://freeswitch.org/http://cluecon.com/> > http://twitter.com/FreeSWITCH
> > ☞ irc.freenode.net #freeswitch ☞ http://freeswitch.org/g+
> >
> > ClueCon Weekly Development Call
> > ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
> >
> > https://www.youtube.com/watch?v=9XXgW34t40s
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://confluence.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>



-- 

*Brian West*
brian at freeswitch.org


*Twitter: @FreeSWITCH , @briankwest*
http://www.freeswitchbook.com
http://www.freeswitchcookbook.com

ClueCon 2015 Call for Speakers <https://www.cluecon.com/call-for-speakers/> |
Register <https://freeswitch.com/cart.php?gid=1> TODAY! | Reddit:
/r/freeswitch <https://www.reddit.com/r/freeswitch>

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378)
*iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150406/17c69754/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list