<div dir="ltr">I've never had issues running under KVM on Proxmox, Looking back at this thread was there a specific issue that you're trying to solve that put you on to this path of testing?</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 6, 2015 at 10:51 AM, Stanislav Sinyagin <span dir="ltr"><<a href="mailto:ssinyagin@gmail.com" target="_blank">ssinyagin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">yes, and we're trying to find the one that works with minimal impact<br>
on quality :)<br>
<br>
I'm thinking to move my test server (the one on receiving side) from<br>
Xen VM to a physical ARM machine at <a href="https://www.scaleway.com/" target="_blank">https://www.scaleway.com/</a> in<br>
France. It would record audio and raw UDP packets and provide them for<br>
downloading to everyone.<br>
<br>
Do we know about any timing issues on ARM platform?<br>
<br>
if someone sponsors some additional DID numbers, that will be appreciated.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
On Mon, Apr 6, 2015 at 5:30 PM, Anthony Minessale<br>
<<a href="mailto:anthony.minessale@gmail.com">anthony.minessale@gmail.com</a>> wrote:<br>
> Depending on the host and guest OS the timing will be a product of several<br>
> thing.<br>
> If some of the features FS tries to detect are emulated, it can be a<br>
> problem.<br>
><br>
> The primary thing FS looks for is a kernel with timerfd support.<br>
> This should be the most accurate. If the guest OS supports timerfd but its<br>
> using emulated hardware, it might not work as well as it should.<br>
><br>
> So on something like openvz, you might have support for timerfd in your<br>
> debian 7/8 instance but the host kernel has no support at all.<br>
> So in theory kvm would be preferred since it actually boots the guest kernel<br>
> and you have a chance for it to hit the real timer hardware.<br>
><br>
> If no timerfd is present, it reverts to the single thread ms precision<br>
> thread with conditional broadcast for timing.<br>
> You can also manually configure a combo of one timerfd timer and shared<br>
> broadcast.<br>
><br>
> This param in switch.conf.xml helps to tune a few behavior you might want to<br>
> compare results on.<br>
><br>
> enable-softtimer-timerfd<br>
><br>
> true|fd-per-timer --- Use one fd for each timer instance (default)<br>
><br>
> false --- Don't use timerfd at all<br>
><br>
> broadcast --- Use one timerfd and share it with conditional<br>
> broadcasts<br>
><br>
> This is one reason we do not support virtualization as its too variable to<br>
> provide definitive policies to make things work every time.<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> On Mon, Apr 6, 2015 at 10:17 AM, Stanislav Sinyagin <<a href="mailto:ssinyagin@gmail.com">ssinyagin@gmail.com</a>><br>
> wrote:<br>
>><br>
>> I didn't yet have time to learn this utility - how do we interpret the<br>
>> results?<br>
>><br>
>> On Apr 6, 2015 4:39 PM, "Errol Samuels" <<a href="mailto:ewsamuels@gmail.com">ewsamuels@gmail.com</a>> wrote:<br>
>>><br>
>>> Ok Sergey,<br>
>>><br>
>>> Here it goes:<br>
>>><br>
>>> Test 1<br>
>>> ====<br>
>>> VM: Digital Ocean KVM at AMS3 / 2 Cores / 2GB Ram / 40GB SSD<br>
>>> OS: Debian Wheezy 7.8 64 bit / Linux 3.2.0-4-amd64 x86_64<br>
>>><br>
>>> policy: fifo: loadavg: 6.22 1.53 0.58 1/164 31050 : 195 Max:<br>
>>> 42869<br>
>>> T:12 (31021) P:80 I:125 C: 238830 Min: 2 Act: 18 Avg: 196 Max:<br>
>>> 42846<br>
>>> T: 0 (31009) P:80 I:125 C: 239000 Min: 2 Act: 12 Avg: 190 Max:<br>
>>> 42845<br>
>>> T: 1 (31010) P:80 I:125 C: 239002 Min: 2 Act: 18 Avg: 196 Max:<br>
>>> 42866<br>
>>> T: 2 (31011) P:80 I:125 C: 239004 Min: 3 Act: 10 Avg: 191 Max:<br>
>>> 42890<br>
>>> T: 3 (31012) P:80 I:125 C: 239004 Min: 3 Act: 8 Avg: 197 Max:<br>
>>> 42876<br>
>>> T: 4 (31013) P:80 I:125 C: 239005 Min: 2 Act: 27 Avg: 194 Max:<br>
>>> 42846<br>
>>> T: 5 (31014) P:80 I:125 C: 239008 Min: 2 Act: 26 Avg: 196 Max:<br>
>>> 42872<br>
>>> T: 6 (31015) P:80 I:125 C: 239013 Min: 2 Act: 12 Avg: 196 Max:<br>
>>> 42843<br>
>>> T: 7 (31016) P:80 I:125 C: 239013 Min: 2 Act: 19 Avg: 194 Max:<br>
>>> 42848<br>
>>> T: 8 (31017) P:80 I:125 C: 239013 Min: 2 Act: 14 Avg: 190 Max:<br>
>>> 42856<br>
>>> T: 9 (31018) P:80 I:125 C: 239014 Min: 2 Act: 14 Avg: 197 Max:<br>
>>> 42865<br>
>>> T:10 (31019) P:80 I:125 C: 239016 Min: 2 Act: 9 Avg: 197 Max:<br>
>>> 42868<br>
>>><br>
>>><br>
>>> Test 2<br>
>>> ====<br>
>>> VM: Linode (Xen) London / 8 Cores (E5-2670 0 @ 2.60GHz) / 4GB Ram<br>
>>> OS: Ubuntu 12.04.4 LTS / Linux 3.18.5-x86_64-linode52 x86_64<br>
>>><br>
>>> policy: fifo: loadavg: 6.50 1.56 0.55 15/246 30752 97 Max:<br>
>>> 1030<br>
>>> T:16 (30726) P:80 I:125 C: 239853 Min: 5 Act: 77 Avg: 97 Max:<br>
>>> 1066<br>
>>> T: 0 (30710) P:80 I:125 C: 239946 Min: 5 Act: 63 Avg: 98 Max:<br>
>>> 1245<br>
>>> T: 1 (30711) P:80 I:125 C: 239946 Min: 5 Act: 67 Avg: 98 Max:<br>
>>> 1091<br>
>>> T: 2 (30712) P:80 I:125 C: 239945 Min: 5 Act: 119 Avg: 97 Max:<br>
>>> 1351<br>
>>> T: 3 (30713) P:80 I:125 C: 239945 Min: 5 Act: 99 Avg: 98 Max:<br>
>>> 932<br>
>>> T: 4 (30714) P:80 I:125 C: 239945 Min: 5 Act: 35 Avg: 98 Max:<br>
>>> 1413<br>
>>> T: 5 (30715) P:80 I:125 C: 239945 Min: 5 Act: 35 Avg: 98 Max:<br>
>>> 1407<br>
>>> T: 6 (30716) P:80 I:125 C: 239944 Min: 5 Act: 110 Avg: 97 Max:<br>
>>> 1173<br>
>>> T: 7 (30717) P:80 I:125 C: 239947 Min: 5 Act: 85 Avg: 99 Max:<br>
>>> 1144<br>
>>> T: 8 (30718) P:80 I:125 C: 239948 Min: 4 Act: 69 Avg: 99 Max:<br>
>>> 916<br>
>>> T: 9 (30719) P:80 I:125 C: 239947 Min: 4 Act: 40 Avg: 97 Max:<br>
>>> 916<br>
>>> T:10 (30720) P:80 I:125 C: 239948 Min: 5 Act: 61 Avg: 97 Max:<br>
>>> 1024<br>
>>> T:11 (30721) P:80 I:125 C: 239946 Min: 6 Act: 84 Avg: 98 Max:<br>
>>> 2082<br>
>>> T:12 (30722) P:80 I:125 C: 239947 Min: 4 Act: 40 Avg: 98 Max:<br>
>>> 2142<br>
>>> T:13 (30723) P:80 I:125 C: 239946 Min: 5 Act: 49 Avg: 98 Max:<br>
>>> 1986<br>
>>> T:14 (30724) P:80 I:125 C: 239946 Min: 4 Act: 113 Avg: 99 Max:<br>
>>> 2061<br>
>>><br>
>>> This is with system defaults.<br>
>>><br>
>>> regards,<br>
>>><br>
>>> Errol<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> On Mon, Apr 6, 2015 at 1:27 PM, Sergey Safarov <<a href="mailto:s.safarov@gmail.com">s.safarov@gmail.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Errol can you test VM timer with utility<br>
>>>> <a href="https://rt.wiki.kernel.org/index.php/Cyclictest" target="_blank">https://rt.wiki.kernel.org/index.php/Cyclictest</a><br>
>>>><br>
>>>> My VM currently have folowing results<br>
>>>><br>
>>>> [root@fs1 ~]# date && /tmp/rt-tests/cyclictest -t42 -p 80 -d 0 -n -i<br>
>>>> 125 -l 240000 && date<br>
>>>> Sun Apr 5 07:01:31 GMT 2015<br>
>>>> # /dev/cpu_dma_latency set to 0us<br>
>>>> policy: fifo: loadavg: 2.38 0.93 0.50 1/194 1574<br>
>>>><br>
>>>> T: 0 ( 1531) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 16 Max:<br>
>>>> 1302<br>
>>>> T: 1 ( 1532) P:80 I:125 C: 240000 Min: 2 Act: 15 Avg: 14 Max:<br>
>>>> 2895<br>
>>>> T: 2 ( 1533) P:80 I:125 C: 240000 Min: 3 Act: 10 Avg: 12 Max:<br>
>>>> 2888<br>
>>>> T: 3 ( 1534) P:80 I:125 C: 240000 Min: 1 Act: 16 Avg: 13 Max:<br>
>>>> 1333<br>
>>>> T: 4 ( 1535) P:80 I:125 C: 240000 Min: 2 Act: 20 Avg: 14 Max:<br>
>>>> 2962<br>
>>>> T: 5 ( 1536) P:80 I:125 C: 240000 Min: 1 Act: 12 Avg: 14 Max:<br>
>>>> 2946<br>
>>>> T: 6 ( 1537) P:80 I:125 C: 240000 Min: 2 Act: 10 Avg: 13 Max:<br>
>>>> 1363<br>
>>>> T: 7 ( 1538) P:80 I:125 C: 240000 Min: 2 Act: 12 Avg: 13 Max:<br>
>>>> 1344<br>
>>>> T: 8 ( 1539) P:80 I:125 C: 240000 Min: 3 Act: 12 Avg: 16 Max:<br>
>>>> 1408<br>
>>>> T: 9 ( 1540) P:80 I:125 C: 240000 Min: 1 Act: 8 Avg: 13 Max:<br>
>>>> 2882<br>
>>>> T:10 ( 1541) P:80 I:125 C: 239974 Min: 2 Act: 12 Avg: 13 Max:<br>
>>>> 2908<br>
>>>><br>
>>>><br>
>>>> On Mon, Apr 6, 2015 at 11:15 AM, Errol Samuels <<a href="mailto:ewsamuels@gmail.com">ewsamuels@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> Good job and Thanks for sharing!<br>
>>>>><br>
>>>>>> KVM at <a href="http://digitalocean.com" target="_blank">digitalocean.com</a>, Debian 7 64bit, FreeSWITCH 1.4.15: bad<br>
>>>>>> precision<br>
>>>>><br>
>>>>><br>
>>>>> As someone who has been using KVM for a few years and has just migrated<br>
>>>>> about 20 VMs across to Digital Ocean about 6 weeks ago I admit I am a little<br>
>>>>> concerned about this finding.<br>
>>>>><br>
>>>>> Just out of curiosity can you tell me the specs of the VM that you<br>
>>>>> tested with and which DO Datacenter? I am using NYC3 so perhaps and I will<br>
>>>>> do some tests as soon as I have some time.<br>
>>>>><br>
>>>>> With regards to Xen was this at Linode or inhouse?<br>
>>>>><br>
>>>>> Thanks<br>
>>>>><br>
>>>>> Errol<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On Sun, Apr 5, 2015 at 1:37 AM, Stanislav Sinyagin<br>
>>>>> <<a href="mailto:ssinyagin@gmail.com">ssinyagin@gmail.com</a>> wrote:<br>
>>>>>><br>
>>>>>> here we go:<br>
>>>>>><br>
>>>>>> <a href="https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/" target="_blank">https://txlab.wordpress.com/2015/04/05/testing-rtp-clock-precision-on-a-virtual-machine/</a><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On Sat, Apr 4, 2015 at 4:26 PM, Stanislav Sinyagin<br>
>>>>>> <<a href="mailto:ssinyagin@gmail.com">ssinyagin@gmail.com</a>> wrote:<br>
>>>>>> > On Sat, Apr 4, 2015 at 1:03 PM, Regis M<br>
>>>>>> > <<a href="mailto:regis.freeswitch.org@tornad.net">regis.freeswitch.org@tornad.net</a>> wrote:<br>
>>>>>> >> How do you check wavs together in audacity ?<br>
>>>>>> ><br>
>>>>>> > you just open two WAV files, align their starting position, and<br>
>>>>>> > visually check the waves. If there's a problem, you would see that<br>
>>>>>> > after few seconds, the waves are not aligned any more.<br>
>>>>>> ><br>
>>>>>> >> Can you send me 2 samples wav (send... and recorded) so I could see<br>
>>>>>> >> the 20ms<br>
>>>>>> >> diff that you exposed...<br>
>>>>>> >><br>
>>>>>> >> Is there a way to automate the result checking, so it could help to<br>
>>>>>> >> monitor<br>
>>>>>> >> itsp quality<br>
>>>>>> ><br>
>>>>>> > Here<br>
>>>>>> > <a href="https://github.com/openpreserve/scape-xcorrsound/issues/17" target="_blank">https://github.com/openpreserve/scape-xcorrsound/issues/17</a><br>
>>>>>> > this is a link to a tool which compares two sound files at the wave<br>
>>>>>> > level. You can also see the sound samples in the ticket.<br>
>>>>>> > The problem is, that currently this tool does not detect such<br>
>>>>>> > slipping frames.<br>
>>>>>> ><br>
>>>>>> > I will also write a detailed article with examples, during the<br>
>>>>>> > weekend.<br>
>>>>>> ><br>
>>>>>> ><br>
>>>>>> ><br>
>>>>>> ><br>
>>>>>> ><br>
>>>>>> >><br>
>>>>>> >><br>
>>>>>> >> 2015-04-04 12:53 GMT+02:00 Stanislav Sinyagin<br>
>>>>>> >> <<a href="mailto:ssinyagin@gmail.com">ssinyagin@gmail.com</a>>:<br>
>>>>>> >>><br>
>>>>>> >>> Important is to send the media through two independent carriers -<br>
>>>>>> >>> then you<br>
>>>>>> >>> can be sure that your clock is compared against a high quality<br>
>>>>>> >>> one.<br>
>>>>>> >>><br>
>>>>>> >>> On Apr 4, 2015 12:46 PM, "Regis M"<br>
>>>>>> >>> <<a href="mailto:regis.freeswitch.org@tornad.net">regis.freeswitch.org@tornad.net</a>><br>
>>>>>> >>> wrote:<br>
>>>>>> >>>><br>
>>>>>> >>>> Hi,<br>
>>>>>> >>>><br>
>>>>>> >>>> It's a very interresting test and conclusion.<br>
>>>>>> >>>><br>
>>>>>> >>>> I will try to build a similar environnement on my freeswitch to<br>
>>>>>> >>>> check<br>
>>>>>> >>>> some system :)<br>
>>>>>> >>>><br>
>>>>>> >>>> thanks<br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>> 2015-04-04 2:16 GMT+02:00 Stanislav Sinyagin<br>
>>>>>> >>>> <<a href="mailto:ssinyagin@gmail.com">ssinyagin@gmail.com</a>>:<br>
>>>>>> >>>>><br>
>>>>>> >>>>> hi,<br>
>>>>>> >>>>><br>
>>>>>> >>>>> Here are some test results which might be interesting:<br>
>>>>>> >>>>><br>
>>>>>> >>>>> I've set up a test FreeSWITCH server at digitalocean (they use<br>
>>>>>> >>>>> KVM),<br>
>>>>>> >>>>> and originated test calls to some ITSP accounts. Each call<br>
>>>>>> >>>>> played a<br>
>>>>>> >>>>> test sound file (voice samples from ITU:<br>
>>>>>> >>>>><br>
>>>>>> >>>>> <a href="http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050" target="_blank">http://www.itu.int/net/itu-t/sigdb/genaudio/AudioForm-g.aspx?val=1000050</a><br>
>>>>>> >>>>> ), and on the receiving end another FreeSWITCH server was<br>
>>>>>> >>>>> recording<br>
>>>>>> >>>>> the received session. The receiving server was on a XEN VM. Both<br>
>>>>>> >>>>> servers talked G711 to their corresponding ITSP accounts.<br>
>>>>>> >>>>><br>
>>>>>> >>>>> Then I compared the source audio and received audio in Audacity<br>
>>>>>> >>>>> editor.<br>
>>>>>> >>>>><br>
>>>>>> >>>>> The tests showed that the received voice wave shifts 20ms from<br>
>>>>>> >>>>> the<br>
>>>>>> >>>>> original sound every few seconds, randomly back and forward.<br>
>>>>>> >>>>> This<br>
>>>>>> >>>>> skewing was always present, regardless of ITSP and time of day.<br>
>>>>>> >>>>><br>
>>>>>> >>>>> Then I moved the sending server to a XEN VM, and the waves<br>
>>>>>> >>>>> appeared<br>
>>>>>> >>>>> almost perfectly synchronized (there was a 5ms skew in one test,<br>
>>>>>> >>>>> and I<br>
>>>>>> >>>>> guess it's related to transcoding buffer, as the sound was<br>
>>>>>> >>>>> obviously<br>
>>>>>> >>>>> transcoded).<br>
>>>>>> >>>>><br>
>>>>>> >>>>> The effect was always visible with a 30-second sample, and I<br>
>>>>>> >>>>> also made<br>
>>>>>> >>>>> additional tests with 2-minute samples.<br>
>>>>>> >>>>><br>
>>>>>> >>>>> It would also be interesting to test this from a VmWare VM. If<br>
>>>>>> >>>>> someone<br>
>>>>>> >>>>> wants to run such a test, feel free to contact me directly.<br>
>>>>>> >>>>><br>
>>>>>> >>>>><br>
>>>>>> >>>>> cheers,<br>
>>>>>> >>>>> stan<br>
>>>>>> >>>>><br>
>>>>>> >>>>><br>
>>>>>> >>>>><br>
>>>>>> >>>>> _________________________________________________________________________<br>
>>>>>> >>>>> Professional FreeSWITCH Consulting Services:<br>
>>>>>> >>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>>> >>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>>> >>>>><br>
>>>>>> >>>>> Official FreeSWITCH Sites<br>
>>>>>> >>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >>>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>>> >>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>>> >>>>><br>
>>>>>> >>>>> FreeSWITCH-users mailing list<br>
>>>>>> >>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>>> >>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>>> >>>>><br>
>>>>>> >>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>>> >>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>><br>
>>>>>> >>>> _________________________________________________________________________<br>
>>>>>> >>>> Professional FreeSWITCH Consulting Services:<br>
>>>>>> >>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>>> >>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>>> >>>><br>
>>>>>> >>>> Official FreeSWITCH Sites<br>
>>>>>> >>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>>> >>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>>> >>>><br>
>>>>>> >>>> FreeSWITCH-users mailing list<br>
>>>>>> >>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>>> >>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>>> >>>><br>
>>>>>> >>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>>> >>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >>><br>
>>>>>> >>><br>
>>>>>> >>><br>
>>>>>> >>> _________________________________________________________________________<br>
>>>>>> >>> Professional FreeSWITCH Consulting Services:<br>
>>>>>> >>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>>> >>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>>> >>><br>
>>>>>> >>> Official FreeSWITCH Sites<br>
>>>>>> >>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>>> >>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>>> >>><br>
>>>>>> >>> FreeSWITCH-users mailing list<br>
>>>>>> >>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>>> >>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>>> >>><br>
>>>>>> >>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>>> >>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >><br>
>>>>>> >><br>
>>>>>> >><br>
>>>>>> >><br>
>>>>>> >> _________________________________________________________________________<br>
>>>>>> >> Professional FreeSWITCH Consulting Services:<br>
>>>>>> >> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>>> >> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>>> >><br>
>>>>>> >> Official FreeSWITCH Sites<br>
>>>>>> >> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> >> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>>> >> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>>> >><br>
>>>>>> >> FreeSWITCH-users mailing list<br>
>>>>>> >> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>>> >> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>>> >><br>
>>>>>> >> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>>> >> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>><br>
>>>>>><br>
>>>>>> _________________________________________________________________________<br>
>>>>>> Professional FreeSWITCH Consulting Services:<br>
>>>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>>><br>
>>>>>> Official FreeSWITCH Sites<br>
>>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>>><br>
>>>>>> FreeSWITCH-users mailing list<br>
>>>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>>><br>
>>>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> _________________________________________________________________________<br>
>>>>> Professional FreeSWITCH Consulting Services:<br>
>>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>><br>
>>>>> Official FreeSWITCH Sites<br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>><br>
>>>>> FreeSWITCH-users mailing list<br>
>>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>><br>
>>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> _________________________________________________________________________<br>
>>>> Professional FreeSWITCH Consulting Services:<br>
>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>><br>
>>>> Official FreeSWITCH Sites<br>
>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>><br>
>>>> FreeSWITCH-users mailing list<br>
>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>><br>
>>><br>
>>><br>
>>> _________________________________________________________________________<br>
>>> Professional FreeSWITCH Consulting Services:<br>
>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>><br>
>>> Official FreeSWITCH Sites<br>
>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>><br>
>>> FreeSWITCH-users mailing list<br>
>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>><br>
>><br>
>> _________________________________________________________________________<br>
>> Professional FreeSWITCH Consulting Services:<br>
>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>><br>
>> Official FreeSWITCH Sites<br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>><br>
>> FreeSWITCH-users mailing list<br>
>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<br>
><br>
> ☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞<br>
> <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a><br>
> ☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a><br>
><br>
> ClueCon Weekly Development Call<br>
> ☎ <a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a> ☎ <a href="tel:%2B19193869900" value="+19193869900">+19193869900</a><br>
><br>
> <a href="https://www.youtube.com/watch?v=9XXgW34t40s" target="_blank">https://www.youtube.com/watch?v=9XXgW34t40s</a><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">
<p><font face="courier new, monospace"><b><i><font size="4">Brian West</font></i></b><br><span style="font-size:x-small"><a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a></span></font></p>
<p><font size="1" face="courier new, monospace"><img src="http://billing.freeswitch.org/templates/default/img/whmcslogo.png"><br></font></p><p><font face="monospace, monospace"><b><i>Twitter: @FreeSWITCH , @briankwest</i></b><br><a href="http://www.freeswitchbook.com" target="_blank">http://www.freeswitchbook.com</a><br><a href="http://www.freeswitchcookbook.com" target="_blank">http://www.freeswitchcookbook.com</a></font></p><p><font face="monospace, monospace">ClueCon 2015 <a href="https://www.cluecon.com/call-for-speakers/" target="_blank">Call for Speakers</a> | <a href="https://freeswitch.com/cart.php?gid=1" target="_blank">Register</a> TODAY! | Reddit: <a href="https://www.reddit.com/r/freeswitch" target="_blank">/r/freeswitch</a></font></p>
<p><font face="monospace, monospace"><b>T:</b>+19184209001 | <b>F:</b>+19184209002 | <b>M:</b>+1918424WEST (9378)<br><b>iNUM:</b>+883 5100 1420 9001 | <b>ISN:</b>410*543 | <b>Skype:</b>briankwest</font></p></div></div></div></div></div></div></div></div>
</div>