On 1/15/09 11:01 AM, "Remko Kloosterman" <R.Kloosterman@mtel.nl> wrote:

Hello Ken, hello all,

I just read about the FreeSWITCH VMware applicance. I'm curious about your experiences with the audio quality on VMWare, so here's a new thread.

I've installed freeswitch on VMware Server for Windows. The IVR audio always plays choppy, while the server itself has no performance issues. The same poor voice quality also goes for Asterisk or Yate, even on a very fast VMware ESX system.

Did you experience the same and/or do you have pointers on how to troubleshoot and fix this?

There is a high resolution timer you need to enable on vmware... I'm not familiar enuff with all the versions of vmware to advise there that switch is, but they have a couple of articles on it in their knowledge base

I have been running FreeSWITCH on a VM ever since I got involved in the project. It's been almost a year now. I didn't do anything special - it works fine. I get audio problems if I go over 10 or 15 simultaneous calls. This is on the following setup: VMWare Server 1.0.6 and VMWare Server 2.0 (2.0 sucks, btw </rant>) Dell Precision 360 (Desktop) Pentium 4 2.66Ghz 2.5GB RAM (512MB allocated to FS) Fedora Core 8, 2.6.23.1-42.fc8 stock kernel (a bit old) 7.2K 80GB hard drive Yes, fancy machine I have, huh? This is my normal day-to-day workstation as well as my VMWare Server. It works fine, I got occassional missed heartbeat alerts and timer sync notices, but they're rare. - Darren -----Original Message----- From: Michael Collins [mailto:msc@freeswitch.org] Sent: Thursday, January 15, 2009 9:32 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] VMWare voice quality If anyone figures this out please post it to this thread. I am working on a wiki page for the VMWare appliance and I would like to be able to inform people on how to handle this situation. Also, IIUC, those running VMWare Fusion on Macs are not experiencing this, correct? What about those using a hypervisor like ESXi? Any known issues? Thanks, MC On Thu, Jan 15, 2009 at 9:15 AM, Ken Rice <krice@suspicious.org> wrote:

On 1/15/09 11:01 AM, "Remko Kloosterman" <R.Kloosterman@mtel.nl> wrote:

...

Hello Ken, hello all,

I just read about the FreeSWITCH VMware applicance. I'm curious about your experiences with the audio quality on VMWare, so here's a new thread.

I've installed freeswitch on VMware Server for Windows. The IVR audio always plays choppy, while the server itself has no performance issues. The same poor voice quality also goes for Asterisk or Yate, even on a very fast VMware ESX system.

Did you experience the same and/or do you have pointers on how to troubleshoot and fix this?

There is a high resolution timer you need to enable on vmware... I'm not familiar enuff with all the versions of vmware to advise there that switch is, but they have a couple of articles on it in their knowledge base

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-use rs http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

I think the problem with changing the CPU's is off your going from a single to multi, or multi to single CPU and not adjusting the kernel from SMP to normal or vise versa? I've got a nice ESXi farm at the moment (32 cores, 48 gigs of ram) thats running at under 5% usage. Haven't noticed any timing issues yet with FS, but haven't put it under any type of load yet. Thanks for the release, I'll see if i can get you some feedback. On Thu, Jan 15, 2009 at 2:23 PM, Chav Paskov <chavpaskov@shaw.ca> wrote:

Michael Collins wrote:

...

If anyone figures this out please post it to this thread. I am working on a wiki page for the VMWare appliance and I would like to be able to inform people on how to handle this situation.

Also, IIUC, those running VMWare Fusion on Macs are not experiencing this, correct? What about those using a hypervisor like ESXi? Any known issues?

Thanks, MC

On Thu, Jan 15, 2009 at 9:15 AM, Ken Rice <krice@suspicious.org> wrote:

...

On 1/15/09 11:01 AM, "Remko Kloosterman" <R.Kloosterman@mtel.nl> wrote:

...

Hello Ken, hello all,

I just read about the FreeSWITCH VMware applicance. I'm curious about your experiences with the audio quality on VMWare, so here's a new thread.

I've installed freeswitch on VMware Server for Windows. The IVR audio always plays choppy, while the server itself has no performance issues. The same poor voice quality also goes for Asterisk or Yate, even on a very fast VMware ESX system.

Did you experience the same and/or do you have pointers on how to troubleshoot and fix this?

There is a high resolution timer you need to enable on vmware... I'm not familiar enuff with all the versions of vmware to advise there that switch is, but they have a couple of articles on it in their knowledge base

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Hi All, I'm using freeswitch in production environment running on ESXi . I have no issues with voice /probably because simply i leave the media to flow between endpoints/ . Performance is amazing and i'd recommend this setup to everybody. it is important though when you set your VM on ESXi to set in advance the number of CPUs. Changing # of CPUs later might affect your performance. My recommendation is NOT to use VMWARE server on top of other OS. ESXi as hipervisor is linux in its core that provides you with enough access to the HW and nothing more so the overhead is as minimal as possible /while this is not the case fro VMware server - it needs underlaying OS and so on/. I hope this info helps. If anybody is interested i'd be glad to share me experience on his matter. Best Regards Chav

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

That won't eliminate the problem. Just reduce the possibility of it happening. Trust me... I've got a large ESX infrastructure, and there is no way that a software based Voice platform is going to provide skip free audio in a virtualized environment.

-----Original Message----- From: freeswitch-dev-bounces@lists.freeswitch.org [mailto:freeswitch- dev-bounces@lists.freeswitch.org] On Behalf Of Ken Rice Sent: Thursday, January 15, 2009 12:15 PM To: freeswitch-users@lists.freeswitch.org; Remko Kloosterman; freeswitch-dev@lists.freeswitch.org Subject: Re: [Freeswitch-dev] [Freeswitch-users] VMWare voice quality

On 1/15/09 11:01 AM, "Remko Kloosterman" <R.Kloosterman@mtel.nl> wrote:

...

Hello Ken, hello all,

I just read about the FreeSWITCH VMware applicance. I'm curious about your experiences with the audio quality on VMWare, so here's a new thread.

I've installed freeswitch on VMware Server for Windows. The IVR audio always plays choppy, while the server itself has no performance issues. The same poor voice quality also goes for Asterisk or Yate, even on a very fast VMware ESX system.

Did you experience the same and/or do you have pointers on how to troubleshoot and fix this?

There is a high resolution timer you need to enable on vmware... I'm not familiar enuff with all the versions of vmware to advise there that switch is, but they have a couple of articles on it in their knowledge base

_______________________________________________ Freeswitch-dev mailing list Freeswitch-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org

-- This message has been scanned for viruses and dangerous content by N2Net Mailshield, and is believed to be clean.

To the contrary, we have had quite good results in virtualized environments and you don't really need timing that is that accurate to make it work.

If you don't handle RTP, I'm sure it is amazing. However, if you have to do voicemail, stream audio from the server or do any kind of actual time/latency/jitter sensitive processing, I don't care how much you tune your hypervisor, it's never going to scale.

We work quite well on amazon EC2 for example. There are 2 issues I know about with vmware, 1 is you need to set a setting on the host to extend somewhat sane clocks being available, the second is I have seen issues with the bridged network adapter actually doubling up all packets causing very strange issues, I suggest not using bridged networking if you experience this.

I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on Vmware Server or Workstation?

We have people running FreeSWITCH in vmware and xen with media and considerable load and it doesn't have a problem. We also work very well inside OpenVZ.

I'd be very interested in seeing that, and knowing how it was done.

On Jan 15, 2009, at 1:02 PM, Brian West <brian@freeswitch.org> wrote:

On that note the OpenVZ instances could live migrate from box to box without dropping calls and usually had a small acceptable blip in audio.

I'd say a small blip is quite acceptable compared to the alternative! -MC

/b

On Jan 15, 2009, at 2:59 PM, Gregory Boehnlein wrote:

...

...

We have people running FreeSWITCH in vmware and xen with media and considerable load and it doesn't have a problem. We also work very well inside OpenVZ.

I'd be very interested in seeing that, and knowing how it was don

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Lot's of experience and suggestions here. Thanks. I believe it should be theoretically possible to have blip-free RTP streaming through the appliance. Most windows ethernet drivers allow for QoS packet scheduling. If the VMware network bridge driver honors this and syncs the buffers at 20ms frames (or whatever frame size applies) you should be able to schale up a bit and maintain low jitter. Anyone knows how the VMware network bridge exactly works? -----Oorspronkelijk bericht----- Van: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] Namens Gregory Boehnlein Verzonden: donderdag 15 januari 2009 21:37 Aan: freeswitch-users@lists.freeswitch.org Onderwerp: Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality

To the contrary, we have had quite good results in virtualized environments and you don't really need timing that is that accurate to

make it work.

If you don't handle RTP, I'm sure it is amazing. However, if you have to do voicemail, stream audio from the server or do any kind of actual time/latency/jitter sensitive processing, I don't care how much you tune your hypervisor, it's never going to scale.

We work quite well on amazon EC2 for example. There are 2 issues I know about with vmware, 1 is you need to set a setting on the host to extend somewhat sane clocks being available, the second is I have seen

issues with the bridged network adapter actually doubling up all packets causing very strange issues, I suggest not using bridged networking if you experience this.

I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on Vmware Server or Workstation? _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Yes it's hard to trust virtualized stuff because you have no idea what they skimp on in terms of realtime access. I won't endorse using FS on a VM as i have not done it very extensively beyond openvz but I can point out a few reasons why it has a fighting chance. FS uses a timer architecture designed to amplify the work of one timer thread into every timer open by FS. This single thread uses the monotonic clock on the system to try and perfrom a 1ms accurate loop. This single loop updates a soft value for current epoch time and microsecond epoch time with the goal of (again) being as close as possible to being accurate to 1 ms. The timer loop also has a global matrix to all of the timing intervals being subscribed to by a timer open by FS. The loop will tick a counter in each unique timing interval (10ms, 20ms, 60ms etc) and fire a conditional broadcast to all of the timers who are blocking for a tick. This is not perfectly accurate but close enough to end up plus or minus 2ms in resulting rtp traces. So as long as the VM will expose the syscall down to the real monotonic clock rather than doing it's own soft timing technique you have a better chance for success. The other issue with VM is with vmware, the bridged networking mode seems to send 2 of every RTP packet to the channel resulting in garbled audio from the obvious timing issue introduced from too many packets. Anyway evaluating FS with a VM is a good way to get acquainted but, with all the money saved choosing FS, many can afford to buy it a nice 8 core box for it to live on and still have money left over to support the project or ClueCon 2009 this august ;) On Fri, Jan 16, 2009 at 2:17 AM, Kristian Kielhofner < kristian.kielhofner@gmail.com> wrote:

Speaking of networking...

After timing that's the next "achilles heel" of RTP handing with virtualization.

Very, very few of these platforms were designed to handle massive numbers of very small RTP packets. Everything from interrupt handling on the actual ethernet adapter to getting the data into userspace within the virtual instance is worrisome:

http://www.xen.org/files/xensummit_4/NetworkIO_Santos.pdf http://forum.openvz.org/index.php?t=msg&goto=11619&

Interestingly enough the Xen paper makes it out to be really bad yet the OpenVZ post praises Xen's performance. Without any real testing, who knows? I just know that scaling 50pps per RTP stream (20ms packetization) can be difficult enough on native hardware, let alone [virtualization technology du jour].

On Thu, Jan 15, 2009 at 5:02 PM, Remko Kloosterman <R.Kloosterman@mtel.nl> wrote:

...

Lot's of experience and suggestions here. Thanks.

I believe it should be theoretically possible to have blip-free RTP streaming through the appliance. Most windows ethernet drivers allow for QoS packet scheduling. If the VMware network bridge driver honors this and syncs the buffers at 20ms frames (or whatever frame size applies) you should be able to schale up a bit and maintain low jitter.

Anyone knows how the VMware network bridge exactly works?

-----Oorspronkelijk bericht----- Van: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] Namens Gregory Boehnlein Verzonden: donderdag 15 januari 2009 21:37 Aan: freeswitch-users@lists.freeswitch.org Onderwerp: Re: [Freeswitch-users] [Freeswitch-dev] VMWare voice quality

...

To the contrary, we have had quite good results in virtualized environments and you don't really need timing that is that accurate to

...

make it work.

If you don't handle RTP, I'm sure it is amazing. However, if you have to do voicemail, stream audio from the server or do any kind of actual time/latency/jitter sensitive processing, I don't care how much you tune your hypervisor, it's never going to scale.

...

We work quite well on amazon EC2 for example. There are 2 issues I know about with vmware, 1 is you need to set a setting on the host to extend somewhat sane clocks being available, the second is I have seen

...

issues with the bridged network adapter actually doubling up all packets causing very strange issues, I suggest not using bridged networking if you experience this.

I've not seen this behavior on Vmware ESX 3.5u2. Maybe an issue on Vmware Server or Workstation?

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Kristian Kielhofner http://blog.krisk.org http://www.submityoursip.com http://www.astlinux.org http://www.star2star.com

_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:213-799-1400

You'll never fix this. Voice is a latency specific application unless you figure out how to manipulate time. Any virtualization platform is going to provide less timing granularity than raw hardware.

Hello Ken, hello all,

I just read about the FreeSWITCH VMware applicance. I'm curious about your experiences with the audio quality on VMWare, so here's a new thread.

I've installed freeswitch on VMware Server for Windows. The IVR audio always plays choppy, while the server itself has no performance issues. The same poor voice quality also goes for Asterisk or Yate, even on a very fast VMware ESX system.

Did you experience the same and/or do you have pointers on how to troubleshoot and fix this?