You mentioned you were using VMWare to run FS in a virtual setup. What's the OS and the physical NIC on the host box? I'm using Linux kernel 3.2 with RTL8111/8168B chipset for the NIC. William King Senior Engineer Quentus Technologies, INC 1037 NE 65th St Suite 273 Seattle, WA 98115 Main: (877) 211-9337 Office: (206) 388-4772 Cell: (253) 686-5518 william.king@quentustech.com On 06/25/2013 05:36 PM, Nathan Neulinger wrote:
Both... Graphs are from voipmonitor, but the wide shifts in delay are very visible when processing the stream in wireshark.
-- Nathan
On 06/25/2013 07:35 PM, William King wrote:
What are you using to analyze the rtp streams? Wireshark? or voipmonitor?
William King Senior Engineer Quentus Technologies, INC 1037 NE 65th St Suite 273 Seattle, WA 98115 Main: (877) 211-9337 Office: (206) 388-4772 Cell: (253) 686-5518 william.king@quentustech.com
On 06/25/2013 05:19 PM, Nathan Neulinger wrote:
Very interesting... look forward to seeing the comparison of those two captures... I'm in the process of trying to rule out vmware as a contributing factor by moving the FS servers to dedicated hardware, but your description sounds like there may be some other issue in addition.
In my case, it's all confined within local network, so there shouldn't be much of any latency spikes, at least not significant ones.
Looking at the graphs of a few calls today, on at least one of the not-so-bad ones, I saw a very periodic blip on the stairstep every 60 seconds. That doesn't cover the large drift ones in the graphs I posted though.
-- Nathan
On 06/25/2013 06:48 PM, William King wrote:
Nathan,
I had a chance to diagnose the issue live on a system today. While I'm not yet sure we are hitting the same issue, I'm now confident the problem I'm hitting is not FS related, but looks to be kernel related. I was able to get multiple calls up all with large 5+ second delays. Restarting the calls did nothing to clear the problem. Restarting Freeswitch repeatedly also had no effect. Upon restarting the box, the problem was gone.
Also in my diagnostics I've found that it isn't all UDP packets that are effected. I now have a switch mirroring the port and a pcap running locally on the machine. I'll see tomorrow if it's just a matter of an inbound or outbound network stack buffer(or network stack priority) issue.
William King Senior Engineer Quentus Technologies, INC 1037 NE 65th St Suite 273 Seattle, WA 98115 Main: (877) 211-9337 Office: (206) 388-4772 Cell: (253) 686-5518 william.king@quentustech.com
On 06/25/2013 05:13 AM, Nathan Neulinger wrote:
I guess the big question is this:
Is it _supposed_ to recover?
From Tony's notes, it's sounding like it's very critically dependent on the timer in the switch, which very well be the current cause of my issues.
What I probably need to do is have a pair of phones automatically calling each other, and just leaving them up - just to generate a continuous independent background stream. That'll tell me if any glitches are specific to a particular channel, or if they are common across multiple calls - which I'd expect if the timer was swinging.
I don't know where my latency is being introduced unfortunately...
-- Nathan
On 06/25/2013 01:59 AM, Avi Marcus wrote:
It sounds like I saw this on a home internet connection to a public server when the connection had varying latency. Once the call jittered, it never recovered. My stop-gap was to add a small jitter buffer <http://wiki.freeswitch.org/wiki/Jitterbuffer>, e.g. <action application="jitterbuffer" data="100:200:20"/> to the user dialplan. This /seemed /to make FS handle rewriting the timestamps for drift rather than relying on the Linksys ATA to do that.
Probably the wrong solution, but all I had time/understanding for. -Avi
On Tue, Jun 25, 2013 at 4:42 AM, Nathan Neulinger <nneul@mst.edu <mailto:nneul@mst.edu>> wrote:
On 06/24/2013 08:24 PM, Anthony Minessale wrote: > fsctl debug_level 10 > Also make sure ve host has high res timing.
Turned up logging, will wait for next occurrence.
Not sure on the latter - it's a vmware ESXi 5.1 box running FC 17 x86_64 as the guest.
/dev/rtc /dev/rtc0 exist posix and soft timers are loaded in fs
I've got a bunch of unused Dell 1950's on hand, so could use a few of those machines if needed to get vmware out of the picture.
-- Nathan
------------------------------------------------------------ Nathan Neulinger nneul@mst.edu <mailto:nneul@mst.edu> Missouri S&T Information Technology (573) 612-1412 <tel:%28573%29%20612-1412> System Administrator - Architect
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users