[Freeswitch-users] Problem with rtp streams and stair-stepped (or ramped) increased lag

Wed Jun 26 04:55:27 MSD 2013

You mentioned you were using VMWare to run FS in a virtual setup. What's
the OS and the physical NIC on the host box?

I'm using Linux kernel 3.2 with RTL8111/8168B chipset for the NIC.

William King
Senior Engineer
Quentus Technologies, INC
1037 NE 65th St Suite 273
Seattle, WA 98115
Main:   (877) 211-9337
Office: (206) 388-4772
Cell:   (253) 686-5518
william.king at quentustech.com

On 06/25/2013 05:36 PM, Nathan Neulinger wrote:
> Both... Graphs are from voipmonitor, but the wide shifts in delay are
> very visible when processing the stream in wireshark.
> 
> -- Nathan
> 
> On 06/25/2013 07:35 PM, William King wrote:
>> What are you using to analyze the rtp streams? Wireshark? or voipmonitor?
>>
>> William King
>> Senior Engineer
>> Quentus Technologies, INC
>> 1037 NE 65th St Suite 273
>> Seattle, WA 98115
>> Main:   (877) 211-9337
>> Office: (206) 388-4772
>> Cell:   (253) 686-5518
>> william.king at quentustech.com
>>
>> On 06/25/2013 05:19 PM, Nathan Neulinger wrote:
>>> Very interesting... look forward to seeing the comparison of those two
>>> captures... I'm in the process of trying to rule out vmware as a
>>> contributing factor by moving the FS servers to dedicated hardware, but
>>> your description sounds like there may be some other issue in addition.
>>>
>>> In my case, it's all confined within local network, so there shouldn't
>>> be much of any latency spikes, at least not significant ones.
>>>
>>> Looking at the graphs of a few calls today, on at least one of the
>>> not-so-bad ones, I saw a very periodic blip on the stairstep every 60
>>> seconds. That doesn't cover the large drift ones in the graphs I posted
>>> though.
>>>
>>> -- Nathan
>>>
>>> On 06/25/2013 06:48 PM, William King wrote:
>>>> Nathan,
>>>>
>>>> I had a chance to diagnose the issue live on a system today. While I'm
>>>> not yet sure we are hitting the same issue, I'm now confident the
>>>> problem I'm hitting is not FS related, but looks to be kernel
>>>> related. I
>>>> was able to get multiple calls up all with large 5+ second delays.
>>>> Restarting the calls did nothing to clear the problem. Restarting
>>>> Freeswitch repeatedly also had no effect. Upon restarting the box, the
>>>> problem was gone.
>>>>
>>>> Also in my diagnostics I've found that it isn't all UDP packets that
>>>> are
>>>> effected. I now have a switch mirroring the port and a pcap running
>>>> locally on the machine. I'll see tomorrow if it's just a matter of an
>>>> inbound or outbound network stack buffer(or network stack priority)
>>>> issue.
>>>>
>>>> William King
>>>> Senior Engineer
>>>> Quentus Technologies, INC
>>>> 1037 NE 65th St Suite 273
>>>> Seattle, WA 98115
>>>> Main:   (877) 211-9337
>>>> Office: (206) 388-4772
>>>> Cell:   (253) 686-5518
>>>> william.king at quentustech.com
>>>>
>>>> On 06/25/2013 05:13 AM, Nathan Neulinger wrote:
>>>>> I guess the big question is this:
>>>>>
>>>>>      Is it _supposed_ to recover?
>>>>>
>>>>>    From Tony's notes, it's sounding like it's very critically
>>>>> dependent on the timer in the switch, which very well be the
>>>>> current cause of my issues.
>>>>>
>>>>> What I probably need to do is have a pair of phones automatically
>>>>> calling each other, and just leaving them up - just to
>>>>> generate a continuous independent background stream. That'll tell me
>>>>> if any glitches are specific to a particular
>>>>> channel, or if they are common across multiple calls - which I'd
>>>>> expect if the timer was swinging.
>>>>>
>>>>> I don't know where my latency is being introduced unfortunately...
>>>>>
>>>>> -- Nathan
>>>>>
>>>>> On 06/25/2013 01:59 AM, Avi Marcus wrote:
>>>>>> It sounds like I saw  this on a home internet connection to a public
>>>>>> server when the connection had varying latency.
>>>>>> Once the call jittered, it never recovered.
>>>>>> My stop-gap was to add a small jitter buffer
>>>>>> <http://wiki.freeswitch.org/wiki/Jitterbuffer>, e.g. <action
>>>>>> application="jitterbuffer" data="100:200:20"/> to the user dialplan.
>>>>>> This /seemed /to make FS handle rewriting the timestamps for drift
>>>>>> rather than relying on the Linksys ATA to do that.
>>>>>>
>>>>>> Probably the wrong solution, but all I had time/understanding for.
>>>>>> -Avi
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 25, 2013 at 4:42 AM, Nathan Neulinger <nneul at mst.edu
>>>>>> <mailto:nneul at mst.edu>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>       On 06/24/2013 08:24 PM, Anthony Minessale wrote:
>>>>>>        > fsctl debug_level 10
>>>>>>        > Also make sure ve host has high res timing.
>>>>>>
>>>>>>       Turned up logging, will wait for next occurrence.
>>>>>>
>>>>>>       Not sure on the latter - it's a vmware ESXi 5.1 box running FC
>>>>>> 17 x86_64 as the guest.
>>>>>>
>>>>>>       /dev/rtc /dev/rtc0 exist
>>>>>>       posix and soft timers are loaded in fs
>>>>>>
>>>>>>       I've got a bunch of unused Dell 1950's on hand, so could use a
>>>>>> few of those machines if needed to get vmware out of the
>>>>>>       picture.
>>>>>>
>>>>>>       -- Nathan
>>>>>>
>>>>>>       ------------------------------------------------------------
>>>>>>       Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu>
>>>>>>       Missouri S&T Information Technology (573) 612-1412
>>>>>> <tel:%28573%29%20612-1412>
>>>>>>       System Administrator - Architect
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>>
>>>>>>
>>>>>>       Professional FreeSWITCH Consulting Services:
>>>>>>       consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>>       http://www.freeswitchsolutions.com
>>>>>>
>>>>>>       
>>>>>>       
>>>>>>
>>>>>>       Official FreeSWITCH Sites
>>>>>>       http://www.freeswitch.org
>>>>>>       http://wiki.freeswitch.org
>>>>>>       http://www.cluecon.com
>>>>>>
>>>>>>       FreeSWITCH-users mailing list
>>>>>>       FreeSWITCH-users at lists.freeswitch.org
>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>>       http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>>
>>>>>>       http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _________________________________________________________________________
>>>>
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>
>>>> http://www.freeswitch.org
>>>>
>>>
> 