[Freeswitch-users] Problem with rtp streams and stair-stepped (or ramped) increased lag

Wed Jun 26 20:09:30 MSD 2013

Got everything moved to real hardware... still seeing an issue with stair stepping.

Hardware is a Dell 1950 with dual dual or dual quad xeons.

I've got the debug level turned up on the boxes again and am capturing a call.

The test case where I saw the stair stepping was:

PolyCom 550 -> FS -> PolyCom 350
All local network w/ < 1ms typically.

Just connected from one to the other and left them connected for a half hour. Over the call, delay drift increased to 80 
ms it looked like. Now, that's not one of the problem calls, still waiting to see if I see any of those while running on 
real hardware.

This test case is on master from within past few days.

-- Nathan

On 06/25/2013 07:55 PM, William King wrote:
> You mentioned you were using VMWare to run FS in a virtual setup. What's
> the OS and the physical NIC on the host box?
>
> I'm using Linux kernel 3.2 with RTL8111/8168B chipset for the NIC.
>
> William King
> Senior Engineer
> Quentus Technologies, INC
> 1037 NE 65th St Suite 273
> Seattle, WA 98115
> Main:   (877) 211-9337
> Office: (206) 388-4772
> Cell:   (253) 686-5518
> william.king at quentustech.com
>
> On 06/25/2013 05:36 PM, Nathan Neulinger wrote:
>> Both... Graphs are from voipmonitor, but the wide shifts in delay are
>> very visible when processing the stream in wireshark.
>>
>> -- Nathan
>>
>> On 06/25/2013 07:35 PM, William King wrote:
>>> What are you using to analyze the rtp streams? Wireshark? or voipmonitor?
>>>
>>> William King
>>> Senior Engineer
>>> Quentus Technologies, INC
>>> 1037 NE 65th St Suite 273
>>> Seattle, WA 98115
>>> Main:   (877) 211-9337
>>> Office: (206) 388-4772
>>> Cell:   (253) 686-5518
>>> william.king at quentustech.com
>>>
>>> On 06/25/2013 05:19 PM, Nathan Neulinger wrote:
>>>> Very interesting... look forward to seeing the comparison of those two
>>>> captures... I'm in the process of trying to rule out vmware as a
>>>> contributing factor by moving the FS servers to dedicated hardware, but
>>>> your description sounds like there may be some other issue in addition.
>>>>
>>>> In my case, it's all confined within local network, so there shouldn't
>>>> be much of any latency spikes, at least not significant ones.
>>>>
>>>> Looking at the graphs of a few calls today, on at least one of the
>>>> not-so-bad ones, I saw a very periodic blip on the stairstep every 60
>>>> seconds. That doesn't cover the large drift ones in the graphs I posted
>>>> though.
>>>>
>>>> -- Nathan
>>>>
>>>> On 06/25/2013 06:48 PM, William King wrote:
>>>>> Nathan,
>>>>>
>>>>> I had a chance to diagnose the issue live on a system today. While I'm
>>>>> not yet sure we are hitting the same issue, I'm now confident the
>>>>> problem I'm hitting is not FS related, but looks to be kernel
>>>>> related. I
>>>>> was able to get multiple calls up all with large 5+ second delays.
>>>>> Restarting the calls did nothing to clear the problem. Restarting
>>>>> Freeswitch repeatedly also had no effect. Upon restarting the box, the
>>>>> problem was gone.
>>>>>
>>>>> Also in my diagnostics I've found that it isn't all UDP packets that
>>>>> are
>>>>> effected. I now have a switch mirroring the port and a pcap running
>>>>> locally on the machine. I'll see tomorrow if it's just a matter of an
>>>>> inbound or outbound network stack buffer(or network stack priority)
>>>>> issue.
>>>>>
>>>>> William King
>>>>> Senior Engineer
>>>>> Quentus Technologies, INC
>>>>> 1037 NE 65th St Suite 273
>>>>> Seattle, WA 98115
>>>>> Main:   (877) 211-9337
>>>>> Office: (206) 388-4772
>>>>> Cell:   (253) 686-5518
>>>>> william.king at quentustech.com
>>>>>
>>>>> On 06/25/2013 05:13 AM, Nathan Neulinger wrote:
>>>>>> I guess the big question is this:
>>>>>>
>>>>>>       Is it _supposed_ to recover?
>>>>>>
>>>>>>     From Tony's notes, it's sounding like it's very critically
>>>>>> dependent on the timer in the switch, which very well be the
>>>>>> current cause of my issues.
>>>>>>
>>>>>> What I probably need to do is have a pair of phones automatically
>>>>>> calling each other, and just leaving them up - just to
>>>>>> generate a continuous independent background stream. That'll tell me
>>>>>> if any glitches are specific to a particular
>>>>>> channel, or if they are common across multiple calls - which I'd
>>>>>> expect if the timer was swinging.
>>>>>>
>>>>>> I don't know where my latency is being introduced unfortunately...
>>>>>>
>>>>>> -- Nathan
>>>>>>
>>>>>> On 06/25/2013 01:59 AM, Avi Marcus wrote:
>>>>>>> It sounds like I saw  this on a home internet connection to a public
>>>>>>> server when the connection had varying latency.
>>>>>>> Once the call jittered, it never recovered.
>>>>>>> My stop-gap was to add a small jitter buffer
>>>>>>> <http://wiki.freeswitch.org/wiki/Jitterbuffer>, e.g. <action
>>>>>>> application="jitterbuffer" data="100:200:20"/> to the user dialplan.
>>>>>>> This /seemed /to make FS handle rewriting the timestamps for drift
>>>>>>> rather than relying on the Linksys ATA to do that.
>>>>>>>
>>>>>>> Probably the wrong solution, but all I had time/understanding for.
>>>>>>> -Avi
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 25, 2013 at 4:42 AM, Nathan Neulinger <nneul at mst.edu
>>>>>>> <mailto:nneul at mst.edu>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>        On 06/24/2013 08:24 PM, Anthony Minessale wrote:
>>>>>>>         > fsctl debug_level 10
>>>>>>>         > Also make sure ve host has high res timing.
>>>>>>>
>>>>>>>        Turned up logging, will wait for next occurrence.
>>>>>>>
>>>>>>>        Not sure on the latter - it's a vmware ESXi 5.1 box running FC
>>>>>>> 17 x86_64 as the guest.
>>>>>>>
>>>>>>>        /dev/rtc /dev/rtc0 exist
>>>>>>>        posix and soft timers are loaded in fs
>>>>>>>
>>>>>>>        I've got a bunch of unused Dell 1950's on hand, so could use a
>>>>>>> few of those machines if needed to get vmware out of the
>>>>>>>        picture.
>>>>>>>
>>>>>>>        -- Nathan
>>>>>>>
>>>>>>>        ------------------------------------------------------------
>>>>>>>        Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu>
>>>>>>>        Missouri S&T Information Technology (573) 612-1412
>>>>>>> <tel:%28573%29%20612-1412>
>>>>>>>        System Administrator - Architect
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>>
>>>>>>>
>>>>>>>        Professional FreeSWITCH Consulting Services:
>>>>>>>        consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>>>        http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>>        
>>>>>>>        
>>>>>>>
>>>>>>>        Official FreeSWITCH Sites
>>>>>>>        http://www.freeswitch.org
>>>>>>>        http://wiki.freeswitch.org
>>>>>>>        http://www.cluecon.com
>>>>>>>
>>>>>>>        FreeSWITCH-users mailing list
>>>>>>>        FreeSWITCH-users at lists.freeswitch.org
>>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>>>        http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>
>>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>
>>>>>>>
>>>>>>>        http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>>
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>

-- 
------------------------------------------------------------
Nathan Neulinger                       nneul at mst.edu
Missouri S&T Information Technology    (573) 612-1412
System Administrator - Architect