[Freeswitch-users] Problem with rtp streams and stair-stepped (or ramped) increased lag
Nathan Neulinger
nneul at mst.edu
Thu Jun 27 18:36:08 MSD 2013
I think I gave it either 2 or 4 cores... On the test instance, most likely 1 or 2 calls including the test one. It's not
a loading issue as near as I can tell - at least not the slow stairstep one.
-- Nathan
On 06/27/2013 09:24 AM, William King wrote:
> How many cores did your VM have assigned to it? And how many concurrent
> calls did you have up at the time?
>
> William King
> Senior Engineer
> Quentus Technologies, INC
> 1037 NE 65th St Suite 273
> Seattle, WA 98115
> Main: (877) 211-9337
> Office: (206) 388-4772
> Cell: (253) 686-5518
> william.king at quentustech.com
>
> On 06/26/2013 11:47 AM, Nathan Neulinger wrote:
>> Do the same caveats apply to voipmonitor? The graphs from voipmonitor are showing very clear stair stepping.
>>
>> This most recent test was 100% SIP, no skinny involved.
>>
>> At the moment, I'm waiting for any additional reports of the "bad" calls - i.e. the one that jumps to 2000+ ms. I
>> haven't seen any of those yet on the real hardware, only the slow climb case.
>>
>> These are the two graphs on voipmonitor for the test call between two phones on my desk left sitting there for 40 minutes:
>>
>> http://i.imgur.com/3kx5q8z.png
>>
>> http://i.imgur.com/pj2cMN7.png
>>
>> -- Nathan
>>
>> On 06/26/2013 12:08 PM, Anthony Minessale wrote:
>>> While a useful tool, You can't 100% rely on wireshark to report delay, it fails to properly deal with things such as
>>> mark bits clearing the buffer etc and you have to trust it's clock to be accurate too.
>>>
>>> You should back it up with audio tests, call yourself and talk into it and measure for latency as well, also looking at
>>> pcap of both ends of the call using the inbound ts vs the outbound.
>>>
>>> Another control you should use is to try sip 2 sip. It's possible that the skinny is lacking some of the RTP flags that
>>> other channels use that help reduce latency.
>>>
>>>
>>>
>>>
>>> On Wed, Jun 26, 2013 at 11:09 AM, Nathan Neulinger <nneul at mst.edu <mailto:nneul at mst.edu>> wrote:
>>>
>>> Got everything moved to real hardware... still seeing an issue with stair stepping.
>>>
>>> Hardware is a Dell 1950 with dual dual or dual quad xeons.
>>>
>>> I've got the debug level turned up on the boxes again and am capturing a call.
>>>
>>> The test case where I saw the stair stepping was:
>>>
>>> PolyCom 550 -> FS -> PolyCom 350
>>> All local network w/ < 1ms typically.
>>>
>>> Just connected from one to the other and left them connected for a half hour. Over the call, delay drift increased to 80
>>> ms it looked like. Now, that's not one of the problem calls, still waiting to see if I see any of those while running on
>>> real hardware.
>>>
>>> This test case is on master from within past few days.
>>>
>>> -- Nathan
>>>
>>> On 06/25/2013 07:55 PM, William King wrote:
>>> > You mentioned you were using VMWare to run FS in a virtual setup. What's
>>> > the OS and the physical NIC on the host box?
>>> >
>>> > I'm using Linux kernel 3.2 with RTL8111/8168B chipset for the NIC.
>>> >
>>> > William King
>>> > Senior Engineer
>>> > Quentus Technologies, INC
>>> > 1037 NE 65th St Suite 273
>>> > Seattle, WA 98115
>>> > Main: (877) 211-9337
>>> > Office: (206) 388-4772
>>> > Cell: (253) 686-5518
>>> > william.king at quentustech.com <mailto:william.king at quentustech.com>
>>> >
>>> > On 06/25/2013 05:36 PM, Nathan Neulinger wrote:
>>> >> Both... Graphs are from voipmonitor, but the wide shifts in delay are
>>> >> very visible when processing the stream in wireshark.
>>> >>
>>> >> -- Nathan
>>> >>
>>> >> On 06/25/2013 07:35 PM, William King wrote:
>>> >>> What are you using to analyze the rtp streams? Wireshark? or voipmonitor?
>>> >>>
>>> >>> William King
>>> >>> Senior Engineer
>>> >>> Quentus Technologies, INC
>>> >>> 1037 NE 65th St Suite 273
>>> >>> Seattle, WA 98115
>>> >>> Main: (877) 211-9337
>>> >>> Office: (206) 388-4772
>>> >>> Cell: (253) 686-5518
>>> >>> william.king at quentustech.com <mailto:william.king at quentustech.com>
>>> >>>
>>> >>> On 06/25/2013 05:19 PM, Nathan Neulinger wrote:
>>> >>>> Very interesting... look forward to seeing the comparison of those two
>>> >>>> captures... I'm in the process of trying to rule out vmware as a
>>> >>>> contributing factor by moving the FS servers to dedicated hardware, but
>>> >>>> your description sounds like there may be some other issue in addition.
>>> >>>>
>>> >>>> In my case, it's all confined within local network, so there shouldn't
>>> >>>> be much of any latency spikes, at least not significant ones.
>>> >>>>
>>> >>>> Looking at the graphs of a few calls today, on at least one of the
>>> >>>> not-so-bad ones, I saw a very periodic blip on the stairstep every 60
>>> >>>> seconds. That doesn't cover the large drift ones in the graphs I posted
>>> >>>> though.
>>> >>>>
>>> >>>> -- Nathan
>>> >>>>
>>> >>>> On 06/25/2013 06:48 PM, William King wrote:
>>> >>>>> Nathan,
>>> >>>>>
>>> >>>>> I had a chance to diagnose the issue live on a system today. While I'm
>>> >>>>> not yet sure we are hitting the same issue, I'm now confident the
>>> >>>>> problem I'm hitting is not FS related, but looks to be kernel
>>> >>>>> related. I
>>> >>>>> was able to get multiple calls up all with large 5+ second delays.
>>> >>>>> Restarting the calls did nothing to clear the problem. Restarting
>>> >>>>> Freeswitch repeatedly also had no effect. Upon restarting the box, the
>>> >>>>> problem was gone.
>>> >>>>>
>>> >>>>> Also in my diagnostics I've found that it isn't all UDP packets that
>>> >>>>> are
>>> >>>>> effected. I now have a switch mirroring the port and a pcap running
>>> >>>>> locally on the machine. I'll see tomorrow if it's just a matter of an
>>> >>>>> inbound or outbound network stack buffer(or network stack priority)
>>> >>>>> issue.
>>> >>>>>
>>> >>>>> William King
>>> >>>>> Senior Engineer
>>> >>>>> Quentus Technologies, INC
>>> >>>>> 1037 NE 65th St Suite 273
>>> >>>>> Seattle, WA 98115
>>> >>>>> Main: (877) 211-9337
>>> >>>>> Office: (206) 388-4772
>>> >>>>> Cell: (253) 686-5518
>>> >>>>> william.king at quentustech.com <mailto:william.king at quentustech.com>
>>> >>>>>
>>> >>>>> On 06/25/2013 05:13 AM, Nathan Neulinger wrote:
>>> >>>>>> I guess the big question is this:
>>> >>>>>>
>>> >>>>>> Is it _supposed_ to recover?
>>> >>>>>>
>>> >>>>>> From Tony's notes, it's sounding like it's very critically
>>> >>>>>> dependent on the timer in the switch, which very well be the
>>> >>>>>> current cause of my issues.
>>> >>>>>>
>>> >>>>>> What I probably need to do is have a pair of phones automatically
>>> >>>>>> calling each other, and just leaving them up - just to
>>> >>>>>> generate a continuous independent background stream. That'll tell me
>>> >>>>>> if any glitches are specific to a particular
>>> >>>>>> channel, or if they are common across multiple calls - which I'd
>>> >>>>>> expect if the timer was swinging.
>>> >>>>>>
>>> >>>>>> I don't know where my latency is being introduced unfortunately...
>>> >>>>>>
>>> >>>>>> -- Nathan
>>> >>>>>>
>>> >>>>>> On 06/25/2013 01:59 AM, Avi Marcus wrote:
>>> >>>>>>> It sounds like I saw this on a home internet connection to a public
>>> >>>>>>> server when the connection had varying latency.
>>> >>>>>>> Once the call jittered, it never recovered.
>>> >>>>>>> My stop-gap was to add a small jitter buffer
>>> >>>>>>> <http://wiki.freeswitch.org/wiki/Jitterbuffer>, e.g. <action
>>> >>>>>>> application="jitterbuffer" data="100:200:20"/> to the user dialplan.
>>> >>>>>>> This /seemed /to make FS handle rewriting the timestamps for drift
>>> >>>>>>> rather than relying on the Linksys ATA to do that.
>>> >>>>>>>
>>> >>>>>>> Probably the wrong solution, but all I had time/understanding for.
>>> >>>>>>> -Avi
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Tue, Jun 25, 2013 at 4:42 AM, Nathan Neulinger <nneul at mst.edu <mailto:nneul at mst.edu>
>>> >>>>>>> <mailto:nneul at mst.edu <mailto:nneul at mst.edu>>> wrote:
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On 06/24/2013 08:24 PM, Anthony Minessale wrote:
>>> >>>>>>> > fsctl debug_level 10
>>> >>>>>>> > Also make sure ve host has high res timing.
>>> >>>>>>>
>>> >>>>>>> Turned up logging, will wait for next occurrence.
>>> >>>>>>>
>>> >>>>>>> Not sure on the latter - it's a vmware ESXi 5.1 box running FC
>>> >>>>>>> 17 x86_64 as the guest.
>>> >>>>>>>
>>> >>>>>>> /dev/rtc /dev/rtc0 exist
>>> >>>>>>> posix and soft timers are loaded in fs
>>> >>>>>>>
>>> >>>>>>> I've got a bunch of unused Dell 1950's on hand, so could use a
>>> >>>>>>> few of those machines if needed to get vmware out of the
>>> >>>>>>> picture.
>>> >>>>>>>
>>> >>>>>>> -- Nathan
>>> >>>>>>>
>>> >>>>>>> ------------------------------------------------------------
>>> >>>>>>> Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu> <mailto:nneul at mst.edu <mailto:nneul at mst.edu>>
>>> >>>>>>> Missouri S&T Information Technology (573) 612-1412
>>> >>>>>>> <tel:%28573%29%20612-1412>
>>> >>>>>>> System Administrator - Architect
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> _________________________________________________________________________
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Professional FreeSWITCH Consulting Services:
>>> >>>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org
>>> <mailto:consulting at freeswitch.org>>
>>> >>>>>>> http://www.freeswitchsolutions.com
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Official FreeSWITCH Sites
>>> >>>>>>> http://www.freeswitch.org
>>> >>>>>>> http://wiki.freeswitch.org
>>> >>>>>>> http://www.cluecon.com
>>> >>>>>>>
>>> >>>>>>> FreeSWITCH-users mailing list
>>> >>>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> >>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>>
>>> >>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >>>>>>>
>>> >>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> http://www.freeswitch.org
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>> _________________________________________________________________________
>>> >>>>>
>>> >>>>> Professional FreeSWITCH Consulting Services:
>>> >>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> >>>>> http://www.freeswitchsolutions.com
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Official FreeSWITCH Sites
>>> >>>>> http://www.freeswitch.org
>>> >>>>> http://wiki.freeswitch.org
>>> >>>>> http://www.cluecon.com
>>> >>>>>
>>> >>>>> FreeSWITCH-users mailing list
>>> >>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> >>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> >>>>>
>>> >>>>> http://www.freeswitch.org
>>> >>>>>
>>> >>>>
>>> >>
>>>
>>> --
>>> ------------------------------------------------------------
>>> Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu>
>>> Missouri S&T Information Technology (573) 612-1412
>>> System Administrator - Architect
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org <mailto:sip%3A888 at conference.freeswitch.org>
>>> googletalk:conf+888 at conference.freeswitch.org <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>> pstn:+19193869900
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
------------------------------------------------------------
Nathan Neulinger nneul at mst.edu
Missouri S&T Information Technology (573) 612-1412
System Administrator - Architect
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list