[Freeswitch-users] Problem with rtp streams and stair-stepped (or ramped) increased lag
William King
william.king at quentustech.com
Thu Jun 27 18:24:53 MSD 2013
How many cores did your VM have assigned to it? And how many concurrent
calls did you have up at the time?
William King
Senior Engineer
Quentus Technologies, INC
1037 NE 65th St Suite 273
Seattle, WA 98115
Main: (877) 211-9337
Office: (206) 388-4772
Cell: (253) 686-5518
william.king at quentustech.com
On 06/26/2013 11:47 AM, Nathan Neulinger wrote:
> Do the same caveats apply to voipmonitor? The graphs from voipmonitor are showing very clear stair stepping.
>
> This most recent test was 100% SIP, no skinny involved.
>
> At the moment, I'm waiting for any additional reports of the "bad" calls - i.e. the one that jumps to 2000+ ms. I
> haven't seen any of those yet on the real hardware, only the slow climb case.
>
> These are the two graphs on voipmonitor for the test call between two phones on my desk left sitting there for 40 minutes:
>
> http://i.imgur.com/3kx5q8z.png
>
> http://i.imgur.com/pj2cMN7.png
>
> -- Nathan
>
> On 06/26/2013 12:08 PM, Anthony Minessale wrote:
>> While a useful tool, You can't 100% rely on wireshark to report delay, it fails to properly deal with things such as
>> mark bits clearing the buffer etc and you have to trust it's clock to be accurate too.
>>
>> You should back it up with audio tests, call yourself and talk into it and measure for latency as well, also looking at
>> pcap of both ends of the call using the inbound ts vs the outbound.
>>
>> Another control you should use is to try sip 2 sip. It's possible that the skinny is lacking some of the RTP flags that
>> other channels use that help reduce latency.
>>
>>
>>
>>
>> On Wed, Jun 26, 2013 at 11:09 AM, Nathan Neulinger <nneul at mst.edu <mailto:nneul at mst.edu>> wrote:
>>
>> Got everything moved to real hardware... still seeing an issue with stair stepping.
>>
>> Hardware is a Dell 1950 with dual dual or dual quad xeons.
>>
>> I've got the debug level turned up on the boxes again and am capturing a call.
>>
>> The test case where I saw the stair stepping was:
>>
>> PolyCom 550 -> FS -> PolyCom 350
>> All local network w/ < 1ms typically.
>>
>> Just connected from one to the other and left them connected for a half hour. Over the call, delay drift increased to 80
>> ms it looked like. Now, that's not one of the problem calls, still waiting to see if I see any of those while running on
>> real hardware.
>>
>> This test case is on master from within past few days.
>>
>> -- Nathan
>>
>> On 06/25/2013 07:55 PM, William King wrote:
>> > You mentioned you were using VMWare to run FS in a virtual setup. What's
>> > the OS and the physical NIC on the host box?
>> >
>> > I'm using Linux kernel 3.2 with RTL8111/8168B chipset for the NIC.
>> >
>> > William King
>> > Senior Engineer
>> > Quentus Technologies, INC
>> > 1037 NE 65th St Suite 273
>> > Seattle, WA 98115
>> > Main: (877) 211-9337
>> > Office: (206) 388-4772
>> > Cell: (253) 686-5518
>> > william.king at quentustech.com <mailto:william.king at quentustech.com>
>> >
>> > On 06/25/2013 05:36 PM, Nathan Neulinger wrote:
>> >> Both... Graphs are from voipmonitor, but the wide shifts in delay are
>> >> very visible when processing the stream in wireshark.
>> >>
>> >> -- Nathan
>> >>
>> >> On 06/25/2013 07:35 PM, William King wrote:
>> >>> What are you using to analyze the rtp streams? Wireshark? or voipmonitor?
>> >>>
>> >>> William King
>> >>> Senior Engineer
>> >>> Quentus Technologies, INC
>> >>> 1037 NE 65th St Suite 273
>> >>> Seattle, WA 98115
>> >>> Main: (877) 211-9337
>> >>> Office: (206) 388-4772
>> >>> Cell: (253) 686-5518
>> >>> william.king at quentustech.com <mailto:william.king at quentustech.com>
>> >>>
>> >>> On 06/25/2013 05:19 PM, Nathan Neulinger wrote:
>> >>>> Very interesting... look forward to seeing the comparison of those two
>> >>>> captures... I'm in the process of trying to rule out vmware as a
>> >>>> contributing factor by moving the FS servers to dedicated hardware, but
>> >>>> your description sounds like there may be some other issue in addition.
>> >>>>
>> >>>> In my case, it's all confined within local network, so there shouldn't
>> >>>> be much of any latency spikes, at least not significant ones.
>> >>>>
>> >>>> Looking at the graphs of a few calls today, on at least one of the
>> >>>> not-so-bad ones, I saw a very periodic blip on the stairstep every 60
>> >>>> seconds. That doesn't cover the large drift ones in the graphs I posted
>> >>>> though.
>> >>>>
>> >>>> -- Nathan
>> >>>>
>> >>>> On 06/25/2013 06:48 PM, William King wrote:
>> >>>>> Nathan,
>> >>>>>
>> >>>>> I had a chance to diagnose the issue live on a system today. While I'm
>> >>>>> not yet sure we are hitting the same issue, I'm now confident the
>> >>>>> problem I'm hitting is not FS related, but looks to be kernel
>> >>>>> related. I
>> >>>>> was able to get multiple calls up all with large 5+ second delays.
>> >>>>> Restarting the calls did nothing to clear the problem. Restarting
>> >>>>> Freeswitch repeatedly also had no effect. Upon restarting the box, the
>> >>>>> problem was gone.
>> >>>>>
>> >>>>> Also in my diagnostics I've found that it isn't all UDP packets that
>> >>>>> are
>> >>>>> effected. I now have a switch mirroring the port and a pcap running
>> >>>>> locally on the machine. I'll see tomorrow if it's just a matter of an
>> >>>>> inbound or outbound network stack buffer(or network stack priority)
>> >>>>> issue.
>> >>>>>
>> >>>>> William King
>> >>>>> Senior Engineer
>> >>>>> Quentus Technologies, INC
>> >>>>> 1037 NE 65th St Suite 273
>> >>>>> Seattle, WA 98115
>> >>>>> Main: (877) 211-9337
>> >>>>> Office: (206) 388-4772
>> >>>>> Cell: (253) 686-5518
>> >>>>> william.king at quentustech.com <mailto:william.king at quentustech.com>
>> >>>>>
>> >>>>> On 06/25/2013 05:13 AM, Nathan Neulinger wrote:
>> >>>>>> I guess the big question is this:
>> >>>>>>
>> >>>>>> Is it _supposed_ to recover?
>> >>>>>>
>> >>>>>> From Tony's notes, it's sounding like it's very critically
>> >>>>>> dependent on the timer in the switch, which very well be the
>> >>>>>> current cause of my issues.
>> >>>>>>
>> >>>>>> What I probably need to do is have a pair of phones automatically
>> >>>>>> calling each other, and just leaving them up - just to
>> >>>>>> generate a continuous independent background stream. That'll tell me
>> >>>>>> if any glitches are specific to a particular
>> >>>>>> channel, or if they are common across multiple calls - which I'd
>> >>>>>> expect if the timer was swinging.
>> >>>>>>
>> >>>>>> I don't know where my latency is being introduced unfortunately...
>> >>>>>>
>> >>>>>> -- Nathan
>> >>>>>>
>> >>>>>> On 06/25/2013 01:59 AM, Avi Marcus wrote:
>> >>>>>>> It sounds like I saw this on a home internet connection to a public
>> >>>>>>> server when the connection had varying latency.
>> >>>>>>> Once the call jittered, it never recovered.
>> >>>>>>> My stop-gap was to add a small jitter buffer
>> >>>>>>> <http://wiki.freeswitch.org/wiki/Jitterbuffer>, e.g. <action
>> >>>>>>> application="jitterbuffer" data="100:200:20"/> to the user dialplan.
>> >>>>>>> This /seemed /to make FS handle rewriting the timestamps for drift
>> >>>>>>> rather than relying on the Linksys ATA to do that.
>> >>>>>>>
>> >>>>>>> Probably the wrong solution, but all I had time/understanding for.
>> >>>>>>> -Avi
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Jun 25, 2013 at 4:42 AM, Nathan Neulinger <nneul at mst.edu <mailto:nneul at mst.edu>
>> >>>>>>> <mailto:nneul at mst.edu <mailto:nneul at mst.edu>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 06/24/2013 08:24 PM, Anthony Minessale wrote:
>> >>>>>>> > fsctl debug_level 10
>> >>>>>>> > Also make sure ve host has high res timing.
>> >>>>>>>
>> >>>>>>> Turned up logging, will wait for next occurrence.
>> >>>>>>>
>> >>>>>>> Not sure on the latter - it's a vmware ESXi 5.1 box running FC
>> >>>>>>> 17 x86_64 as the guest.
>> >>>>>>>
>> >>>>>>> /dev/rtc /dev/rtc0 exist
>> >>>>>>> posix and soft timers are loaded in fs
>> >>>>>>>
>> >>>>>>> I've got a bunch of unused Dell 1950's on hand, so could use a
>> >>>>>>> few of those machines if needed to get vmware out of the
>> >>>>>>> picture.
>> >>>>>>>
>> >>>>>>> -- Nathan
>> >>>>>>>
>> >>>>>>> ------------------------------------------------------------
>> >>>>>>> Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu> <mailto:nneul at mst.edu <mailto:nneul at mst.edu>>
>> >>>>>>> Missouri S&T Information Technology (573) 612-1412
>> >>>>>>> <tel:%28573%29%20612-1412>
>> >>>>>>> System Administrator - Architect
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _________________________________________________________________________
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Professional FreeSWITCH Consulting Services:
>> >>>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org> <mailto:consulting at freeswitch.org
>> <mailto:consulting at freeswitch.org>>
>> >>>>>>> http://www.freeswitchsolutions.com
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Official FreeSWITCH Sites
>> >>>>>>> http://www.freeswitch.org
>> >>>>>>> http://wiki.freeswitch.org
>> >>>>>>> http://www.cluecon.com
>> >>>>>>>
>> >>>>>>> FreeSWITCH-users mailing list
>> >>>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> >>>>>>> <mailto:FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>>
>> >>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >>>>>>>
>> >>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> http://www.freeswitch.org
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> _________________________________________________________________________
>> >>>>>
>> >>>>> Professional FreeSWITCH Consulting Services:
>> >>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> >>>>> http://www.freeswitchsolutions.com
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Official FreeSWITCH Sites
>> >>>>> http://www.freeswitch.org
>> >>>>> http://wiki.freeswitch.org
>> >>>>> http://www.cluecon.com
>> >>>>>
>> >>>>> FreeSWITCH-users mailing list
>> >>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >>>>>
>> >>>>> http://www.freeswitch.org
>> >>>>>
>> >>>>
>> >>
>>
>> --
>> ------------------------------------------------------------
>> Nathan Neulinger nneul at mst.edu <mailto:nneul at mst.edu>
>> Missouri S&T Information Technology (573) 612-1412
>> System Administrator - Architect
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>> --
>> Anthony Minessale II
>>
>> FreeSWITCH http://www.freeswitch.org/
>> ClueCon http://www.cluecon.com/
>> Twitter: http://twitter.com/FreeSWITCH_wire
>>
>> AIM: anthm
>> MSN:anthony_minessale at hotmail.com <mailto:MSN%3Aanthony_minessale at hotmail.com>
>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
>>
>> FreeSWITCH Developer Conference
>> sip:888 at conference.freeswitch.org <mailto:sip%3A888 at conference.freeswitch.org>
>> googletalk:conf+888 at conference.freeswitch.org <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>> pstn:+19193869900
>
Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users
mailing list