How to inspect the jitterbuffer?
Hi all, I have a setup with a FreeTDM/wanpipe trunk and most of the time the audio skips on the path to this trunk (e.g. after a period of silence, RTP marker bit set by the sender). It is not an issue with FreeTDM/wanpipe but more the way the RTP input is handled by FreeSWITCH. My theory is the audio skips because the RTP 'queue' has an under-run (while it shouldn't, there is no packet loss). So how can I see if the jitterbuffer is enabled for a given call (or globally)? Can I somehow see under-runs? The recovered clock/ clock skew? I couldn't find any of this in the wiki[1]. thanks holger [1] http://wiki.freeswitch.org/wiki/Jitterbuffer
The jitter buffer is not enabled by default. On Feb 9, 2012, at 15:52, Holger Freyther <holger@freyther.de> wrote:
Hi all,
I have a setup with a FreeTDM/wanpipe trunk and most of the time the audio skips on the path to this trunk (e.g. after a period of silence, RTP marker bit set by the sender). It is not an issue with FreeTDM/wanpipe but more the way the RTP input is handled by FreeSWITCH. My theory is the audio skips because the RTP 'queue' has an under-run (while it shouldn't, there is no packet loss).
So how can I see if the jitterbuffer is enabled for a given call (or globally)? Can I somehow see under-runs? The recovered clock/ clock skew? I couldn't find any of this in the wiki[1].
thanks holger
[1] http://wiki.freeswitch.org/wiki/Jitterbuffer
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
what does /etc/wanpipe/wanpipe1.conf look like? and which rev of FS and wanpipe driver are you on. What ptime is the sip traffic? On Thu, Feb 9, 2012 at 2:52 PM, Holger Freyther <holger@freyther.de> wrote:
Hi all,
I have a setup with a FreeTDM/wanpipe trunk and most of the time the audio skips on the path to this trunk (e.g. after a period of silence, RTP marker bit set by the sender). It is not an issue with FreeTDM/wanpipe but more the way the RTP input is handled by FreeSWITCH. My theory is the audio skips because the RTP 'queue' has an under-run (while it shouldn't, there is no packet loss).
So how can I see if the jitterbuffer is enabled for a given call (or globally)? Can I somehow see under-runs? The recovered clock/ clock skew? I couldn't find any of this in the wiki[1].
thanks holger
[1] http://wiki.freeswitch.org/wiki/Jitterbuffer
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony Minessale <anthony.minessale@...> writes:
what does /etc/wanpipe/wanpipe1.conf look like? and which rev of FS and wanpipe driver are you on. What ptime is the sip traffic?
Hi, the wanpipe1.conf is here[1], I am using wanpipe 3.5.24, I started with a FreeSWITCH from around 20111010, but I retested with the a version from 20120205. What I have done so far to try to understand things: fs_cli> originate freetdm/1/nr &echo(1000) (audio quality is good in both ways, so FreeTDM by itself appears to work) On a 'normal' SIP call (the path to ISDN produces the skips): I used the 'bug' application to record the audio file and it had no skips. Wireshark's RTP Analysis shows no packet loss, little jitter, just a bit of clock drifting. To me it looks like if FreeTDM doesn't get the audio samples in time and then 'nothing' is transmitted. This is why I asked about inspecting the state of the jitterbuffer. regards holger [1] http://paste.lisp.org/display/127633
If there is no RTP then the TDM should be generating its own audio. Try MTU = 80 IDLE_FLAG = 0xFF On Fri, Feb 10, 2012 at 3:41 AM, Holger Freyther <holger@freyther.de> wrote:
Anthony Minessale <anthony.minessale@...> writes:
what does /etc/wanpipe/wanpipe1.conf look like? and which rev of FS and wanpipe driver are you on. What ptime is the sip traffic?
Hi,
the wanpipe1.conf is here[1], I am using wanpipe 3.5.24, I started with a FreeSWITCH from around 20111010, but I retested with the a version from 20120205.
What I have done so far to try to understand things: fs_cli> originate freetdm/1/nr &echo(1000) (audio quality is good in both ways, so FreeTDM by itself appears to work)
On a 'normal' SIP call (the path to ISDN produces the skips): I used the 'bug' application to record the audio file and it had no skips. Wireshark's RTP Analysis shows no packet loss, little jitter, just a bit of clock drifting.
To me it looks like if FreeTDM doesn't get the audio samples in time and then 'nothing' is transmitted. This is why I asked about inspecting the state of the jitterbuffer.
regards holger
[1] http://paste.lisp.org/display/127633
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony Minessale <anthony.minessale@...> writes:
If there is no RTP then the TDM should be generating its own audio.
Try
MTU = 80 IDLE_FLAG = 0xFF
Writing about audio is difficult. Imagine one says the word "Two".. and one called a mailbox. Opening the file in audacity gives you the wave of what you said, you zoom in and you see that in the middle of the word the the amplitude is zero for a bit (where it should not be zero). As echoing works on FreeTDM, there is no packet loss in RTP, the bug has the whole audio... it means something is broken in FreeSWITCH to get the audio to FreeTDM in time? Any ideas?
That made no sense in regards to my last message. I asked you to try particular wanpipe parameters, so you should try them and restart your wanpipe and report the results. Your MTU is very high and may cause timing issues for interop with voip. set MTU to 80 (10ms) set IDLE_FLAG to 0xFF (send the 255 byte on idle stream) 1) Turn off FreeSWITCH 2) wanrouter stop 3) wanrouter start 4) start FreeSWITCH You are pushing RTP up against TDM, RTP can start and stop, TDM cannot, When there is no RTP the TDM will carry on sending the IDLE_FLAG byte over the wire. I made both the RTP stack and the TDM endpoint from scratch so you should assume I may have some insight. On Fri, Feb 10, 2012 at 12:28 PM, Holger Freyther <holger@freyther.de> wrote:
Anthony Minessale <anthony.minessale@...> writes:
If there is no RTP then the TDM should be generating its own audio.
Try
MTU = 80 IDLE_FLAG = 0xFF
Writing about audio is difficult. Imagine one says the word "Two".. and one called a mailbox. Opening the file in audacity gives you the wave of what you said, you zoom in and you see that in the middle of the word the the amplitude is zero for a bit (where it should not be zero). As echoing works on FreeTDM, there is no packet loss in RTP, the bug has the whole audio... it means something is broken in FreeSWITCH to get the audio to FreeTDM in time?
Any ideas?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony Minessale <anthony.minessale@...> writes:
You are pushing RTP up against TDM, RTP can start and stop, TDM cannot, When there is no RTP the TDM will carry on sending the IDLE_FLAG byte over the wire.
I made both the RTP stack and the TDM endpoint from scratch so you should assume I may have some insight.
What I am arguing with... is that if I say a word like 'two' and RTP looks a bit like this. RTP: Wall Time Seq.No Time 'Sound' 0.0 1 0 'T' 20ms* 2 160 'W' 40ms* 3 320 'O' ... some skew 100 100*160 'T' +20ms 101 101*160 'W' +20ms 102 102*160 'O' and on the remote it arrives as: 'TW(?)O'..'T'..'O' something is wrong with the playback speed/queue/buffer of the audio stack. All I ask for is (because I didn't see anything in the wiki about that): 1.) How do I check if the jitter buffer is enabled on a call. 2.) How do I check if arriving RTP packets arrived too late for the speed of the FreeTDM voice queue. More details: I started with a setup like this: RTP/AMR -> FreeSWITCH (sangoma transcoder) -> FreeTDM.. For debugging I changed it to: AMR -> FreeSWITCH -> RTP/PCMA -> FreeSWITCH -> FreeTDM. (SIP on 5060) (SIP on 5062) The idea was that the first FreeSWITCH has to deal with the jitter and the second receives a clean linear in wall clock time RTP stream. My sofia sip profile includes these params: <param name="rtp-timer-name" value="soft"/> <param name="manual-rtp-bugs" value="SEND_LINEAR_TIMESTAMPS|NEVER_SEND_MARKER|\n IGNORE_DTMF_DURATION" /> <param name="rtp-rewrite-timestamps" value="true"/> The resulting output of the frontend FreeSWITCH is not linear though (so no idea what rtp-timer-name soft should do): But the resulting stream has packets like this: P.No Wall Time Codec 723 12.761045 G.711 PCMA, Seq=18533, Time=90720 731 12.841107 G.711 PCMA, Seq=18534, Time=90880 so somehow the expected arrival time was 17.78 but it ended up being 12.84 and I think FreeSWITCH/FreeTDM/STFU do not deal well with that. any ideas? holger * add a little jitter
You are arrogantly refusing to cooperate with my questions (why are people so stubborn!!!!) My patience is limited when dealing with someone playing games like this intentionally ignoring all my questions then writing a whole new paragraph of other non-related statements ....... This is what causes people to be mean on forums and mailing lists and I am doing my best to hold on to my patience. I am asking you question to try and weed out what your problem is, not to argue with you in any way. I bet when the cable guy tells you to reboot your device, you only pretend to and tell him you did it....... Why are you adding the manual-rtp-bugs if you don't even know what they do? They are all documented in the header file RTP_BUG_SEND_LINEAR_TIMESTAMPS = (1 << 3), /* Our friends at Sonus get real mad when the timestamps are not in perfect sequence even during periods of silence. With this flag, we will only increment the timestamp when write packets even if they are eons apart. */ RTP_BUG_NEVER_SEND_MARKER = (1 << 5), /* Our friends at Sonus are on a roll, They also get easily dumbfounded by marker bits. This flag will never send any. Sheesh.... */ If your packets are out of order it means your network is saturated or your network stack on your box is overloaded. Are you taking packet captures from your box sending the packets or the one receiving of a switch in between or both? Somewhere you have a problem...... if you want to use a jitter buffer to fix it, you can but you need it on the receiving box by using the jitterbuffer_ms variable set to the desired size in ms 60-80 recommended. <action application="set" data="jitterbuffer_msec=60"/> BEFORE you answer the call. once its up, at the cli you can do uuid_jitterbuffer <uuid> debug:DEBUG to get live stats...... This mailing list thread has turned into a bug report about a bug with no data to even look at and I hate putting issues on this mailing list. Within a few hours this email will be 5 pages outside the top of my inbox and I will have a hard time keeping track of it. I think I write 3 emails a day begging people to keep issues on Jira. That way we can at least get some confirmation what version the FS is and all the other On Fri, Feb 10, 2012 at 5:31 PM, Holger Freyther <holger@freyther.de> wrote:
Anthony Minessale <anthony.minessale@...> writes:
You are pushing RTP up against TDM, RTP can start and stop, TDM cannot, When there is no RTP the TDM will carry on sending the IDLE_FLAG byte over the wire.
I made both the RTP stack and the TDM endpoint from scratch so you should assume I may have some insight.
What I am arguing with... is that if I say a word like 'two' and RTP looks a bit like this.
RTP: Wall Time Seq.No Time 'Sound' 0.0 1 0 'T' 20ms* 2 160 'W' 40ms* 3 320 'O' ... some skew 100 100*160 'T' +20ms 101 101*160 'W' +20ms 102 102*160 'O'
and on the remote it arrives as: 'TW(?)O'..'T'..'O'
something is wrong with the playback speed/queue/buffer of the audio stack. All I ask for is (because I didn't see anything in the wiki about that):
1.) How do I check if the jitter buffer is enabled on a call. 2.) How do I check if arriving RTP packets arrived too late for the speed of the FreeTDM voice queue.
More details: I started with a setup like this:
RTP/AMR -> FreeSWITCH (sangoma transcoder) -> FreeTDM..
For debugging I changed it to:
AMR -> FreeSWITCH -> RTP/PCMA -> FreeSWITCH -> FreeTDM. (SIP on 5060) (SIP on 5062)
The idea was that the first FreeSWITCH has to deal with the jitter and the second receives a clean linear in wall clock time RTP stream. My sofia sip profile includes these params: <param name="rtp-timer-name" value="soft"/> <param name="manual-rtp-bugs" value="SEND_LINEAR_TIMESTAMPS|NEVER_SEND_MARKER|\n IGNORE_DTMF_DURATION" /> <param name="rtp-rewrite-timestamps" value="true"/>
The resulting output of the frontend FreeSWITCH is not linear though (so no idea what rtp-timer-name soft should do): But the resulting stream has packets like this: P.No Wall Time Codec 723 12.761045 G.711 PCMA, Seq=18533, Time=90720 731 12.841107 G.711 PCMA, Seq=18534, Time=90880
so somehow the expected arrival time was 17.78 but it ended up being 12.84 and I think FreeSWITCH/FreeTDM/STFU do not deal well with that.
any ideas? holger
* add a little jitter
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Anthony Minessale <anthony.minessale@...> writes:
You are arrogantly refusing to cooperate with my questions (why are people so stubborn!!!!)
I am sorry you have this impression. See, I am not coming here with "Please fix my problem, right now", the Sangoma techdesk helped me to verify the wanpipe/FreeTDM setup but the problem was not there. What I wanted and asked for is in the title of this email and in the body, I wanted to know how to inspect the jitter buffer as I couldn't find anything about that in the wiki. You were nice enough to answer that question in your last email. So thank you for that.
Why are you adding the manual-rtp-bugs if you don't even know what they do?
Well, I read the wiki page (otherwise I would not have found the options). For me knowing referred to actually know how this impacts switch_rtp.c and others. cheers holger
Also do not set the rewrite timestamps on the upstream box or the jitterbuffer will be useless. You need to preserve the originals or the jitter cannot be unraveled. On Feb 11, 2012 3:27 AM, "Holger Freyther" <holger@freyther.de> wrote:
Anthony Minessale <anthony.minessale@...> writes:
You are arrogantly refusing to cooperate with my questions (why are people so stubborn!!!!)
I am sorry you have this impression. See, I am not coming here with "Please fix my problem, right now", the Sangoma techdesk helped me to verify the wanpipe/FreeTDM setup but the problem was not there.
What I wanted and asked for is in the title of this email and in the body, I wanted to know how to inspect the jitter buffer as I couldn't find anything about that in the wiki. You were nice enough to answer that question in your last email. So thank you for that.
Why are you adding the manual-rtp-bugs if you don't even know what they
do?
Well, I read the wiki page (otherwise I would not have found the options). For me knowing referred to actually know how this impacts switch_rtp.c and others.
cheers holger
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Holger, Anthony, I'm seeing a similar/identical 'feature' where the voice is intermittently broken on ISDN; I've raised ticket #1310 with Sangoma. My MTU is still on the default 80 but there is no IDLE_FLAG setting (default 0, I guess?). Interestingly(!), it is intermittent and unpredictable, but when it manifests there is no clear pattern as to what will be disrupted; A wav audio played by a Lua IVR will be fine, then on the same call the MOH will be broken, as will the voicemail outgoing message and finally the internal user's speech. The path from ISDN into FreeSWITCH is never disrupted (no surprise). Record_session doesn't show the breaks and there's nothing in the FS logs. I have sample audio taken from another (SIP only) FreeSWITCH instance which demonstrates the problem, but as the fault is intermittent and on a production system, debugging is taking some time. Sangoma has asked me to take samples at the ISDN (S?) interface and the echo canceller on the B700 board. When I next have the problem manifest, and grab those traces, I should be able to share the results once Sangoma has decoded them. I can make available the audio samples I already have if that would help. John On 10/02/12 21:24, Anthony Minessale wrote:
That made no sense in regards to my last message.
I asked you to try particular wanpipe parameters, so you should try them and restart your wanpipe and report the results. Your MTU is very high and may cause timing issues for interop with voip.
set MTU to 80 (10ms) set IDLE_FLAG to 0xFF (send the 255 byte on idle stream)
1) Turn off FreeSWITCH 2) wanrouter stop 3) wanrouter start 4) start FreeSWITCH
You are pushing RTP up against TDM, RTP can start and stop, TDM cannot, When there is no RTP the TDM will carry on sending the IDLE_FLAG byte over the wire.
I made both the RTP stack and the TDM endpoint from scratch so you should assume I may have some insight.
On Fri, Feb 10, 2012 at 12:28 PM, Holger Freyther<holger@freyther.de> wrote:
Anthony Minessale<anthony.minessale@...> writes:
If there is no RTP then the TDM should be generating its own audio.
Try
MTU = 80 IDLE_FLAG = 0xFF Writing about audio is difficult. Imagine one says the word "Two".. and one called a mailbox. Opening the file in audacity gives you the wave of what you said, you zoom in and you see that in the middle of the word the the amplitude is zero for a bit (where it should not be zero). As echoing works on FreeTDM, there is no packet loss in RTP, the bug has the whole audio... it means something is broken in FreeSWITCH to get the audio to FreeTDM in time?
Any ideas?
participants (4)
-
Anthony Minessale
-
cmrienzo＠gmail.com
-
Holger Freyther
-
John