Sequence numbers are created on outgoing call leg without taking into account lost sequence number on incoming call leg
Hello, It would be much appreciated to have some advice on following use case. We establish a call on a Freeswitch by creating two call legs separately and then bridged them together: Client <--Call Leg A--> Freeswtich <--Call Leg B--> Gstreamer During a call there are packet losses on Call Leg A between Client and Freeswitch and we have appropriate notifications in a Freeswitch debug logs. As well as it's clear from the pcap logs taken from a Freeswitch that RTP packets are missing with certain sequence number. Since Client and Gstreamer using same Opus codec Freeswitch just sends packets straightly with same payload and timestamp to Gstreamer. The only thing is changed in RTP packets are SSRC and Sequence Numbers. In a case when RTP packet is lost sequence numbers are not preserved between Call Leg A and Call Leg B. So Call Leg B just got packets which are make it up to Freeswitch. That means that Call Leg B gets for all received on a Freeswitch RTP packets consecutive sequence numbers (unfortunately lost packets are not count) and so Gstreamer is not aware of the packet loss which took precedence on a previous Call Leg A. Due to this since all RTP packets are consecutive on Call Leg B Gstreamer have no notion of a lost packet. So it is not starting a procedure of Forward Error Correction (FEC) on a lost packet and eventually we got audio drops which are not handled. As well as Freeswitch is not decoding payload so it doesn't do FEC on Opus side. That's why we are wondering what could be a solution for that? From first glance into RTP RFC it seems ok to have such behavior, i.e. not take into account previous sequence numbers for a packet loss. But maybe it's just not considering such cases. Is there other RFCs which cover such situations? We thinking maybe it make sense to have a patch to FS so that we would receive if not the same but at least have the same gaps in the sequence numbers as on Call Leg A? Or maybe some other solution to that problem which is available on a Freeswitch? If you need us to share any FS logs will be happy to provide. I'll attach pcap logs for the time being. 188.130.168.205 - Client 10.240.0.12 - Freeswitch 10.240.0.7 - Gstreamer Please take a look at following filter. Where 24632 is missing packet on Call leg A: rtp.seq== 24631 || rtp.seq== 24632 || rtp.seq== 24633 || rtp.seq == 56583 || rtp.seq == 56582 ___________ Best Regards, Dmitriy Ageev
This is why audio jitter buffers need to look at timestamp and not seq numbers to determine lost packets. Its required by the RFC to always write new seq numbers when generating a new stream. On Mon, Jan 30, 2017 at 10:12 AM, Dmitriy Ageev <dmitriy.a@soniccloud.com> wrote:
Hello,
It would be much appreciated to have some advice on following use case.
We establish a call on a Freeswitch by creating two call legs separately and then bridged them together:
Client <--Call Leg A--> Freeswtich <--Call Leg B--> Gstreamer
During a call there are packet losses on Call Leg A between Client and Freeswitch and we have appropriate notifications in a Freeswitch debug logs. As well as it's clear from the pcap logs taken from a Freeswitch that RTP packets are missing with certain sequence number.
Since Client and Gstreamer using same Opus codec Freeswitch just sends packets straightly with same payload and timestamp to Gstreamer. The only thing is changed in RTP packets are SSRC and Sequence Numbers.
In a case when RTP packet is lost sequence numbers are not preserved between Call Leg A and Call Leg B. So Call Leg B just got packets which are make it up to Freeswitch. That means that Call Leg B gets for all received on a Freeswitch RTP packets consecutive sequence numbers (unfortunately lost packets are not count) and so Gstreamer is not aware of the packet loss which took precedence on a previous Call Leg A.
Due to this since all RTP packets are consecutive on Call Leg B Gstreamer have no notion of a lost packet. So it is not starting a procedure of Forward Error Correction (FEC) on a lost packet and eventually we got audio drops which are not handled. As well as Freeswitch is not decoding payload so it doesn't do FEC on Opus side.
That's why we are wondering what could be a solution for that? From first glance into RTP RFC it seems ok to have such behavior, i.e. not take into account previous sequence numbers for a packet loss. But maybe it's just not considering such cases. Is there other RFCs which cover such situations?
We thinking maybe it make sense to have a patch to FS so that we would receive if not the same but at least have the same gaps in the sequence numbers as on Call Leg A?
Or maybe some other solution to that problem which is available on a Freeswitch?
If you need us to share any FS logs will be happy to provide.
I’ll attach pcap logs for the time being.
188.130.168.205 - Client
10.240.0.12 – Freeswitch
10.240.0.7 – Gstreamer
Please take a look at following filter. Where 24632 is missing packet on Call leg A:
rtp.seq== 24631 || rtp.seq== 24632 || rtp.seq== 24633 || rtp.seq == 56583 || rtp.seq == 56582
___________
Best Regards,
Dmitriy Ageev
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Thank you for quick response. Maybe it’s not clearly related but we have another question related to the situation described below. Probably partially due to the spoken problem with sequence numbers we clearly observe audio drops in previous situation. But we have another use case when we make a direct client to client call with Opus codec through Freesiwtch with invoking media bugs. And in such case we don’t notice any audio drops. So from this there are questions in which we would be thankful to have answer: - Maybe there are some difference in how Freesiwtch handles packet out of order or lost packets when media bug is involved? Why there is a clear difference in audio for these two situations because in both cases we don’t enable Freeswitch Jitter Buffer? - I assume that the reason is that for media bug it performs opus decoding so mod_opus performs accurate FEC based on the sequence number and maybe some other opus internal work. Also looking into mod_opus code we see that even jitter buffer is used there. Is that correct? Is there something we are missing? - Would you recommend to enable jitter buffer at Freeswitch for the use case below? Thanks a lot for all you efforts in Freeswitch and making it clear to us on how it works. ___________ Best Regards, Dmitriy Ageev From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Monday, January 30, 2017 7:54 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Sequence numbers are created on outgoing call leg without taking into account lost sequence number on incoming call leg This is why audio jitter buffers need to look at timestamp and not seq numbers to determine lost packets. Its required by the RFC to always write new seq numbers when generating a new stream. On Mon, Jan 30, 2017 at 10:12 AM, Dmitriy Ageev <dmitriy.a@soniccloud.com<mailto:dmitriy.a@soniccloud.com>> wrote: Hello, It would be much appreciated to have some advice on following use case. We establish a call on a Freeswitch by creating two call legs separately and then bridged them together: Client <--Call Leg A--> Freeswtich <--Call Leg B--> Gstreamer During a call there are packet losses on Call Leg A between Client and Freeswitch and we have appropriate notifications in a Freeswitch debug logs. As well as it's clear from the pcap logs taken from a Freeswitch that RTP packets are missing with certain sequence number. Since Client and Gstreamer using same Opus codec Freeswitch just sends packets straightly with same payload and timestamp to Gstreamer. The only thing is changed in RTP packets are SSRC and Sequence Numbers. In a case when RTP packet is lost sequence numbers are not preserved between Call Leg A and Call Leg B. So Call Leg B just got packets which are make it up to Freeswitch. That means that Call Leg B gets for all received on a Freeswitch RTP packets consecutive sequence numbers (unfortunately lost packets are not count) and so Gstreamer is not aware of the packet loss which took precedence on a previous Call Leg A. Due to this since all RTP packets are consecutive on Call Leg B Gstreamer have no notion of a lost packet. So it is not starting a procedure of Forward Error Correction (FEC) on a lost packet and eventually we got audio drops which are not handled. As well as Freeswitch is not decoding payload so it doesn't do FEC on Opus side. That's why we are wondering what could be a solution for that? From first glance into RTP RFC it seems ok to have such behavior, i.e. not take into account previous sequence numbers for a packet loss. But maybe it's just not considering such cases. Is there other RFCs which cover such situations? We thinking maybe it make sense to have a patch to FS so that we would receive if not the same but at least have the same gaps in the sequence numbers as on Call Leg A? Or maybe some other solution to that problem which is available on a Freeswitch? If you need us to share any FS logs will be happy to provide. I’ll attach pcap logs for the time being. 188.130.168.205 - Client 10.240.0.12 – Freeswitch 10.240.0.7 – Gstreamer Please take a look at following filter. Where 24632 is missing packet on Call leg A: rtp.seq== 24631 || rtp.seq== 24632 || rtp.seq== 24633 || rtp.seq == 56583 || rtp.seq == 56582 ___________ Best Regards, Dmitriy Ageev _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Media bugs can force a transcoding situation in some cases because it must decode the audio in order to tap into it. If you use the jitterbuffer with opus its possible to configure it to use PLC to handle packet loss. https://freeswitch.org/confluence/display/FREESWITCH/FreeSWITCH+And+The+Opus... On Tue, Jan 31, 2017 at 1:02 PM, Dmitriy Ageev <dmitriy.a@soniccloud.com> wrote:
Thank you for quick response.
Maybe it’s not clearly related but we have another question related to the situation described below. Probably partially due to the spoken problem with sequence numbers we clearly observe audio drops in previous situation.
But we have another use case when we make a direct client to client call with Opus codec through Freesiwtch with invoking media bugs. And in such case we don’t notice any audio drops.
So from this there are questions in which we would be thankful to have answer:
- Maybe there are some difference in how Freesiwtch handles packet out of order or lost packets when media bug is involved? Why there is a clear difference in audio for these two situations because in both cases we don’t enable Freeswitch Jitter Buffer?
- I assume that the reason is that for media bug it performs opus decoding so mod_opus performs accurate FEC based on the sequence number and maybe some other opus internal work. Also looking into mod_opus code we see that even jitter buffer is used there. Is that correct? Is there something we are missing?
- Would you recommend to enable jitter buffer at Freeswitch for the use case below?
Thanks a lot for all you efforts in Freeswitch and making it clear to us on how it works.
___________
Best Regards,
Dmitriy Ageev
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Monday, January 30, 2017 7:54 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Sequence numbers are created on outgoing call leg without taking into account lost sequence number on incoming call leg
This is why audio jitter buffers need to look at timestamp and not seq numbers to determine lost packets.
Its required by the RFC to always write new seq numbers when generating a new stream.
On Mon, Jan 30, 2017 at 10:12 AM, Dmitriy Ageev <dmitriy.a@soniccloud.com> wrote:
Hello,
It would be much appreciated to have some advice on following use case.
We establish a call on a Freeswitch by creating two call legs separately and then bridged them together:
Client <--Call Leg A--> Freeswtich <--Call Leg B--> Gstreamer
During a call there are packet losses on Call Leg A between Client and Freeswitch and we have appropriate notifications in a Freeswitch debug logs. As well as it's clear from the pcap logs taken from a Freeswitch that RTP packets are missing with certain sequence number.
Since Client and Gstreamer using same Opus codec Freeswitch just sends packets straightly with same payload and timestamp to Gstreamer. The only thing is changed in RTP packets are SSRC and Sequence Numbers.
In a case when RTP packet is lost sequence numbers are not preserved between Call Leg A and Call Leg B. So Call Leg B just got packets which are make it up to Freeswitch. That means that Call Leg B gets for all received on a Freeswitch RTP packets consecutive sequence numbers (unfortunately lost packets are not count) and so Gstreamer is not aware of the packet loss which took precedence on a previous Call Leg A.
Due to this since all RTP packets are consecutive on Call Leg B Gstreamer have no notion of a lost packet. So it is not starting a procedure of Forward Error Correction (FEC) on a lost packet and eventually we got audio drops which are not handled. As well as Freeswitch is not decoding payload so it doesn't do FEC on Opus side.
That's why we are wondering what could be a solution for that? From first glance into RTP RFC it seems ok to have such behavior, i.e. not take into account previous sequence numbers for a packet loss. But maybe it's just not considering such cases. Is there other RFCs which cover such situations?
We thinking maybe it make sense to have a patch to FS so that we would receive if not the same but at least have the same gaps in the sequence numbers as on Call Leg A?
Or maybe some other solution to that problem which is available on a Freeswitch?
If you need us to share any FS logs will be happy to provide.
I’ll attach pcap logs for the time being.
188.130.168.205 - Client
10.240.0.12 – Freeswitch
10.240.0.7 – Gstreamer
Please take a look at following filter. Where 24632 is missing packet on Call leg A:
rtp.seq== 24631 || rtp.seq== 24632 || rtp.seq== 24633 || rtp.seq == 56583 || rtp.seq == 56582
___________
Best Regards,
Dmitriy Ageev
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888@conference.freeswitch.org ☎ +19193869900 <(919)%20386-9900>
https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
participants (2)
-
Anthony Minessale
-
Dmitriy Ageev