Freeswitch (incorrectly?) reporting packet loss via RTCP
I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote:
I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>_________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote:
Good point - but I have not enabled jitter buffer.
On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB.
On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote:
I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org <mailto:dragos@freeswitch.org>> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>_________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>_________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote:
No, they are audio only.
On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Are these video calls ? JB gets enabled by default for video.
On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote:
Good point - but I have not enabled jitter buffer.
On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB.
On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote:
I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file is finished playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed). I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets? Here, step by step, is what I observe: 1. I issue a command to play an mp3 file on the channel 2. When the file completes playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on. Therefore, every time a play file command ends, there appears to be a chance of dropping some packets. Below is some logging showing the above sequence of events (I’ve added some additional logging). So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we finish streaming an *outbound* audio file? What exactly are we trying to solve for here? Am I doing something wrong somehow? ******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] ********* On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
One correction to my description: the AUDIO_SYNC (and subsequent fliushed packets) happen on the start of the file play operation, not on its completion. Still need to understand why starting to play a file should cause us to drop incoming packets. On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote: I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file is finished playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed). I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets? Here, step by step, is what I observe: 1. I issue a command to play an mp3 file on the channel 2. When the file completes playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on. Therefore, every time a play file command ends, there appears to be a chance of dropping some packets. Below is some logging showing the above sequence of events (I’ve added some additional logging). So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we finish streaming an *outbound* audio file? What exactly are we trying to solve for here? Am I doing something wrong somehow? ******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] ********* On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Any response to this? On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote: I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed). I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets? Here, step by step, is what I observe: 1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on. Therefore, every time a play file command starts, there appears to be a chance of dropping some packets. Below is some logging showing the above sequence of events (I’ve added some additional logging). So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file? What exactly are we trying to solve for here? Am I doing something wrong somehow? ******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] ********* On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Could someone on the freeswitch team please respond to this? I spent a fair bit of time/effort gathering logs and investigating, and would like a quick answer to the question “why does freeswitch discard incoming RTP packets when starting to pay a file?” Dave On Nov 11, 2020, at 3:44 PM, David Horton <daveh@drachtio.org> wrote: Any response to this? On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote: I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed). I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets? Here, step by step, is what I observe: 1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on. Therefore, every time a play file command starts, there appears to be a chance of dropping some packets. Below is some logging showing the above sequence of events (I’ve added some additional logging). So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file? What exactly are we trying to solve for here? Am I doing something wrong somehow? ******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] ********* On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Hi Dave, I appreciate the time you and many in our community spend understanding how freeswitch works when reporting issues. For audio, RTP is usually handled by the same thread that is executing dialplan apps. Dialplan apps must not spend a lot of time on blocking operations otherwise RTP will not be processed in time. switch_ivr_play_file() is one of many core functions dialplan apps execute and it is responsible for reading and writing audio frames. Knowing that FS can fall behind, it is a common pattern in FS to flush buffers prior to playing. In 1-legged calls this should have no bad effect, however you've noticed a side-effect of that in the RTCP reports. Chris On Fri, Nov 20, 2020 at 12:57 PM David Horton <daveh@drachtio.org> wrote:
Could someone on the freeswitch team please respond to this? I spent a fair bit of time/effort gathering logs and investigating, and would like a quick answer to the question “why does freeswitch discard incoming RTP packets when starting to pay a file?”
Dave
On Nov 11, 2020, at 3:44 PM, David Horton <daveh@drachtio.org> wrote:
Any response to this?
On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote:
I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed).
I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets?
Here, step by step, is what I observe:
1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on.
Therefore, every time a play file command starts, there appears to be a chance of dropping some packets.
Below is some logging showing the above sequence of events (I’ve added some additional logging).
So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file?
What exactly are we trying to solve for here?
Am I doing something wrong somehow?
******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/ nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] *********
On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers.
On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only.
On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Are these video calls ? JB gets enabled by default for video.
On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer.
On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB.
On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Thanks for the insight on this — exactly what I wanted to know. In my deployments I’ve seen two side affects to this: 1) lower quality results from “brittle” downstream voice analysis systems that don’t do well with missing packets (e.g some speaker verification systems that I’ve integrated with) 2) consistent reports of poor QoS on calls to freeswitch which can lead to a lot of time/effort trying to troubleshoot things on the network etc when that isnt really the cause. Now that I know the background on this I can consider the best path for me to take in my applications; e.g. fork/patch or accept/ignore. Thanks again, Dave On Nov 20, 2020, at 2:53 PM, Christopher Rienzo <cmrienzo@gmail.com> wrote: Hi Dave, I appreciate the time you and many in our community spend understanding how freeswitch works when reporting issues. For audio, RTP is usually handled by the same thread that is executing dialplan apps. Dialplan apps must not spend a lot of time on blocking operations otherwise RTP will not be processed in time. switch_ivr_play_file() is one of many core functions dialplan apps execute and it is responsible for reading and writing audio frames. Knowing that FS can fall behind, it is a common pattern in FS to flush buffers prior to playing. In 1-legged calls this should have no bad effect, however you've noticed a side-effect of that in the RTCP reports. Chris On Fri, Nov 20, 2020 at 12:57 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: Could someone on the freeswitch team please respond to this? I spent a fair bit of time/effort gathering logs and investigating, and would like a quick answer to the question “why does freeswitch discard incoming RTP packets when starting to pay a file?” Dave On Nov 11, 2020, at 3:44 PM, David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: Any response to this? On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed). I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets? Here, step by step, is what I observe: 1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on. Therefore, every time a play file command starts, there appears to be a chance of dropping some packets. Below is some logging showing the above sequence of events (I’ve added some additional logging). So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file? What exactly are we trying to solve for here? Am I doing something wrong somehow? ******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/nobody@172.31.32.120:5060 <http://nobody@172.31.32.120:5060/> audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] ********* On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org <mailto:dragos@freeswitch.org>> wrote: Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers. On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: No, they are audio only. On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org <mailto:dragos@freeswitch.org>> wrote: Are these video calls ? JB gets enabled by default for video. On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: Good point - but I have not enabled jitter buffer. On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org <mailto:dragos@freeswitch.org>> wrote: What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB. On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org <mailto:daveh@drachtio.org>> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch? dave _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/> _________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com <https://signalwire.com/> Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community <https://signalwire.community/> Professional FreeSWITCH Services sales@freeswitch.com <mailto:sales@freeswitch.com> https://freeswitch.com <https://freeswitch.com/> Official FreeSWITCH Sites https://freeswitch.com/oss <https://freeswitch.com/oss> https://freeswitch.org/confluence <https://freeswitch.org/confluence> https://cluecon.com <https://cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users> https://freeswitch.com <https://freeswitch.com/>_________________________________________________________________________ The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform. Join our online community to chat in real time https://signalwire.community Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
Wow, event the thanks is bad... On Fri, 20 Nov 2020 at 20:40, David Horton <daveh@drachtio.org> wrote:
Thanks for the insight on this — exactly what I wanted to know. In my deployments I’ve seen two side affects to this:
1) lower quality results from “brittle” downstream voice analysis systems that don’t do well with missing packets (e.g some speaker verification systems that I’ve integrated with) 2) consistent reports of poor QoS on calls to freeswitch which can lead to a lot of time/effort trying to troubleshoot things on the network etc when that isnt really the cause.
Now that I know the background on this I can consider the best path for me to take in my applications; e.g. fork/patch or accept/ignore.
Thanks again,
Dave
On Nov 20, 2020, at 2:53 PM, Christopher Rienzo <cmrienzo@gmail.com> wrote:
Hi Dave,
I appreciate the time you and many in our community spend understanding how freeswitch works when reporting issues.
For audio, RTP is usually handled by the same thread that is executing dialplan apps. Dialplan apps must not spend a lot of time on blocking operations otherwise RTP will not be processed in time. switch_ivr_play_file() is one of many core functions dialplan apps execute and it is responsible for reading and writing audio frames. Knowing that FS can fall behind, it is a common pattern in FS to flush buffers prior to playing. In 1-legged calls this should have no bad effect, however you've noticed a side-effect of that in the RTCP reports.
Chris
On Fri, Nov 20, 2020 at 12:57 PM David Horton <daveh@drachtio.org> wrote:
Could someone on the freeswitch team please respond to this? I spent a fair bit of time/effort gathering logs and investigating, and would like a quick answer to the question “why does freeswitch discard incoming RTP packets when starting to pay a file?”
Dave
On Nov 11, 2020, at 3:44 PM, David Horton <daveh@drachtio.org> wrote:
Any response to this?
On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote:
I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed).
I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets?
Here, step by step, is what I observe:
1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on.
Therefore, every time a play file command starts, there appears to be a chance of dropping some packets.
Below is some logging showing the above sequence of events (I’ve added some additional logging).
So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file?
What exactly are we trying to solve for here?
Am I doing something wrong somehow?
******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/ nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] *********
On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers.
On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only.
On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Are these video calls ? JB gets enabled by default for video.
On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer.
On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB.
On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Regards, David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
Dave, While on the subject of time and effort Imagine how much time/effort has gone into creating the project in the first place and how offensive your statements are that because you are anxious for an answer that you start replying to your own post demanding a response from a community forum especially the FreeSWITCH team who has literally been fostering this codebase for over 15 years. You also opened a GITHUB ticket asking the same question, a more appropriate forum for such a discussion IMHO. So IDK where to even answer you now. Also, this is an open-source project. Did you try changing the code around at all and build a case for making a change? Off the top of my head, I'd say try enabling the jitterbuffer because the sync thing is a fallback to dealing with delay when no jb is available. I think it's clear in the code to skip flushing any packets when the jb exists since its much better at managing network delay than a raw udp socket. Perhaps the use of audio sync in this file playing situation is not prudent. You could try removing it and make a PR etc. On Fri, Nov 20, 2020 at 11:55 AM David Horton <daveh@drachtio.org> wrote:
Could someone on the freeswitch team please respond to this? I spent a fair bit of time/effort gathering logs and investigating, and would like a quick answer to the question “why does freeswitch discard incoming RTP packets when starting to pay a file?”
Dave
On Nov 11, 2020, at 3:44 PM, David Horton <daveh@drachtio.org> wrote:
Any response to this?
On Nov 7, 2020, at 9:26 PM, David Horton <daveh@drachtio.org> wrote:
I have investigated this further, and for some reason the issue seems to be related to the fact that I am playing a file on the channel, and when the file starts playing freeswitch for some reason flushes packets, which then occasionally leads to freeswitch reporting lost packets (the very packets it flushed).
I’m lacking some information to fully understand this — most importantly, I would like to know what the function of doing an AUDIO_SYNC on the channel when a file finishes playing, and why this should cause freeswitch to flush incoming packets?
Here, step by step, is what I observe:
1. I issue a command to play an mp3 file on the channel 2. When the file starts playing, I see freeswitch issue an AUDIO_SYNC event 3. This causes freeswitch to set the SWITCH_RTP_FLAG_FLUSH flag on the channel 4. Freeswitch then does a non-blocking read on the UDP socket and discards any packets it receives — this is where incoming packets get dropped on the floor — not all the times, only if the non-blocking read happens just as a packet was received at the network layer 5. Once the non-blocking read fails, freeswitch turns off the SWITCH_RTP_FLAG_FLUSH flag and continues on.
Therefore, every time a play file command starts, there appears to be a chance of dropping some packets.
Below is some logging showing the above sequence of events (I’ve added some additional logging).
So, for the freeswitch team — can you provide some background info on why we would want to discard *incoming* packets when we start streaming an *outbound* audio file?
What exactly are we trying to solve for here?
Am I doing something wrong somehow?
******** EXECUTE [depth=1] sofia/drachtio_mrf/nobody@172.31.32.120:5060 playback(/tmp/tts-02413eaa-538f-49cc-be9f-54f71e45f389e572411e7d5b2eda396848cd8c4ff467c47513a9.mp3) 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:2889 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [APPLICATION_EXEC] 2020-11-08 02:14:49.087939 [ALERT] switch_core_session.c:1047 Send signal sofia/drachtio_mrf/nobody@172.31.32.120:5060 [BREAK] 2020-11-08 02:14:49.087939 [DEBUG] switch_ivr_play_say.c:1492 Codec Activated L16@8000hz 1 channels 20ms 2020-11-08 02:14:49.087939 [ALERT] switch_ivr_play_say.c:1423 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [AUDIO_SYNC] 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7627 starting to flush packets 2020-11-08 02:14:49.087939 [ERR] switch_rtp.c:5704 *** do_flush: total flushed packets: 2 *** 2020-11-08 02:14:49.087939 [DEBUG] switch_rtp.c:7631 done flushing packets 2020-11-08 02:14:49.087939 [ALERT] switch_core_media.c:15808 sofia/drachtio_mrf/nobody@172.31.32.120:5060 receive message [TRANSCODING_NECESSARY] 2020-11-08 02:14:49.107937 [ERR] switch_rtp.c:6123 Missed 2 RTP frames from sequence [11395] to [11396] (flushed by FS). Time since last read [19998] 2020-11-08 02:14:49.307936 [ALERT] switch_rtp.c:1572 sofia/drachtio_mrf/ nobody@172.31.32.120:5060 audio 2 consecutive flaws, adding 2 flaw penalty 2020-11-08 02:14:49.727937 [ALERT] switch_rtp.c:1846 LOST PACKETS!! expected 53, got 51, highest: 11427, last_rpt_ext_seq 11374, extra 0 2020-11-08 02:14:49.727937 [CRIT] switch_rtp.c:1867 rtcp_generate_sr: stats_ssrc[288672665] received[51] expected[53] cum[3] lost[2|9/256]pkt last_seq[11427] cyc[0] last_rpt_seq[11374] cyc[0] ssrc[288672665] *********
On Nov 5, 2020, at 11:12 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Chris pointed out that the packets need to be there, at the app level, when FS wants them, even if the JB is not used. And how are you sure that those packets are not actually lost for FS ? What network stats ? Also, OS/network interfaces have very large buffers.
On Wed, Nov 4, 2020 at 5:29 PM David Horton <daveh@drachtio.org> wrote: No, they are audio only.
On Nov 4, 2020, at 10:22 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
Are these video calls ? JB gets enabled by default for video.
On Wed, Nov 4, 2020 at 5:12 PM David Horton <daveh@drachtio.org> wrote: Good point - but I have not enabled jitter buffer.
On Nov 4, 2020, at 10:08 AM, Dragos Oancea <dragos@freeswitch.org> wrote:
What's in the RTCP report is what comes after passing through the Jitter Buffer, if there's jitter , some packets are late and they will be dropped by JB.
On Tue, Nov 3, 2020 at 7:26 PM David Horton <daveh@drachtio.org> wrote: I have a freeswitch server v1.10.1 running on AWS EC2 which I have noticed is reporting lost packets in the RTCP reports it is sending out. This happens even with a single call. Meanwhile, the network stats on the same instances show no dropped packets and no packet receive errors. Is there anything which could explain this? Have others seen errors in the RTCP stats reported by freeswitch?
dave _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com _________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
_________________________________________________________________________
The FreeSWITCH project is sponsored by SignalWire https://signalwire.com Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services. Build your next product on our scalable cloud platform.
Join our online community to chat in real time https://signalwire.community
Professional FreeSWITCH Services sales@freeswitch.com https://freeswitch.com
Official FreeSWITCH Sites https://freeswitch.com/oss https://freeswitch.org/confluence https://cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users https://freeswitch.com
-- Anthony Minessale II Founder, FreeSWITCH. http://freeswitch.com https://youtu.be/l_hOxzCt6X4 https://www.youtube.com/watch?v=oAxXgyx5jUw https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
participants (5)
-
Anthony Minessale
-
Christopher Rienzo
-
David Horton
-
David Villasmil
-
Dragos Oancea