<div dir="ltr">Alright, so I've captured a recording on the PSTN leg and WebRTC leg simultaneously. On the PSTN leg, the audio is perfectly smooth with no dropped packets. The recording on the WebRTC leg does have noticeable dropped packets. I believe this would all but eliminate network concerns, unless I'm misunderstanding something.<div><br></div><div>Additionally, running iperf between a machine that experiences the issue, and the media server, easily pushes !00Mbps with < 0.5ms jitter. This happens on multiple client internet connections ranging from 50Mbps to 1Gbps, both residential and commercial.</div><div><br></div><div>Any guidance on diving deeper into timer issues? Or other settings to change?</div><div><br></div><div>Thanks,</div><div>Colin</div><div><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Feb 5, 2018 at 9:38 PM, Colin Morelli <span dir="ltr"><<a href="mailto:colin.morelli@gmail.com" target="_blank">colin.morelli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Appreciate the info. We're running on Kernel 4.4.0, with freeswitch 1.9.0 (compiled from git). Large differences on versions there but this seems so fundamental and core to FS that I'd be surprised if later versions made things worse. That said, I'm not sure when things like timerfd became standard in freeswitch, and/or if that (or related concerns) could cause the issue. We're also running in Docker, but we run it without any limits (and can replicate the issue running outside of a conctainer).<div><br></div><div>Similar situation, TLS Websocket w/ DTLS-SRTP on one side and standard TLS+SRTP on the other. All of our issues appear to occur on the WebRTC legs, however, which would leads me to lean towards network issues being the primary culprit. However, iperf shows 400Mbps easily (and consistently) between the endpoints. Issue occurs whether we do G.711 on both ends, or opus on the public internet side (although opus PLC can help mask some of the issues)</div><div><br></div><div>Just to be clear, the audio is still intelligible, there's just noticeable dropped packets every 5-10 seconds on average.</div><div><br></div><div>Do you know what timer your instance is using? And your settings for rtp-timer-name?</div><div><br></div><div>Thanks again,</div><div>Colin</div><div><div class="h5"><div><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Feb 5, 2018 at 9:28 PM, Geoff Mina <span dir="ltr"><<a href="mailto:gmina@connectfirst.com" target="_blank">gmina@connectfirst.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Stock CentOS w/ Kernel 3.10.0-693.el7.x86_64.  Freeswitch 1.6.19~64bit<div><br></div><div>We have TLS Websocket /DTLS-SRTP connections on one side and standard UDP SIP on the other.  Both sides using G.711, so no transcoding.</div><div><div class="m_3165570796628489145h5"><div><br></div><div><br><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Mon, Feb 5, 2018 at 7:07 PM, Colin Morelli <span dir="ltr"><<a href="mailto:colin.morelli@gmail.com" target="_blank">colin.morelli@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Ubuntu 16.04 LTS. Do you have any custom kernel settings or just a stock CentOS instance?<div class="m_3165570796628489145m_-5749850677826923188gmail-HOEnZb"><div class="m_3165570796628489145m_-5749850677826923188gmail-h5"><br><div class="gmail_quote"><div dir="ltr">On Mon, Feb 5, 2018 at 8:23 PM Geoff Mina <<a href="mailto:gmina@connectfirst.com" target="_blank">gmina@connectfirst.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West.<br>
<br>
Running FS Installed from YUM. CentOS7.<br>
<br>
> On Feb 5, 2018, at 5:56 PM, Colin Morelli <<a href="mailto:colin.morelli@gmail.com" target="_blank">colin.morelli@gmail.com</a>> wrote:<br>
><br>
> Hey list,<br>
><br>
> I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).<br>
><br>
> Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.<br>
><br>
> Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?<br>
><br>
> My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?<br>
><br>
> Thanks in advance.<br>
><br>
> Best,<br>
> Colin<br>
><br>
><br>
> ______________________________<wbr>______________________________<wbr>_____________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions<wbr>.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.o<wbr>rg</a><br>
> <a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-users mailing list<br>
> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a><br>
> <a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<br>
______________________________<wbr>______________________________<wbr>_____________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions<wbr>.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.o<wbr>rg</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a></blockquote></div>
</div></div><br>______________________________<wbr>______________________________<wbr>_____________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions<wbr>.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.o<wbr>rg</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/free<wbr>switch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div></div></div></div></div>
<br>______________________________<wbr>______________________________<wbr>_____________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions<wbr>.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.o<wbr>rg</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswi<wbr>tch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/ma<wbr>ilman/listinfo/freeswitch-user<wbr>s</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.frees<wbr>witch.org/mailman/options/<wbr>freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div>