rtp-timer-name / timer issues
Hey list, I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances). Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting. Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify? My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies? Thanks in advance. Best, Colin
What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West. Running FS Installed from YUM. CentOS7.
On Feb 5, 2018, at 5:56 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Ubuntu 16.04 LTS. Do you have any custom kernel settings or just a stock CentOS instance? On Mon, Feb 5, 2018 at 8:23 PM Geoff Mina <gmina@connectfirst.com> wrote:
What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West.
Running FS Installed from YUM. CentOS7.
On Feb 5, 2018, at 5:56 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Stock CentOS w/ Kernel 3.10.0-693.el7.x86_64. Freeswitch 1.6.19~64bit We have TLS Websocket /DTLS-SRTP connections on one side and standard UDP SIP on the other. Both sides using G.711, so no transcoding. On Mon, Feb 5, 2018 at 7:07 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Ubuntu 16.04 LTS. Do you have any custom kernel settings or just a stock CentOS instance?
On Mon, Feb 5, 2018 at 8:23 PM Geoff Mina <gmina@connectfirst.com> wrote:
What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West.
Running FS Installed from YUM. CentOS7.
On Feb 5, 2018, at 5:56 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/ options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Appreciate the info. We're running on Kernel 4.4.0, with freeswitch 1.9.0 (compiled from git). Large differences on versions there but this seems so fundamental and core to FS that I'd be surprised if later versions made things worse. That said, I'm not sure when things like timerfd became standard in freeswitch, and/or if that (or related concerns) could cause the issue. We're also running in Docker, but we run it without any limits (and can replicate the issue running outside of a conctainer). Similar situation, TLS Websocket w/ DTLS-SRTP on one side and standard TLS+SRTP on the other. All of our issues appear to occur on the WebRTC legs, however, which would leads me to lean towards network issues being the primary culprit. However, iperf shows 400Mbps easily (and consistently) between the endpoints. Issue occurs whether we do G.711 on both ends, or opus on the public internet side (although opus PLC can help mask some of the issues) Just to be clear, the audio is still intelligible, there's just noticeable dropped packets every 5-10 seconds on average. Do you know what timer your instance is using? And your settings for rtp-timer-name? Thanks again, Colin On Mon, Feb 5, 2018 at 9:28 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Stock CentOS w/ Kernel 3.10.0-693.el7.x86_64. Freeswitch 1.6.19~64bit
We have TLS Websocket /DTLS-SRTP connections on one side and standard UDP SIP on the other. Both sides using G.711, so no transcoding.
On Mon, Feb 5, 2018 at 7:07 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Ubuntu 16.04 LTS. Do you have any custom kernel settings or just a stock CentOS instance?
On Mon, Feb 5, 2018 at 8:23 PM Geoff Mina <gmina@connectfirst.com> wrote:
What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West.
Running FS Installed from YUM. CentOS7.
On Feb 5, 2018, at 5:56 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Alright, so I've captured a recording on the PSTN leg and WebRTC leg simultaneously. On the PSTN leg, the audio is perfectly smooth with no dropped packets. The recording on the WebRTC leg does have noticeable dropped packets. I believe this would all but eliminate network concerns, unless I'm misunderstanding something. Additionally, running iperf between a machine that experiences the issue, and the media server, easily pushes !00Mbps with < 0.5ms jitter. This happens on multiple client internet connections ranging from 50Mbps to 1Gbps, both residential and commercial. Any guidance on diving deeper into timer issues? Or other settings to change? Thanks, Colin On Mon, Feb 5, 2018 at 9:38 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Appreciate the info. We're running on Kernel 4.4.0, with freeswitch 1.9.0 (compiled from git). Large differences on versions there but this seems so fundamental and core to FS that I'd be surprised if later versions made things worse. That said, I'm not sure when things like timerfd became standard in freeswitch, and/or if that (or related concerns) could cause the issue. We're also running in Docker, but we run it without any limits (and can replicate the issue running outside of a conctainer).
Similar situation, TLS Websocket w/ DTLS-SRTP on one side and standard TLS+SRTP on the other. All of our issues appear to occur on the WebRTC legs, however, which would leads me to lean towards network issues being the primary culprit. However, iperf shows 400Mbps easily (and consistently) between the endpoints. Issue occurs whether we do G.711 on both ends, or opus on the public internet side (although opus PLC can help mask some of the issues)
Just to be clear, the audio is still intelligible, there's just noticeable dropped packets every 5-10 seconds on average.
Do you know what timer your instance is using? And your settings for rtp-timer-name?
Thanks again, Colin
On Mon, Feb 5, 2018 at 9:28 PM, Geoff Mina <gmina@connectfirst.com> wrote:
Stock CentOS w/ Kernel 3.10.0-693.el7.x86_64. Freeswitch 1.6.19~64bit
We have TLS Websocket /DTLS-SRTP connections on one side and standard UDP SIP on the other. Both sides using G.711, so no transcoding.
On Mon, Feb 5, 2018 at 7:07 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Ubuntu 16.04 LTS. Do you have any custom kernel settings or just a stock CentOS instance?
On Mon, Feb 5, 2018 at 8:23 PM Geoff Mina <gmina@connectfirst.com> wrote:
What OS are you using? We run FS at over 600 concurrent calls on a smaller EC2 (c4.xlarge) size and have not had any issues in either US-East or US-West.
Running FS Installed from YUM. CentOS7.
On Feb 5, 2018, at 5:56 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/free switch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/free switch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Colin, Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP> B On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Appreciate the input, Brian. I’ll definitely try to avoid setting the timer option. In other news. I deployed The exact same FS instance (same docker container) on baremetal last night and it experiences the same issue. So, virtualization does not appear to be the problem. I just can’t figure out what else would cause this. I’m sure it’s something simple. On Tue, Feb 6, 2018 at 3:49 AM Brian : <brians@iptel.co> wrote:
Hi Colin,
Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP>
B
On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
What WebRTC client are you using? Does this happen in all browsers or just one?
On Feb 6, 2018, at 5:09 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Appreciate the input, Brian. I’ll definitely try to avoid setting the timer option.
In other news. I deployed The exact same FS instance (same docker container) on baremetal last night and it experiences the same issue. So, virtualization does not appear to be the problem. I just can’t figure out what else would cause this. I’m sure it’s something simple.
On Tue, Feb 6, 2018 at 3:49 AM Brian : <brians@iptel.co> wrote: Hi Colin,
Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP>
B
On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Happens on all browsers. Just want to clarify my previous message, though. I had a call bridged A -> B (A is the WebRTC side, B the PSTN). I recorded both legs of the call individually. On the recording for B, B's audio is clear and smooth. On the recording for A, B's audio has dropped packets that correspond with the logs mentioned on FS. Unless I'm misunderstanding something I believe this should eliminate network/WebRTC/clients as being the issue. On Tue, Feb 6, 2018 at 8:45 AM, Geoff Mina <gmina@connectfirst.com> wrote:
What WebRTC client are you using? Does this happen in all browsers or just one?
On Feb 6, 2018, at 5:09 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Appreciate the input, Brian. I’ll definitely try to avoid setting the timer option.
In other news. I deployed The exact same FS instance (same docker container) on baremetal last night and it experiences the same issue. So, virtualization does not appear to be the problem. I just can’t figure out what else would cause this. I’m sure it’s something simple. On Tue, Feb 6, 2018 at 3:49 AM Brian : <brians@iptel.co> wrote:
Hi Colin,
Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP>
B
On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/ options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Tested again on a fresh EC2 instances (c5.xlarge) running Debian Jessie (Kernel 3.16.0-4-amd64), since I believe that's the current recommendation, with packages installed from the Freeswitch mainline (version 1.6.20-37-987c9b9~64bit) and vanilla configs. I am still able to reproduce issues where one side's audio recording drops packets that are present in the other side. Running out of things to look at here, since I was able to repro on baremetal as well. On Tue, Feb 6, 2018 at 9:04 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Happens on all browsers.
Just want to clarify my previous message, though. I had a call bridged A -> B (A is the WebRTC side, B the PSTN). I recorded both legs of the call individually. On the recording for B, B's audio is clear and smooth. On the recording for A, B's audio has dropped packets that correspond with the logs mentioned on FS. Unless I'm misunderstanding something I believe this should eliminate network/WebRTC/clients as being the issue.
On Tue, Feb 6, 2018 at 8:45 AM, Geoff Mina <gmina@connectfirst.com> wrote:
What WebRTC client are you using? Does this happen in all browsers or just one?
On Feb 6, 2018, at 5:09 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Appreciate the input, Brian. I’ll definitely try to avoid setting the timer option.
In other news. I deployed The exact same FS instance (same docker container) on baremetal last night and it experiences the same issue. So, virtualization does not appear to be the problem. I just can’t figure out what else would cause this. I’m sure it’s something simple. On Tue, Feb 6, 2018 at 3:49 AM Brian : <brians@iptel.co> wrote:
Hi Colin,
Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP>
B
On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/ freeswitch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Alright, it looks like CNG may be to blame here. We were using bridge_generate_comfort_noise to deal with some provider issues and it may be responding poorly to even the slightest latency. If my understanding is correct, bridge_generate_comfort_noise will generate a silence packet into the media stream transparently to the endpoints. The result would be that if the timer hits and no audio is available on a channel, CNG is immediately generated and placed into the stream, rendering jitter buffers on either end fairly useless, since they're still receiving consistent audio streams (just some packets may be silent) Does this sound reasonable? Short of timing issues it's the only thing I can think of that would cause the audio differences between the A and B leg given that a dedicated host didn't solve the problem either. Thanks in advance, Colin On Tue, Feb 6, 2018 at 10:32 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Tested again on a fresh EC2 instances (c5.xlarge) running Debian Jessie (Kernel 3.16.0-4-amd64), since I believe that's the current recommendation, with packages installed from the Freeswitch mainline (version 1.6.20-37-987c9b9~64bit) and vanilla configs. I am still able to reproduce issues where one side's audio recording drops packets that are present in the other side. Running out of things to look at here, since I was able to repro on baremetal as well.
On Tue, Feb 6, 2018 at 9:04 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Happens on all browsers.
Just want to clarify my previous message, though. I had a call bridged A -> B (A is the WebRTC side, B the PSTN). I recorded both legs of the call individually. On the recording for B, B's audio is clear and smooth. On the recording for A, B's audio has dropped packets that correspond with the logs mentioned on FS. Unless I'm misunderstanding something I believe this should eliminate network/WebRTC/clients as being the issue.
On Tue, Feb 6, 2018 at 8:45 AM, Geoff Mina <gmina@connectfirst.com> wrote:
What WebRTC client are you using? Does this happen in all browsers or just one?
On Feb 6, 2018, at 5:09 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Appreciate the input, Brian. I’ll definitely try to avoid setting the timer option.
In other news. I deployed The exact same FS instance (same docker container) on baremetal last night and it experiences the same issue. So, virtualization does not appear to be the problem. I just can’t figure out what else would cause this. I’m sure it’s something simple. On Tue, Feb 6, 2018 at 3:49 AM Brian : <brians@iptel.co> wrote:
Hi Colin,
Depending on what you are doing with Freeswitch setting the rtp-timer to none can produce all sorts of subtle weirdness. I would advise against it. 2 things that I remember from our tests with this was lots of blocked / hung calls that would build and need to be HUPed and also carriers that would send SDP but no RTP when silence was being sent - the call wouldn't progress through dialplan - it just got blocked waiting on RTP>
B
On Tue, Feb 6, 2018 at 12:56 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey list,
I'm running FS on EC2 (I know, I know). Having some issues with random packet loss, which I believe almost certainly I have narrowed down to timer issues and/or network latency/jitter (seems surprising since I'm using c5.xlarge instances).
Behavior is that, during a call, brief pauses or notable audio loss will occur. This is on high bandwidth links that are otherwise stable. Freeswitch logs with max debug spew out "Hot Hit 1" through "Hot Hit 10" and eventually "auto-flush catching up 1 packet(s)" in rapid succession (usually going through the cycle 4-5 times) before things settle again. Obviously that means a minimum of 4-5 audio packets were dropped within the span of a second which results in considerable audio artifacting.
Changing rtp-timer-name to none, which I understand to perform synchronous reads of RTP audio (as opposed to timer-based async reads) makes the audio notably smoother. That said, I'm having a hard time uncovering the consequences of doing this. Obviously I understand that reads will block the RTP thread, but I can't seem to understand the potential ramifications of this. Could anyone help clarify?
My other question is: assuming "timer while hot" indicates what I believe it does (that when the timer hit there was >1 packet in the queue to be read), couldn't this issue also just be caused by network jitter, and not necessarily just timer inconsistencies?
Thanks in advance.
Best, Colin
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/free switch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/free switch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
____________________________________________________________ _____________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (3)
-
Brian :
-
Colin Morelli
-
Geoff Mina