<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">There is nothing at all in freeswitch with a buffer big enough to account for those massive delays, so a timer issue would not explain this issue. Could it be possible that your nic driver is actually holding packets that long. How does the pcap look as far as that delay, try local pcap and remote and compare. What type of nic is it?<div><br></div><div>Mike</div><div><br><div><div>On Feb 4, 2014, at 5:08 AM, Markus von Arx <<a href="mailto:mkvonarx@gmail.com">mkvonarx@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div class="gmail_default"><div class="gmail_default"><font face="arial, helvetica, sans-serif">Hi</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default">
<font face="arial, helvetica, sans-serif">I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">More details:</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">- FreeSwitch 1.2.17, 64bit</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms)</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">- audio codec: G.711 alaw only</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch.</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback)</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately.</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">- timer: soft timer (no other available on Windows)</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Usage scenarios:</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">- failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97)</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> ~24% CPU load</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- failing scenario 2 (medium load): ...</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> ~51% CPU load</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">- failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385)</font></div><div class="gmail_default">
<font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> ~74% CPU load</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>-> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like </font><span style="font-family:arial,helvetica,sans-serif">the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.</span></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Thanks and best regards,</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Markus</font></div>
</div></div></blockquote></div><br></div></body></html>