[Freeswitch-users] Audio delays from conferences under heavy CPU load on Windows

[Markus von Arx]

Hi

I've got a nasty problem with FreeSwitch conferences: after some time, I
observe that audio that goes through FreeSwitch conferences gets more and
more delayed (by the FreeSwitch conferences). Sometimes it's only 2
seconds, but I have also observerd audio delays up to 5:30 minutes. Audio
is never missing, only delayed, and even transmitted correctly after many
minutes of delays.

More details:
- FreeSwitch 1.2.17, 64bit
- OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both
platforms)
- audio codec: G.711 alaw only
- external channels: only SIP channels (mod_sofia), RTP with 20ms packets,
G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch.
- dialplan: very very simple; only two actions: answer and connect every
incoming/outgoing SIP channel directly to its private conference
(mod_conference); and I set and export the following variables on all
channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true,
send_silence_when_idle=-1, bridge_generate_comfort_noise=-1,
suppress_cng=true
- my own application: does some logic over mod_event_socket (on localhost);
basically connects/disconnects these conferences using loopback channels
(mod_loopback)
-> simple example: two SIP channels from SIP users 1001 and 1002, dialplan
creates two conferences 1001c and 1002c, my application connectes 1001c and
1002c with a loopback channel, audio from SIP user 1001 travels through SIP
channel to FreeSwitch conference 1001c, then through loopback channel to
conference 1002c, then through SIP channel to SIP user 1002. The audio
delay I'm talking about is that what user 1001 speaks into his phone
arrives at user 1002 N seconds later instead of almost immediately.
- mod_conference config: rate=8000, interval=20, max-members=99,
energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always
- mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none
- timer: soft timer (no other available on Windows)

Usage scenarios:
- failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97
loopback channels, 228 FreeSwitch channel objects (34 + 2*97)
-> ~24% CPU load
-> audio delays only sometimes, a bit unpredictable, mostly none, but
sometimes in the region of 2-3 seconds
- failing scenario 2 (medium load): ...
-> ~51% CPU load
-> audio delays start slowly, in the region of 2-10 seconds, grow slowly
with time while keeping channels and conferences alive
- failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385
loopback channels, 836 FreeSwitch channel objects (66 + 2*385)
-> ~74% CPU load
-> audio delays start almost immediately: 18 seconds delay after 15
minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.

=> My guess: it could be that under heavy CPU load, the (soft) timers that
are responsible for the RTP audio streams don't get fired reliably anymore
but sometimes (often) too late.Theoretically this should not cause an ever
increasing delay, as the next timer event after the delayed one should be
queued a bit earlier. But still somehow the delay happens and it looks like the
mixed RTP packets from the conferences are sent later and later and the
delay is never recovered.

=> Any ideas? Is this a known issue? Maybe some combination of Windows,
soft timers, high CPU load, loopback channels, disabled CNG/VAD, long
running conferences? Maybe a FreeSwitch (timer) bug? Any help would be
appreciated!

Thanks and best regards,
Markus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140204/38382738/attachment.html