Audio delays from conferences under heavy CPU load on Windows
Hi I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays. More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows) Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours. => My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered. => Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated! Thanks and best regards, Markus
There is nothing at all in freeswitch with a buffer big enough to account for those massive delays, so a timer issue would not explain this issue. Could it be possible that your nic driver is actually holding packets that long. How does the pcap look as far as that delay, try local pcap and remote and compare. What type of nic is it? Mike On Feb 4, 2014, at 5:08 AM, Markus von Arx <mkvonarx@gmail.com> wrote:
Hi
I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.
More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows)
Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.
=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!
Thanks and best regards, Markus
Also, what if you cross connect the conferences with sip channels instead of loopback? 200 or 300 channels causing that much cpu usage may indicate an underpowered box. On Feb 4, 2014 7:16 AM, "Michael Jerris" <mike@jerris.com> wrote:
There is nothing at all in freeswitch with a buffer big enough to account for those massive delays, so a timer issue would not explain this issue. Could it be possible that your nic driver is actually holding packets that long. How does the pcap look as far as that delay, try local pcap and remote and compare. What type of nic is it?
Mike
On Feb 4, 2014, at 5:08 AM, Markus von Arx <mkvonarx@gmail.com> wrote:
Hi
I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.
More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows)
Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.
=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!
Thanks and best regards, Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks a lot for you feedback and suggestions! Unfortunately my problem remains unsolved. Concerning your suggestions: 1) @Peter: starting FreeSwitchConsole.exe with -monotonic-clock => no visible change nor improvement. 2) @Michael: compareing local & remote Wireshark capture: => The delays are already visible on the local Wireshark capture via the RTP stream analysis: timestamps of audio on sent RTP stream is ~10 seconds later then the same audio on the received RTP stream (in my current test). Although I believe that there are no multi-seconds/minutes buffers in FreeSwitch itself, these delays must come from somewhere. Obviously from "inside" the system that Wireshark is observing. I still have no idea from where... 3) @Michael: Ethernet drivers updated to versions from 13.1.2014: => no visible change nor improvement. 4) @Anthony: hardware data: - CPU: Core i7-E610 2.53 GHz (from ~2010, Dual Core with HT), 4GB RAM - Ethernet #2 (in use): Intel(R) 82574L Gigabit Network Connection - Ethernet #1 (not used): Intel(R) 82577LM Gigabit Network Connection => I know that this is pretty old and slow hardware. Unfortunately this is what I have to use at the moment. I think the Ethernet cards are pretty standard. The goal is not to have thousands of channels on this machine. But to be stable in the region of around 500 channels (that would be my scenario 2 from above). 5) @Anthony: about using SIP instead of loopback channels: => How can I connect two conferences on the same FreeSwitch instance using SIP channels via a mod_event_socket command? I was not aware of this possibility until now. Wouldn't the originate command start somehow like "originate sofia/...." and expect the arguments directly after sofia/... to match to some external SIP entity? Any more ideas or suggestions? Thanks, Markus 2014-02-04 Anthony Minessale <anthony.minessale@gmail.com>:
Also, what if you cross connect the conferences with sip channels instead of loopback?
200 or 300 channels causing that much cpu usage may indicate an underpowered box. On Feb 4, 2014 7:16 AM, "Michael Jerris" <mike@jerris.com> wrote:
There is nothing at all in freeswitch with a buffer big enough to account for those massive delays, so a timer issue would not explain this issue. Could it be possible that your nic driver is actually holding packets that long. How does the pcap look as far as that delay, try local pcap and remote and compare. What type of nic is it?
Mike
On Feb 4, 2014, at 5:08 AM, Markus von Arx <mkvonarx@gmail.com> wrote:
Hi
I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.
More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows)
Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.
=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!
Thanks and best regards, Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
assuming <local ip> is the ip of your box and <confA ext> and <confB ext> are the 2 extensions of your conferences. originate sofia/internal/<confA ext>@<local ip> <confB ext> On Wed, Feb 5, 2014 at 8:38 AM, Markus von Arx <mkvonarx@gmail.com> wrote:
Thanks a lot for you feedback and suggestions! Unfortunately my problem remains unsolved.
Concerning your suggestions:
1) @Peter: starting FreeSwitchConsole.exe with -monotonic-clock => no visible change nor improvement.
2) @Michael: compareing local & remote Wireshark capture: => The delays are already visible on the local Wireshark capture via the RTP stream analysis: timestamps of audio on sent RTP stream is ~10 seconds later then the same audio on the received RTP stream (in my current test). Although I believe that there are no multi-seconds/minutes buffers in FreeSwitch itself, these delays must come from somewhere. Obviously from "inside" the system that Wireshark is observing. I still have no idea from where...
3) @Michael: Ethernet drivers updated to versions from 13.1.2014: => no visible change nor improvement.
4) @Anthony: hardware data: - CPU: Core i7-E610 2.53 GHz (from ~2010, Dual Core with HT), 4GB RAM - Ethernet #2 (in use): Intel(R) 82574L Gigabit Network Connection - Ethernet #1 (not used): Intel(R) 82577LM Gigabit Network Connection => I know that this is pretty old and slow hardware. Unfortunately this is what I have to use at the moment. I think the Ethernet cards are pretty standard. The goal is not to have thousands of channels on this machine. But to be stable in the region of around 500 channels (that would be my scenario 2 from above).
5) @Anthony: about using SIP instead of loopback channels: => How can I connect two conferences on the same FreeSwitch instance using SIP channels via a mod_event_socket command? I was not aware of this possibility until now. Wouldn't the originate command start somehow like "originate sofia/...." and expect the arguments directly after sofia/... to match to some external SIP entity?
Any more ideas or suggestions?
Thanks, Markus
2014-02-04 Anthony Minessale <anthony.minessale@gmail.com>:
Also, what if you cross connect the conferences with sip channels instead
of loopback?
200 or 300 channels causing that much cpu usage may indicate an underpowered box. On Feb 4, 2014 7:16 AM, "Michael Jerris" <mike@jerris.com> wrote:
There is nothing at all in freeswitch with a buffer big enough to account for those massive delays, so a timer issue would not explain this issue. Could it be possible that your nic driver is actually holding packets that long. How does the pcap look as far as that delay, try local pcap and remote and compare. What type of nic is it?
Mike
On Feb 4, 2014, at 5:08 AM, Markus von Arx <mkvonarx@gmail.com> wrote:
Hi
I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.
More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows)
Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.
=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!
Thanks and best regards, Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Try to start with the param "--monotonic-clock", it might enable a more accurate clock on these kind of systems. 2014-02-04 Markus von Arx <mkvonarx@gmail.com>:
Hi
I've got a nasty problem with FreeSwitch conferences: after some time, I observe that audio that goes through FreeSwitch conferences gets more and more delayed (by the FreeSwitch conferences). Sometimes it's only 2 seconds, but I have also observerd audio delays up to 5:30 minutes. Audio is never missing, only delayed, and even transmitted correctly after many minutes of delays.
More details: - FreeSwitch 1.2.17, 64bit - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both platforms) - audio codec: G.711 alaw only - external channels: only SIP channels (mod_sofia), RTP with 20ms packets, G.711 alaw, all SIP endpoints in the local LAN on the same Ethernet switch. - dialplan: very very simple; only two actions: answer and connect every incoming/outgoing SIP channel directly to its private conference (mod_conference); and I set and export the following variables on all channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true, send_silence_when_idle=-1, bridge_generate_comfort_noise=-1, suppress_cng=true - my own application: does some logic over mod_event_socket (on localhost); basically connects/disconnects these conferences using loopback channels (mod_loopback) -> simple example: two SIP channels from SIP users 1001 and 1002, dialplan creates two conferences 1001c and 1002c, my application connectes 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels through SIP channel to FreeSwitch conference 1001c, then through loopback channel to conference 1002c, then through SIP channel to SIP user 1002. The audio delay I'm talking about is that what user 1001 speaks into his phone arrives at user 1002 N seconds later instead of almost immediately. - mod_conference config: rate=8000, interval=20, max-members=99, energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none - timer: soft timer (no other available on Windows)
Usage scenarios: - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97 loopback channels, 228 FreeSwitch channel objects (34 + 2*97) -> ~24% CPU load -> audio delays only sometimes, a bit unpredictable, mostly none, but sometimes in the region of 2-3 seconds - failing scenario 2 (medium load): ... -> ~51% CPU load -> audio delays start slowly, in the region of 2-10 seconds, grow slowly with time while keeping channels and conferences alive - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385 loopback channels, 836 FreeSwitch channel objects (66 + 2*385) -> ~74% CPU load -> audio delays start almost immediately: 18 seconds delay after 15 minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
=> My guess: it could be that under heavy CPU load, the (soft) timers that are responsible for the RTP audio streams don't get fired reliably anymore but sometimes (often) too late.Theoretically this should not cause an ever increasing delay, as the next timer event after the delayed one should be queued a bit earlier. But still somehow the delay happens and it looks like the mixed RTP packets from the conferences are sent later and later and the delay is never recovered.
=> Any ideas? Is this a known issue? Maybe some combination of Windows, soft timers, high CPU load, loopback channels, disabled CNG/VAD, long running conferences? Maybe a FreeSwitch (timer) bug? Any help would be appreciated!
Thanks and best regards, Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (4)
-
Anthony Minessale
-
Markus von Arx
-
Michael Jerris
-
Peter Olsson