[Freeswitch-users] Audio delays from conferences under heavy CPU load on Windows

Wed Feb 5 19:34:57 MSK 2014

assuming <local ip> is the ip of your box and <confA ext> and <confB ext>
are the 2 extensions of your conferences.

originate sofia/internal/<confA ext>@<local ip> <confB ext>

On Wed, Feb 5, 2014 at 8:38 AM, Markus von Arx <mkvonarx at gmail.com> wrote:

> Thanks a lot for you feedback and suggestions! Unfortunately my problem
> remains unsolved.
>
> Concerning your suggestions:
>
> 1) @Peter: starting FreeSwitchConsole.exe with -monotonic-clock
> => no visible change nor improvement.
>
> 2) @Michael: compareing local & remote Wireshark capture:
> => The delays are already visible on the local Wireshark capture via the
> RTP stream analysis: timestamps of audio on sent RTP stream is ~10 seconds
> later then the same audio on the received RTP stream (in my current test).
> Although I believe that there are no multi-seconds/minutes buffers in
> FreeSwitch itself, these delays must come from somewhere. Obviously from
> "inside" the system that Wireshark is observing. I still have no idea from
> where...
>
> 3) @Michael: Ethernet drivers updated to versions from 13.1.2014:
> => no visible change nor improvement.
>
> 4) @Anthony: hardware data:
> - CPU: Core i7-E610 2.53 GHz (from ~2010, Dual Core with HT), 4GB RAM
> - Ethernet #2 (in use): Intel(R) 82574L Gigabit Network Connection
> - Ethernet #1 (not used): Intel(R) 82577LM Gigabit Network Connection
> => I know that this is pretty old and slow hardware. Unfortunately this is
> what I have to use at the moment. I think the Ethernet cards are pretty
> standard. The goal is not to have thousands of channels on this machine.
> But to be stable in the region of around 500 channels (that would be my
> scenario 2 from above).
>
> 5) @Anthony: about using SIP instead of loopback channels:
> => How can I connect two conferences on the same FreeSwitch instance using
> SIP channels via a mod_event_socket command? I was not aware of this
> possibility until now. Wouldn't the originate command start somehow like
> "originate sofia/...." and expect the arguments directly after sofia/... to
> match to some external SIP entity?
>
> Any more ideas or suggestions?
>
> Thanks, Markus
>
>
>
> 2014-02-04 Anthony Minessale <anthony.minessale at gmail.com>:
>
> Also, what if you cross connect the conferences with sip channels instead
>> of loopback?
>>
>> 200 or 300 channels causing that much cpu usage may indicate an
>> underpowered box.
>> On Feb 4, 2014 7:16 AM, "Michael Jerris" <mike at jerris.com> wrote:
>>
>>>  There is nothing at all in freeswitch with a buffer big enough to
>>> account for those massive delays, so a timer issue would not explain this
>>> issue.  Could it be possible that your nic driver is actually holding
>>> packets that long.  How does the pcap look as far as that delay, try local
>>> pcap and remote and compare.  What type of nic is it?
>>>
>>> Mike
>>>
>>> On Feb 4, 2014, at 5:08 AM, Markus von Arx <mkvonarx at gmail.com> wrote:
>>>
>>> Hi
>>>
>>> I've got a nasty problem with FreeSwitch conferences: after some time, I
>>> observe that audio that goes through FreeSwitch conferences gets more and
>>> more delayed (by the FreeSwitch conferences). Sometimes it's only 2
>>> seconds, but I have also observerd audio delays up to 5:30 minutes. Audio
>>> is never missing, only delayed, and even transmitted correctly after many
>>> minutes of delays.
>>>
>>> More details:
>>> - FreeSwitch 1.2.17, 64bit
>>> - OS: Windows Server 2012 64bit and Windows 7 64bit (observed on both
>>> platforms)
>>> - audio codec: G.711 alaw only
>>> - external channels: only SIP channels (mod_sofia), RTP with 20ms
>>> packets, G.711 alaw, all SIP endpoints in the local LAN on the same
>>> Ethernet switch.
>>> - dialplan: very very simple; only two actions: answer and connect every
>>> incoming/outgoing SIP channel directly to its private conference
>>> (mod_conference); and I set and export the following variables on all
>>> channels: rtp_disable_vad_in=true, rtp_disable_vad_out=true,
>>> send_silence_when_idle=-1, bridge_generate_comfort_noise=-1,
>>> suppress_cng=true
>>> - my own application: does some logic over mod_event_socket (on
>>> localhost); basically connects/disconnects these conferences using loopback
>>> channels (mod_loopback)
>>>  -> simple example: two SIP channels from SIP users 1001 and 1002,
>>> dialplan creates two conferences 1001c and 1002c, my application connectes
>>> 1001c and 1002c with a loopback channel, audio from SIP user 1001 travels
>>> through SIP channel to FreeSwitch conference 1001c, then through loopback
>>> channel to conference 1002c, then through SIP channel to SIP user 1002. The
>>> audio delay I'm talking about is that what user 1001 speaks into his phone
>>> arrives at user 1002 N seconds later instead of almost immediately.
>>> - mod_conference config: rate=8000, interval=20, max-members=99,
>>> energy-level=0, *-sound="", comfort-noise=0, conference-flags=audio-always
>>> - mod_sofia: rtp-timer-name=soft, suppress-cng=true, vad=none
>>> - timer: soft timer (no other available on Windows)
>>>
>>> Usage scenarios:
>>> - failing scenario 1 (low load): 34 SIP channels, 34 conferences, 97
>>> loopback channels, 228 FreeSwitch channel objects (34 + 2*97)
>>>  -> ~24% CPU load
>>> -> audio delays only sometimes, a bit unpredictable, mostly none, but
>>> sometimes in the region of 2-3 seconds
>>> - failing scenario 2 (medium load): ...
>>> -> ~51% CPU load
>>>  -> audio delays start slowly, in the region of 2-10 seconds, grow
>>> slowly with time while keeping channels and conferences alive
>>> - failing scenario 3 (high load): 66 SIP channels, 66 conferences, 385
>>> loopback channels, 836 FreeSwitch channel objects (66 + 2*385)
>>>  -> ~74% CPU load
>>> -> audio delays start almost immediately: 18 seconds delay after 15
>>> minutes, 41 seconds delay after 3 hours, 5:30 minutes delay after 19 hours.
>>>
>>> => My guess: it could be that under heavy CPU load, the (soft) timers
>>> that are responsible for the RTP audio streams don't get fired reliably
>>> anymore but sometimes (often) too late.Theoretically this should not cause
>>> an ever increasing delay, as the next timer event after the delayed one
>>> should be queued a bit earlier. But still somehow the delay happens and it
>>> looks like the mixed RTP packets from the conferences are sent later
>>> and later and the delay is never recovered.
>>>
>>> => Any ideas? Is this a known issue? Maybe some combination of Windows,
>>> soft timers, high CPU load, loopback channels, disabled CNG/VAD, long
>>> running conferences? Maybe a FreeSwitch (timer) bug? Any help would be
>>> appreciated!
>>>
>>> Thanks and best regards,
>>> Markus
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140205/abf1a1dd/attachment-0001.html 