Session recording performance
I'm building an environment for voice quality testing, and found a strange behavior. Maybe someone has an explanation. Two identical physical computers (PC Engines APU2), connected back to back (A and B). A initiates the test calls to B's SIP address, and then both ends go through a small dialplan (less than 100 lines of XML, similar to https://github.com/voxserv/fsqa/blob/master/fsqa.xml and start recording the session. The dialplan initiates session recording to a Raw file, and plays back a 20-second audio sample, and then inserts the filename into a Redis queue, then hangs up. Then the files get processed by Sevana AQuA software in lowest CPU priority, and deleted. The files are written into a tmpfs filesystem. With 2 calls per second, I get 48 simultaneous calls, and B indicates frame drops in the audio recordings, while A shows good results. The CPU average idle time is always above 50% on both computers. Why is the B side (the one receiving INVITEs) getting more frame drops than the A (the one originating the calls)? I flipped the sides, and then the other machine started to get more drops. At first I started recording in WAV format, and similar effect started to appear at a call every 2 seconds, resulting in 12 simultaneous calls. With raw recording, the performance is much better, but still it's a mistery why there are dropped frames in the audio. The CPU is always running at its highest frequency, under "performance" governor.
Same kernel same everything? Are they "exact replicas" identical? Perhaps see if you can do this test again and receive the exact system calls A vs B are doing and then B to A should give you further insight to your answer. On Wed, Sep 20, 2017 at 6:46 AM Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
I'm building an environment for voice quality testing, and found a strange behavior. Maybe someone has an explanation.
Two identical physical computers (PC Engines APU2), connected back to back (A and B).
A initiates the test calls to B's SIP address, and then both ends go through a small dialplan (less than 100 lines of XML, similar to https://github.com/voxserv/fsqa/blob/master/fsqa.xml and start recording the session.
The dialplan initiates session recording to a Raw file, and plays back a 20-second audio sample, and then inserts the filename into a Redis queue, then hangs up. Then the files get processed by Sevana AQuA software in lowest CPU priority, and deleted. The files are written into a tmpfs filesystem.
With 2 calls per second, I get 48 simultaneous calls, and B indicates frame drops in the audio recordings, while A shows good results. The CPU average idle time is always above 50% on both computers.
Why is the B side (the one receiving INVITEs) getting more frame drops than the A (the one originating the calls)? I flipped the sides, and then the other machine started to get more drops.
At first I started recording in WAV format, and similar effect started to appear at a call every 2 seconds, resulting in 12 simultaneous calls. With raw recording, the performance is much better, but still it's a mistery why there are dropped frames in the audio.
The CPU is always running at its highest frequency, under "performance" governor.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from Gmail Mobile
Yes, totally identical configs and software. And yes, the effect moves to the other party if I change their roles. Another hypothesis is that the calling party is sending bad RTP for some reason. I'll collect the statistics. On 20 Sep 2017 17:30, "Brandon Armstead" <brandon@cryy.com> wrote:
Same kernel same everything?
Are they "exact replicas" identical?
Perhaps see if you can do this test again and receive the exact system calls A vs B are doing and then B to A should give you further insight to your answer.
On Wed, Sep 20, 2017 at 6:46 AM Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
I'm building an environment for voice quality testing, and found a strange behavior. Maybe someone has an explanation.
Two identical physical computers (PC Engines APU2), connected back to back (A and B).
A initiates the test calls to B's SIP address, and then both ends go through a small dialplan (less than 100 lines of XML, similar to https://github.com/voxserv/fsqa/blob/master/fsqa.xml and start recording the session.
The dialplan initiates session recording to a Raw file, and plays back a 20-second audio sample, and then inserts the filename into a Redis queue, then hangs up. Then the files get processed by Sevana AQuA software in lowest CPU priority, and deleted. The files are written into a tmpfs filesystem.
With 2 calls per second, I get 48 simultaneous calls, and B indicates frame drops in the audio recordings, while A shows good results. The CPU average idle time is always above 50% on both computers.
Why is the B side (the one receiving INVITEs) getting more frame drops than the A (the one originating the calls)? I flipped the sides, and then the other machine started to get more drops.
At first I started recording in WAV format, and similar effect started to appear at a call every 2 seconds, resulting in 12 simultaneous calls. With raw recording, the performance is much better, but still it's a mistery why there are dropped frames in the audio.
The CPU is always running at its highest frequency, under "performance" governor.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from Gmail Mobile
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
RTP flows seem to be ok so far. Also using mod_native_file did not help the situation, although it improved the performance. On Wed, Sep 20, 2017 at 5:53 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
Yes, totally identical configs and software. And yes, the effect moves to the other party if I change their roles.
Another hypothesis is that the calling party is sending bad RTP for some reason. I'll collect the statistics.
On 20 Sep 2017 17:30, "Brandon Armstead" <brandon@cryy.com> wrote:
Same kernel same everything?
Are they "exact replicas" identical?
Perhaps see if you can do this test again and receive the exact system calls A vs B are doing and then B to A should give you further insight to your answer.
On Wed, Sep 20, 2017 at 6:46 AM Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
I'm building an environment for voice quality testing, and found a strange behavior. Maybe someone has an explanation.
Two identical physical computers (PC Engines APU2), connected back to back (A and B).
A initiates the test calls to B's SIP address, and then both ends go through a small dialplan (less than 100 lines of XML, similar to https://github.com/voxserv/fsqa/blob/master/fsqa.xml and start recording the session.
The dialplan initiates session recording to a Raw file, and plays back a 20-second audio sample, and then inserts the filename into a Redis queue, then hangs up. Then the files get processed by Sevana AQuA software in lowest CPU priority, and deleted. The files are written into a tmpfs filesystem.
With 2 calls per second, I get 48 simultaneous calls, and B indicates frame drops in the audio recordings, while A shows good results. The CPU average idle time is always above 50% on both computers.
Why is the B side (the one receiving INVITEs) getting more frame drops than the A (the one originating the calls)? I flipped the sides, and then the other machine started to get more drops.
At first I started recording in WAV format, and similar effect started to appear at a call every 2 seconds, resulting in 12 simultaneous calls. With raw recording, the performance is much better, but still it's a mistery why there are dropped frames in the audio.
The CPU is always running at its highest frequency, under "performance" governor.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from Gmail Mobile
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I added <action application="jitterbuffer" data="40"/> and the drops reduced significantly (still testing to see if they're gone completely). So, probably the hardware, or NIC driver have added some jitter that caused the drops. On Wed, Sep 20, 2017 at 7:38 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
RTP flows seem to be ok so far. Also using mod_native_file did not help the situation, although it improved the performance.
On Wed, Sep 20, 2017 at 5:53 PM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
Yes, totally identical configs and software. And yes, the effect moves to the other party if I change their roles.
Another hypothesis is that the calling party is sending bad RTP for some reason. I'll collect the statistics.
On 20 Sep 2017 17:30, "Brandon Armstead" <brandon@cryy.com> wrote:
Same kernel same everything?
Are they "exact replicas" identical?
Perhaps see if you can do this test again and receive the exact system calls A vs B are doing and then B to A should give you further insight to your answer.
On Wed, Sep 20, 2017 at 6:46 AM Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
I'm building an environment for voice quality testing, and found a strange behavior. Maybe someone has an explanation.
Two identical physical computers (PC Engines APU2), connected back to back (A and B).
A initiates the test calls to B's SIP address, and then both ends go through a small dialplan (less than 100 lines of XML, similar to https://github.com/voxserv/fsqa/blob/master/fsqa.xml and start recording the session.
The dialplan initiates session recording to a Raw file, and plays back a 20-second audio sample, and then inserts the filename into a Redis queue, then hangs up. Then the files get processed by Sevana AQuA software in lowest CPU priority, and deleted. The files are written into a tmpfs filesystem.
With 2 calls per second, I get 48 simultaneous calls, and B indicates frame drops in the audio recordings, while A shows good results. The CPU average idle time is always above 50% on both computers.
Why is the B side (the one receiving INVITEs) getting more frame drops than the A (the one originating the calls)? I flipped the sides, and then the other machine started to get more drops.
At first I started recording in WAV format, and similar effect started to appear at a call every 2 seconds, resulting in 12 simultaneous calls. With raw recording, the performance is much better, but still it's a mistery why there are dropped frames in the audio.
The CPU is always running at its highest frequency, under "performance" governor.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sent from Gmail Mobile
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (2)
-
Brandon Armstead
-
Stanislav Sinyagin