Hi all, I'm testing Freeswitch using ESL interface, recording and SIP UDP Calls at a rate of around 5 CPS. Freeswitch receives ESL requests and proceeds to send call to PSTN Emulator, once call is answered I start recording. I have noticed around 40% CPU utilization when recording is not enabled and ~60%+ CPU when recording is. I created/mount a volume specifically to store the recording files. (ext4) *Server specifications:* EC2 server (t2.medium) 2 vCPUs, 2.5 GHz, Intel Xeon Family, 4 GiB memory. FreeSWITCH (Version 1.6.9 -16-d574870 64bit) I followed instructions at: https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations *[Questions]* a) How to strip down Freeswitch and remove non-used modules? Features I need: *esl, recording, loopback, bridge, sip, spandsp, cdr_csv.* My modules: *fs_cli -x "show modules"* http://pastebin.com/BQSxzFDE b) I would like to see which module is using X/Y amount of CPU cycles/memory in Freeswitch is it possible?. *fs_cli -x 'show status'* UP 0 years, 0 days, 1 hour, 2 minutes, 15 seconds, 541 milliseconds, 286 microseconds FreeSWITCH (Version 1.6.9 -16-d574870 64bit) is ready 18192 session(s) since startup 90 session(s) - peak 93, last 5min 92 5 session(s) per Sec out of max 30, peak 7, last 5min 6 1000 session(s) max min idle cpu 0.00/75.57 Current Stack Size/Max 240K/8192K A sample image is below: 40% steady (no recording) a peak when I enabled recording. [image: Inline image 1] # top top - 03:25:16 up 22:42, 1 user, load average: 3.88, 5.74, 10.39 Tasks:* 68 *total,* 1 *running,* 67 *sleeping,* 0 *stopped,* 0 * zombie %Cpu(s):* 11.0 *us,* 2.3 *sy,* 0.0 *ni,* 85.7 *id,* 0.0 *wa,* 0.0 *hi,* 0.7 *si,* 0.4 *st KiB Mem: * 4057584 *total,* 865964 *used,* 3191620 *free,* 147088 * buffers KiB Swap:* 1048572 *total,* 0 *used,* 1048572 *free.* 213572 *cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26217 freeswi+ -2 -10 1427444 302584 16904 S *65.9* 7.5 28:47.49 freeswitch
How are you actually recording the files? Native? mp3? wav? Something else? keep in mind that recording also means disk io which means more CPU while its recording to handle the IO load… there are way too many undisclosed factors here that can and will impact your performance… one example is 2 cores? That’s all? FS is a highly threaded application… that means for each call leg there is atleast 1 thread… so the more cores the merrier… not to mention you are running on amazon that leaves a whole different set of variables to attend too…. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Gonzalo Gasca Meza Sent: Sunday, October 2, 2016 10:43 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: [Freeswitch-users] Performance testing Hi all, I'm testing Freeswitch using ESL interface, recording and SIP UDP Calls at a rate of around 5 CPS. Freeswitch receives ESL requests and proceeds to send call to PSTN Emulator, once call is answered I start recording. I have noticed around 40% CPU utilization when recording is not enabled and ~60%+ CPU when recording is. I created/mount a volume specifically to store the recording files. (ext4) Server specifications: EC2 server (t2.medium) 2 vCPUs, 2.5 GHz, Intel Xeon Family, 4 GiB memory. FreeSWITCH (Version 1.6.9 -16-d574870 64bit) I followed instructions at: https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations [Questions] a) How to strip down Freeswitch and remove non-used modules? Features I need: esl, recording, loopback, bridge, sip, spandsp, cdr_csv. My modules: fs_cli -x "show modules" http://pastebin.com/BQSxzFDE b) I would like to see which module is using X/Y amount of CPU cycles/memory in Freeswitch is it possible?. fs_cli -x 'show status' UP 0 years, 0 days, 1 hour, 2 minutes, 15 seconds, 541 milliseconds, 286 microseconds FreeSWITCH (Version 1.6.9 -16-d574870 64bit) is ready 18192 session(s) since startup 90 session(s) - peak 93, last 5min 92 5 session(s) per Sec out of max 30, peak 7, last 5min 6 1000 session(s) max min idle cpu 0.00/75.57 Current Stack Size/Max 240K/8192K A sample image is below: 40% steady (no recording) a peak when I enabled recording. # top top - 03:25:16 up 22:42, 1 user, load average: 3.88, 5.74, 10.39 Tasks: 68 total, 1 running, 67 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.0 us, 2.3 sy, 0.0 ni, 85.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.4 st KiB Mem: 4057584 total, 865964 used, 3191620 free, 147088 buffers KiB Swap: 1048572 total, 0 used, 1048572 free. 213572 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26217 freeswi+ -2 -10 1427444 302584 16904 S 65.9 7.5 28:47.49 freeswitch
Hi Ken, I'm using .wav, this is my ESL command: originate {sip_h_X-UUID=9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89,sip_h_X-Campaign=3A252BEX9W,ignore_early_media=false,bridge_early_media=false,originate_timeout=70,bridge_answer_timeout=70,call_timeout=70,[leg_timeout=70],execute_on_answer_1='sched_hangup 18 ALLOTTED_TIMEOUT',origination_uuid=9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89,origination_caller_id_number=+14081116805,RECORD_STEREO=true,execute_on_answer_2='record_session /var/www/html/recordings/9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89.wav'}sofia/gateway/kamailio/+14080001963 handle_calls I created a separate volume for writing only recording files & logs. I will consider increasing CPUs, do you think I still can remove some of the modules I currently using to improve performance? Thanks! On Sun, Oct 2, 2016 at 9:07 PM, Ken Rice <krice@freeswitch.org> wrote:
How are you actually recording the files? Native? mp3? wav? Something else?
keep in mind that recording also means disk io which means more CPU while its recording to handle the IO load… there are way too many undisclosed factors here that can and will impact your performance…
one example is 2 cores? That’s all? FS is a highly threaded application… that means for each call leg there is atleast 1 thread… so the more cores the merrier…
not to mention you are running on amazon that leaves a whole different set of variables to attend too….
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Gonzalo Gasca Meza *Sent:* Sunday, October 2, 2016 10:43 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Performance testing
Hi all,
I'm testing Freeswitch using ESL interface, recording and SIP UDP Calls at a rate of around 5 CPS.
Freeswitch receives ESL requests and proceeds to send call to PSTN Emulator, once call is answered I start recording. I have noticed around 40% CPU utilization when recording is not enabled and ~60%+ CPU when recording is.
I created/mount a volume specifically to store the recording files. (ext4)
*Server specifications:*
EC2 server (t2.medium) 2 vCPUs, 2.5 GHz, Intel Xeon Family, 4 GiB memory.
FreeSWITCH (Version 1.6.9 -16-d574870 64bit)
I followed instructions at: https://wiki.freeswitch. org/wiki/Performance_testing_and_configurations
*[Questions]*
a) How to strip down Freeswitch and remove non-used modules?
Features I need:
*esl, recording, loopback, bridge, sip, spandsp, cdr_csv.*
My modules:
*fs_cli -x "show modules"*
b) I would like to see which module is using X/Y amount of CPU cycles/memory in Freeswitch is it possible?.
*fs_cli -x 'show status'*
UP 0 years, 0 days, 1 hour, 2 minutes, 15 seconds, 541 milliseconds, 286 microseconds
FreeSWITCH (Version 1.6.9 -16-d574870 64bit) is ready
18192 session(s) since startup
90 session(s) - peak 93, last 5min 92
5 session(s) per Sec out of max 30, peak 7, last 5min 6
1000 session(s) max
min idle cpu 0.00/75.57
Current Stack Size/Max 240K/8192K
A sample image is below: 40% steady (no recording) a peak when I enabled recording.
[image: Inline image 1]
# top
top - 03:25:16 up 22:42, 1 user, load average: 3.88, 5.74, 10.39
Tasks:* 68 *total,* 1 *running,* 67 *sleeping,* 0 *stopped,* 0 * zombie
%Cpu(s):* 11.0 *us,* 2.3 *sy,* 0.0 *ni,* 85.7 *id,* 0.0 *wa,* 0.0 *hi,* 0.7 *si,* 0.4 *st
KiB Mem: * 4057584 *total,* 865964 *used,* 3191620 *free,* 147088 * buffers
KiB Swap:* 1048572 *total,* 0 *used,* 1048572 *free.* 213572 *cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26217 freeswi+ -2 -10 1427444 302584 16904 S *65.9* 7.5 28:47.49 freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
well consider you are doing 90 transcode sessions there…. Not sure what the actual outbound codec is, but that’s just a guess based on 90 session(s) - peak 93, last 5min 92… theres also what else is going on with the call that you have to contend with… obviously you are creating some sort of dialer… and for help with that and optimization I would contact the FS core team @ consulting@freeswitch.org <mailto:consulting@freeswitch.org> for some pro help… the reason we don’t release any soft of scaling metrics is changing 1 setting can invalidate the testing… things like recording, transcoding, amount of media handling, any call scripting and specifically what else is going on there in the media processing… as far as capacity planning for doing something like you are doing, I wouldn’t even bother with less than a 12 or 24 vcore machine From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Gonzalo Gasca Meza Sent: Sunday, October 2, 2016 11:19 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Performance testing Hi Ken, I'm using .wav, this is my ESL command: originate {sip_h_X-UUID=9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89,sip_h_X-Campaign=3A252BEX9W,ignore_early_media=false,bridge_early_media=false,originate_timeout=70,bridge_answer_timeout=70,call_timeout=70,[leg_timeout=70],execute_on_answer_1='sched_hangup 18 ALLOTTED_TIMEOUT',origination_uuid=9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89,origination_caller_id_number=+14081116805,RECORD_STEREO=true,execute_on_answer_2='record_session /var/www/html/recordings/9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89.wav'}sofia/gateway/kamailio/+14080001963 handle_calls I created a separate volume for writing only recording files & logs. I will consider increasing CPUs, do you think I still can remove some of the modules I currently using to improve performance? Thanks! On Sun, Oct 2, 2016 at 9:07 PM, Ken Rice <krice@freeswitch.org <mailto:krice@freeswitch.org> > wrote: How are you actually recording the files? Native? mp3? wav? Something else? keep in mind that recording also means disk io which means more CPU while its recording to handle the IO load… there are way too many undisclosed factors here that can and will impact your performance… one example is 2 cores? That’s all? FS is a highly threaded application… that means for each call leg there is atleast 1 thread… so the more cores the merrier… not to mention you are running on amazon that leaves a whole different set of variables to attend too…. From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Gonzalo Gasca Meza Sent: Sunday, October 2, 2016 10:43 PM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org <mailto:freeswitch-users@lists.freeswitch.org> > Subject: [Freeswitch-users] Performance testing Hi all, I'm testing Freeswitch using ESL interface, recording and SIP UDP Calls at a rate of around 5 CPS. Freeswitch receives ESL requests and proceeds to send call to PSTN Emulator, once call is answered I start recording. I have noticed around 40% CPU utilization when recording is not enabled and ~60%+ CPU when recording is. I created/mount a volume specifically to store the recording files. (ext4) Server specifications: EC2 server (t2.medium) 2 vCPUs, 2.5 GHz, Intel Xeon Family, 4 GiB memory. FreeSWITCH (Version 1.6.9 -16-d574870 64bit) I followed instructions at: https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations [Questions] a) How to strip down Freeswitch and remove non-used modules? Features I need: esl, recording, loopback, bridge, sip, spandsp, cdr_csv. My modules: fs_cli -x "show modules" http://pastebin.com/BQSxzFDE b) I would like to see which module is using X/Y amount of CPU cycles/memory in Freeswitch is it possible?. fs_cli -x 'show status' UP 0 years, 0 days, 1 hour, 2 minutes, 15 seconds, 541 milliseconds, 286 microseconds FreeSWITCH (Version 1.6.9 -16-d574870 64bit) is ready 18192 session(s) since startup 90 session(s) - peak 93, last 5min 92 5 session(s) per Sec out of max 30, peak 7, last 5min 6 1000 session(s) max min idle cpu 0.00/75.57 Current Stack Size/Max 240K/8192K A sample image is below: 40% steady (no recording) a peak when I enabled recording. # top top - 03:25:16 up 22:42, 1 user, load average: 3.88, 5.74, 10.39 Tasks: 68 total, 1 running, 67 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.0 us, 2.3 sy, 0.0 ni, 85.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.4 st KiB Mem: 4057584 total, 865964 used, 3191620 free, 147088 buffers KiB Swap: 1048572 total, 0 used, 1048572 free. 213572 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26217 freeswi+ -2 -10 1427444 302584 16904 S 65.9 7.5 28:47.49 freeswitch _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks Ken, Just upgraded to a more powerful machine (AWS m4.xlarge) 4 vCPU and 16 GB RAM and will be upgrading to 16 by the end of this week (m4.4xlarge 16 vCPU 64 GB RAM) Thanks On Sun, Oct 2, 2016 at 9:26 PM, Ken Rice <krice@freeswitch.org> wrote:
well consider you are doing 90 transcode sessions there…. Not sure what the actual outbound codec is, but that’s just a guess based on 90 session(s) - peak 93, last 5min 92…
theres also what else is going on with the call that you have to contend with… obviously you are creating some sort of dialer… and for help with that and optimization I would contact the FS core team @ consulting@freeswitch.org for some pro help… the reason we don’t release any soft of scaling metrics is changing 1 setting can invalidate the testing… things like recording, transcoding, amount of media handling, any call scripting and specifically what else is going on there in the media processing…
as far as capacity planning for doing something like you are doing, I wouldn’t even bother with less than a 12 or 24 vcore machine
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Gonzalo Gasca Meza *Sent:* Sunday, October 2, 2016 11:19 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] Performance testing
Hi Ken,
I'm using .wav, this is my ESL command:
originate {sip_h_X-UUID=9d33a760-gmwz-qn1n-xvqz-qal4-9d33b25a89,sip_ h_X-Campaign=3A252BEX9W,ignore_early_media=false,bridge_early_media=false, originate_timeout=70,bridge_answer_timeout=70,call_ timeout=70,[leg_timeout=70],execute_on_answer_1='sched_hangup 18 ALLOTTED_TIMEOUT',origination_uuid=9d33a760-gmwz-qn1n-xvqz- qal4-9d33b25a89,origination_caller_id_number=+14081116805, RECORD_STEREO=true,execute_on_answer_2='record_session /var/www/html/recordings/9d33a760-gmwz-qn1n-xvqz-qal4- 9d33b25a89.wav'}sofia/gateway/kamailio/+14080001963 handle_calls
I created a separate volume for writing only recording files & logs.
I will consider increasing CPUs, do you think I still can remove some of the modules I currently using to improve performance?
Thanks!
On Sun, Oct 2, 2016 at 9:07 PM, Ken Rice <krice@freeswitch.org> wrote:
How are you actually recording the files? Native? mp3? wav? Something else?
keep in mind that recording also means disk io which means more CPU while its recording to handle the IO load… there are way too many undisclosed factors here that can and will impact your performance…
one example is 2 cores? That’s all? FS is a highly threaded application… that means for each call leg there is atleast 1 thread… so the more cores the merrier…
not to mention you are running on amazon that leaves a whole different set of variables to attend too….
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Gonzalo Gasca Meza *Sent:* Sunday, October 2, 2016 10:43 PM *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* [Freeswitch-users] Performance testing
Hi all,
I'm testing Freeswitch using ESL interface, recording and SIP UDP Calls at a rate of around 5 CPS.
Freeswitch receives ESL requests and proceeds to send call to PSTN Emulator, once call is answered I start recording. I have noticed around 40% CPU utilization when recording is not enabled and ~60%+ CPU when recording is.
I created/mount a volume specifically to store the recording files. (ext4)
*Server specifications:*
EC2 server (t2.medium) 2 vCPUs, 2.5 GHz, Intel Xeon Family, 4 GiB memory.
FreeSWITCH (Version 1.6.9 -16-d574870 64bit)
I followed instructions at: https://wiki.freeswitch. org/wiki/Performance_testing_and_configurations
*[Questions]*
a) How to strip down Freeswitch and remove non-used modules?
Features I need:
*esl, recording, loopback, bridge, sip, spandsp, cdr_csv.*
My modules:
*fs_cli -x "show modules"*
b) I would like to see which module is using X/Y amount of CPU cycles/memory in Freeswitch is it possible?.
*fs_cli -x 'show status'*
UP 0 years, 0 days, 1 hour, 2 minutes, 15 seconds, 541 milliseconds, 286 microseconds
FreeSWITCH (Version 1.6.9 -16-d574870 64bit) is ready
18192 session(s) since startup
90 session(s) - peak 93, last 5min 92
5 session(s) per Sec out of max 30, peak 7, last 5min 6
1000 session(s) max
min idle cpu 0.00/75.57
Current Stack Size/Max 240K/8192K
A sample image is below: 40% steady (no recording) a peak when I enabled recording.
[image: Inline image 1]
# top
top - 03:25:16 up 22:42, 1 user, load average: 3.88, 5.74, 10.39
Tasks:* 68 *total,* 1 *running,* 67 *sleeping,* 0 *stopped,* 0 * zombie
%Cpu(s):* 11.0 *us,* 2.3 *sy,* 0.0 *ni,* 85.7 *id,* 0.0 *wa,* 0.0 *hi,* 0.7 *si,* 0.4 *st
KiB Mem: * 4057584 *total,* 865964 *used,* 3191620 *free,* 147088 * buffers
KiB Swap:* 1048572 *total,* 0 *used,* 1048572 *free.* 213572 *cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26217 freeswi+ -2 -10 1427444 302584 16904 S *65.9* 7.5 28:47.49 freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (2)
-
Gonzalo Gasca Meza
-
Ken Rice