High syscall percentage on Centos 7
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
Could you build and test FS 1.2.23 on CentOS 7. Probably it can help to locate problem on OS or FS. Sergey On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Without media, FS seems to spend around 30-40% of its CPU time in malloc/free[1]. While I wouldn’t think this would directly create so many syscalls, it might be something to look at. Otherwise, start by seeing what syscalls are actually being made and go from there. -Michael From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Sergey Safarov Sent: Friday, 27 November, 2015 6:44 To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] High syscall percentage on Centos 7 Could you build and test FS 1.2.23 on CentOS 7. Probably it can help to locate problem on OS or FS. Sergey On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com<mailto:dvmorte@gmail.com>> wrote: Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11? _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The syscalls are the read and write from the RTP stack probably. We don't give official help on load related queries because its too subjective to the environment. 300cps for a hint though, is 6 times the industry standard that even a commercial product will claim to do in terms of CPS. On Mon, Nov 30, 2015 at 3:51 AM, Michael Giagnocavo <mgg@giagnocavo.net> wrote:
Without media, FS seems to spend around 30-40% of its CPU time in malloc/free[1]. While I wouldn’t think this would directly create so many syscalls, it might be something to look at. Otherwise, start by seeing what syscalls are actually being made and go from there.
-Michael
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Sergey Safarov *Sent:* Friday, 27 November, 2015 6:44 *To:* FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject:* Re: [Freeswitch-users] High syscall percentage on Centos 7
Could you build and test FS 1.2.23 on CentOS 7.
Probably it can help to locate problem on OS or FS.
Sergey
On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch?
We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
We made couple of synthetic tests with sipp uac scenario (unfortunately we cannot move the current load from the production to the test environment right now) - 300 cps, 1500 calls limi;, ulimits for FS set as recommended, same dialplan : Debian 8.2 + FS 1.2.24 - up to 95% CPU utilization with ~65% syscalls Debian 8.2 + FS 1.6.5 - up to 95% CPU utilization with ~65% syscalls Centos 5.11 + FS 1.2.23 - up to 85% CPU utilization with ~20% syscalls Centos 6.7 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.2.24 - up to97% CPU utilization with ~67% syscalls FreeBSD 10.2 + FS 1.6.5 - up to 80% CPU utilization with ~20% syscalls So, it seems that we've something missed when configuring FS on modern Linux distributives. On Fri, Nov 27, 2015 at 2:44 PM, Sergey Safarov <s.safarov@gmail.com> wrote:
Could you build and test FS 1.2.23 on CentOS 7. Probably it can help to locate problem on OS or FS.
Sergey
On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Could you document your testing methodology so others can replicate it, and are you testing this all on the exact same hardware? On Fri, Dec 4, 2015 at 10:20 AM, Denys D <dvmorte@gmail.com> wrote:
We made couple of synthetic tests with sipp uac scenario (unfortunately we cannot move the current load from the production to the test environment right now) - 300 cps, 1500 calls limi;, ulimits for FS set as recommended, same dialplan : Debian 8.2 + FS 1.2.24 - up to 95% CPU utilization with ~65% syscalls Debian 8.2 + FS 1.6.5 - up to 95% CPU utilization with ~65% syscalls Centos 5.11 + FS 1.2.23 - up to 85% CPU utilization with ~20% syscalls Centos 6.7 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.2.24 - up to97% CPU utilization with ~67% syscalls FreeBSD 10.2 + FS 1.6.5 - up to 80% CPU utilization with ~20% syscalls So, it seems that we've something missed when configuring FS on modern Linux distributives.
On Fri, Nov 27, 2015 at 2:44 PM, Sergey Safarov <s.safarov@gmail.com> wrote:
Could you build and test FS 1.2.23 on CentOS 7. Probably it can help to locate problem on OS or FS.
Sergey
On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
What are the hardware specs for your test? and which specific kernel were you testing against? Was the testing instance using bare metal, or virtualized? William King Senior Engineer Quentus Technologies, INC 1037 NE 65th St Suite 273 Seattle, WA 98115 Main: (877) 211-9337 Office: (206) 388-4772 Cell: (253) 686-5518 william.king@quentustech.com On 12/4/15 8:20 AM, Denys D wrote:
We made couple of synthetic tests with sipp uac scenario (unfortunately we cannot move the current load from the production to the test environment right now) - 300 cps, 1500 calls limi;, ulimits for FS set as recommended, same dialplan : Debian 8.2 + FS 1.2.24 - up to 95% CPU utilization with ~65% syscalls Debian 8.2 + FS 1.6.5 - up to 95% CPU utilization with ~65% syscalls Centos 5.11 + FS 1.2.23 - up to 85% CPU utilization with ~20% syscalls Centos 6.7 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.6.5 - up to 97% CPU utilization with ~67% syscalls Centos 7.1 + FS 1.2.24 - up to97% CPU utilization with ~67% syscalls FreeBSD 10.2 + FS 1.6.5 - up to 80% CPU utilization with ~20% syscalls So, it seems that we've something missed when configuring FS on modern Linux distributives.
On Fri, Nov 27, 2015 at 2:44 PM, Sergey Safarov <s.safarov@gmail.com <mailto:s.safarov@gmail.com>> wrote:
Could you build and test FS 1.2.23 on CentOS 7. Probably it can help to locate problem on OS or FS.
Sergey
On Thu, Nov 26, 2015 at 11:21 AM, Denys D <dvmorte@gmail.com <mailto:dvmorte@gmail.com>> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I really like how you characterize 300CPS as moderate, your test is somewhat unclear, are you doing 300CPS with a duration of 3 seconds or are you just ramping up to to 1500 calls total at 300CPS. ./b On Thu, Nov 26, 2015 at 2:21 AM, Denys D <dvmorte@gmail.com> wrote:
Hi! Has anybody faced with high system calls on Centos 7 while running FreeSwitch? We have migrated from FS 1.2.23 on Centos 5.11 to FS 1.6.2 on Centos 7 and faced with very high syscall (up to 60% CPU time) rate under moderate load (300 CPS, 1500 simultaneous calls). We also faced the same issue on Centos 6.7 and Debian 8.2 with FS 1.6.2 on the same hardware. On Centos 5.11 with the same load and h/w we had no more than 10% syscalls CPU time. We have already tried nohz=off kernel option and it didn't solve the problem. What should we change in our environment to achieve the same performance level as it was on Centos 5.11?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit: /r/freeswitch <https://www.reddit.com/r/freeswitch> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
participants (7)
-
Anthony Minessale
-
Brian West
-
Denys D
-
Giovanni Maruzzelli
-
Michael Giagnocavo
-
Sergey Safarov
-
William King