FreeSWITCH under the Linux 2.6.29 kernel
After upgrading to the 2.6.29 kernel (the Debian packaged version), FreeSWITCH takes up more CPU time than usual, e.g., 7% as reported by top, and the load average is high (e.g., 0.87) even when the machine is idle and there are no calls in progress. When top is run, FreeSWITCH appears at the start of the list. Is anybody else seeing this? The proportion of CPU time devoted to system calls seems higher than it should be. I would be interested in reports from anyone else who is running FreeSWITCH under Linux 2.6.29.1 or equivalent from a distribution.
boot Your kernel with "divider=10 nohz=off" options :) Recent kernels are tickless which basically causes all freeswitch timers/sleeps to fire at requested microsecond intervals. With nohz kernels You get hundred times more system calls with freeswitch :( On 2009-04-27, at 03:00, Jason White wrote:
After upgrading to the 2.6.29 kernel (the Debian packaged version), FreeSWITCH takes up more CPU time than usual, e.g., 7% as reported by top, and the load average is high (e.g., 0.87) even when the machine is idle and there are no calls in progress. When top is run, FreeSWITCH appears at the start of the list.
Is anybody else seeing this? The proportion of CPU time devoted to system calls seems higher than it should be.
I would be interested in reports from anyone else who is running FreeSWITCH under Linux 2.6.29.1 or equivalent from a distribution.
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Paweł Pierścionek <pawel@voiceworks.pl> wrote:
boot Your kernel with "divider=10 nohz=off" options :)
Recent kernels are tickless which basically causes all freeswitch timers/sleeps to fire at requested microsecond intervals. With nohz kernels You get hundred times more system calls with freeswitch :(
Is there anything FreeSWITCH could do about this, i.e., a configuration parameter to adjust the timers appropriately? I'm speaking from ignorance here; it just occurs to me that the application should be able to control this behaviour without kernel options having to be changed, but perhaps this is not the case.
Just to add data to this: PowerTOP 1.11 (C) 2007, 2008 Intel Corporation Collecting data for 15 seconds < Detailed C-state information is not available.> P-states (frequencies) 2.34 Ghz 0.0% 2.00 Ghz 100.0% Wakeups-from-idle per second : 405.4 interval: 15.0s no ACPI power usage estimate available Top causes for wakeups: 82.2% (1067.3) freeswitch : schedule_hrtimeout_range (hrtimer_wakeup) [snip]
Paweł Pierścionek <pawel@voiceworks.pl> writes: G'day Paweł.
boot Your kernel with "divider=10 nohz=off" options :)
Recent kernels are tickless which basically causes all freeswitch timers/sleeps to fire at requested microsecond intervals. With nohz kernels You get hundred times more system calls with freeswitch :(
Like Jason, I am also interested to know why the tickless kernel causes the timers to generate so much more load. I can't find anything documented anywhere, really, about the issue — and this thread is the only thing Google turns up on the topic. I am looking to move my SIP system to FreeSwitch some time soon, if I can, but I would love to know why nohz is so hostile to FreeSwitch before I do, if possible. Regards, Daniel
Hi, With really old kernels (100Hz) if You do sleep(1ms) You sleep for 10ms on average. With enterprise kernels (250Hz) Your sleep resolution increases by a factor of 4. With fresh kernels (1000Hz) You get real 1ms timer resolution - 10fold increase compared to old kernels. With tickless You get whatever resolution You want - eg when You sleep for 100 microseconds(micro not mili) then You get exactly what You wish for. Now for reasons I do no try to understand :) there are a lot of really short sleeps and fast timers in FreeSwitch - like 100 micro(1/10th of a ms). So with CentOS such a 100 microsecond sleep cannot "fire" faster then 250 times a second. With tickless kernel same 100 microsecond sleep "fires" 10k times a second. Pawel,
Then it would be recommended to not do tickless clock :P /b On Apr 30, 2009, at 1:28 PM, Paweł Pierścionek wrote:
Hi,
With really old kernels (100Hz) if You do sleep(1ms) You sleep for 10ms on average. With enterprise kernels (250Hz) Your sleep resolution increases by a factor of 4. With fresh kernels (1000Hz) You get real 1ms timer resolution - 10fold increase compared to old kernels.
With tickless You get whatever resolution You want - eg when You sleep for 100 microseconds(micro not mili) then You get exactly what You wish for.
Now for reasons I do no try to understand :) there are a lot of really short sleeps and fast timers in FreeSwitch - like 100 micro(1/10th of a ms). So with CentOS such a 100 microsecond sleep cannot "fire" faster then 250 times a second. With tickless kernel same 100 microsecond sleep "fires" 10k times a second.
Pawel,
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Brian West brian@freeswitch.org -- Meet us at ClueCon! http://www.cluecon.com
Can you point out any place we do sub milli second sleeps? The timer thread should be doing 1ms, I can't think of any that would be less. MIke On Apr 30, 2009, at 2:28 PM, Paweł Pierścionek wrote:
Hi,
With really old kernels (100Hz) if You do sleep(1ms) You sleep for 10ms on average. With enterprise kernels (250Hz) Your sleep resolution increases by a factor of 4. With fresh kernels (1000Hz) You get real 1ms timer resolution - 10fold increase compared to old kernels.
With tickless You get whatever resolution You want - eg when You sleep for 100 microseconds(micro not mili) then You get exactly what You wish for.
Now for reasons I do no try to understand :) there are a lot of really short sleeps and fast timers in FreeSwitch - like 100 micro(1/10th of a ms). So with CentOS such a 100 microsecond sleep cannot "fire" faster then 250 times a second. With tickless kernel same 100 microsecond sleep "fires" 10k times a second.
Pawel,
Jason: IMHO, it is related with the following kernel options: CONFIG_HZ, CONFIG_NO_HZ and CONFIG_HIGH_RES_TIMERS. Take a look at those options in your kernel and try modifying them until get the desired result. Google that options and you will find lots of discussions that will clarify your mind. Here you will find a simple explanation: http://www.smk.co.za/2007/07/21/a-tickless-kernel/ My configuration is 100 Hz + tickless + High-Res timers and everything goes perfect. When system is idle and FS has no calls, I see my CPU at 0%. Hope it helps. Pablo On Thu, Apr 30, 2009 at 3:48 PM, Michael Jerris <mike@jerris.com> wrote:
Can you point out any place we do sub milli second sleeps? The timer thread should be doing 1ms, I can't think of any that would be less.
MIke
On Apr 30, 2009, at 2:28 PM, Paweł Pierścionek wrote:
Hi,
With really old kernels (100Hz) if You do sleep(1ms) You sleep for 10ms on average. With enterprise kernels (250Hz) Your sleep resolution increases by a factor of 4. With fresh kernels (1000Hz) You get real 1ms timer resolution - 10fold increase compared to old kernels.
With tickless You get whatever resolution You want - eg when You sleep for 100 microseconds(micro not mili) then You get exactly what You wish for.
Now for reasons I do no try to understand :) there are a lot of really short sleeps and fast timers in FreeSwitch - like 100 micro(1/10th of a ms). So with CentOS such a 100 microsecond sleep cannot "fire" faster then 250 times a second. With tickless kernel same 100 microsecond sleep "fires" 10k times a second.
Pawel,
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Pablo Hernan Saro <pablosaro@gmail.com> wrote:
IMHO, it is related with the following kernel options: CONFIG_HZ, CONFIG_NO_HZ and CONFIG_HIGH_RES_TIMERS. Take a look at those options in your kernel and try modifying them until get the desired result. Google that options and you will find lots of discussions that will clarify your mind. Here you will find a simple explanation: http://www.smk.co.za/2007/07/21/a-tickless-kernel/
Thank you for the references. I think I'll modify my kernel parameters in the grub configuration, since I am using Debian kernels at the moment. (I do know how to compile my own, which I will gladly do if it becomes necessary).
Folks; Bear in mind that the frequency is (X)Hz * (num cores), hence saying 100Hz on a dual core winds up being 200Hz. My setup is 250Hz on a Dual-Core and the quality is perfect. Oh, btw folks, don't attempt to do anything involving QOS (be it TBF, CBQ, HTB, or whatnot) on anything less than kernel 2.6.28.4 I don't know why exactly that is, but extensive testing here in the lab showed that this was entirely FUBAR until 2.6.25.7 where it got better, but not perfect until 2.6.28.4 (I may not be exact on the revisions, but close enough...) There's also some options in the kernel that you must disable (not compile in) if you expect packet shaping to work. If there's interest in this, e-mail me directly and I'll see if I can toss it into the FS Wiki some time this upcoming weekend. Best Regards, Karl J. Vesterling kjv@ken-ton.com 202-461-3231 x0 On May 9, 2009, at 2:14 AM, Jason White wrote:
Pablo Hernan Saro <pablosaro@gmail.com> wrote:
IMHO, it is related with the following kernel options: CONFIG_HZ, CONFIG_NO_HZ and CONFIG_HIGH_RES_TIMERS. Take a look at those options in your kernel and try modifying them until get the desired result. Google that options and you will find lots of discussions that will clarify your mind. Here you will find a simple explanation: http://www.smk.co.za/2007/07/21/a-tickless-kernel/
Thank you for the references. I think I'll modify my kernel parameters in the grub configuration, since I am using Debian kernels at the moment. (I do know how to compile my own, which I will gladly do if it becomes necessary).
_______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Brian West
-
Daniel Pittman
-
Jason White
-
Karl Vesterling
-
Michael Jerris
-
Pablo Hernan Saro
-
Paweł Pierścionek