[Freeswitch-dev] On high CPU usage and NONBLOCK-ed sockets
Hi, I have made some investigations on high CPU usage (Methodology below). In the most CPU-consuming threads, there is: * Several threads related to queues handling (Can't we use interrupt here?) * The time thread * Several threads related to opened sockets. i think there can be some improvement here: Example: Thread 3 (Thread 0xb626fb70 (LWP 5922)): #0 0xb7693f6a in clock_nanosleep (clock_id=-1215964116, flags=0, req=0xb626d928, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:49 #1 0xb7764faa in do_sleep (t=882) at src/switch_time.c:165 #2 0xb7765971 in switch_cond_next () at src/switch_time.c:428 #3 0xb6bb553d in read_packet (listener=0x966c418, event=0xb626ecd4, timeout=0) at /home/mathieu/apps/freeswitch/freeswitch.git/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:1316 #4 0xb6bb95a6 in listener_run (thread=0x965b0f0, obj=0x966c418) at /home/mathieu/apps/freeswitch/freeswitch.git/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:2468 #5 0xb77933f5 in dummy_worker (opaque=0x965b0f0) at threadproc/unix/thread.c:138 #6 0xb7645955 in start_thread (arg=0xb626fb70) at pthread_create.c:300 #7 0xb73e910e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Modules concerned are mod_event_socket, mod_sofia, mod_skinny and probably all endpoints modules. Currently, FS try to read from the socket and if there is no data in the buffer, some checks are made then the thread sleeps for some time (by using do_sleep() or switch_cond_next()). Most of the time this is not needed to do those extra checks that cost CPU. I propose to move to BLOCK-ed sockets with timeout. Also, ajusting the requested length will reduce execution time. Most of the job will be done on the kernel side which is more efficient. Maybe I'm missing some corner-cases, but for mod_skinny, the move improves greatly the performance. This protocol was easy to handle because we first should to read the header then we read the remaining data whose length is in the header. Mathieu Parent Methodology: - Started FS - Connecting various kind of clients (event_socket, SIP, Skinny) - getting list of high CPU thread: ps -eLf | egrep '(freeswitch|CMD)' | sort -n -k 5 - debugging freeswitch with gdb to show backtraces on all thread: "thread apply all bt" - analyse
in my findings not more efficient, not at all reliable. What do you mean by high cpu exactly? what are you using to measure it? what OS are you on? On Fri, Sep 3, 2010 at 12:19 PM, Mathieu Parent <math.parent@gmail.com> wrote:
Hi,
I have made some investigations on high CPU usage (Methodology below). In the most CPU-consuming threads, there is: * Several threads related to queues handling (Can't we use interrupt here?) * The time thread * Several threads related to opened sockets. i think there can be some improvement here:
Example: Thread 3 (Thread 0xb626fb70 (LWP 5922)): #0 0xb7693f6a in clock_nanosleep (clock_id=-1215964116, flags=0, req=0xb626d928, rem=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:49 #1 0xb7764faa in do_sleep (t=882) at src/switch_time.c:165 #2 0xb7765971 in switch_cond_next () at src/switch_time.c:428 #3 0xb6bb553d in read_packet (listener=0x966c418, event=0xb626ecd4, timeout=0) at /home/mathieu/apps/freeswitch/freeswitch.git/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:1316 #4 0xb6bb95a6 in listener_run (thread=0x965b0f0, obj=0x966c418) at /home/mathieu/apps/freeswitch/freeswitch.git/src/mod/event_handlers/mod_event_socket/mod_event_socket.c:2468 #5 0xb77933f5 in dummy_worker (opaque=0x965b0f0) at threadproc/unix/thread.c:138 #6 0xb7645955 in start_thread (arg=0xb626fb70) at pthread_create.c:300 #7 0xb73e910e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Modules concerned are mod_event_socket, mod_sofia, mod_skinny and probably all endpoints modules.
Currently, FS try to read from the socket and if there is no data in the buffer, some checks are made then the thread sleeps for some time (by using do_sleep() or switch_cond_next()). Most of the time this is not needed to do those extra checks that cost CPU.
I propose to move to BLOCK-ed sockets with timeout. Also, ajusting the requested length will reduce execution time. Most of the job will be done on the kernel side which is more efficient.
Maybe I'm missing some corner-cases, but for mod_skinny, the move improves greatly the performance. This protocol was easy to handle because we first should to read the header then we read the remaining data whose length is in the header.
Mathieu Parent
Methodology: - Started FS - Connecting various kind of clients (event_socket, SIP, Skinny) - getting list of high CPU thread: ps -eLf | egrep '(freeswitch|CMD)' | sort -n -k 5 - debugging freeswitch with gdb to show backtraces on all thread: "thread apply all bt" - analyse
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
On Fri, Sep 3, 2010 at 8:22 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
What do you mean by high cpu exactly? when going blocking-mode, the user CPU gets lower and the kernel CPU is similar.
what are you using to measure it? top, vmstat
without mod_skinny mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 48 175552 161664 787636 0 0 8 10 184 361 6 8 86 1 1 0 48 171948 161672 791372 0 0 0 5 4141 9191 14 12 74 0 0 0 48 171700 161684 791496 0 0 0 11 4125 9081 13 9 78 0 0 0 48 175048 161688 787860 0 0 0 3 4344 9177 7 10 84 0 6 0 48 175048 161692 787860 0 0 0 1 4225 9035 9 11 80 0 with mod_skinny NONBLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 48 172668 162104 789052 0 0 8 10 190 374 6 8 86 1 1 0 48 173528 162108 788652 0 0 0 2 4792 10346 15 13 72 0 1 0 48 173528 162124 788664 0 0 0 17 4678 10326 9 12 78 0 0 0 48 173528 162124 788628 0 0 0 1 4788 10343 10 14 77 0 9 0 48 166956 162136 792764 0 0 0 10 4592 10535 16 13 71 0 with mod_skinny BLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 48 175288 161648 787832 0 0 8 10 182 358 6 8 86 1 1 0 48 175280 161648 787824 0 0 0 0 4236 9054 7 9 83 0 7 0 48 175404 161648 787804 0 0 0 3 4173 8927 7 10 83 0 0 0 48 175280 161652 787576 0 0 0 5 4288 9182 9 10 81 0 6 0 48 175404 161656 787580 0 0 0 4 4209 9001 8 10 82 0 So it seems that both %user and %system are lower. I have not done complete tests but mod_skinny is more responsive in BLOCKING-mode
what OS are you on? Linux netthieu 2.6.32-5-686 #1 SMP Thu Aug 12 13:38:27 UTC 2010 i686 GNU/Linux
this is Debian testing using FS built from recent git using debian packaging.
in my findings not more efficient, not at all reliable. Googling some, I found (from http://www.ibm.com/developerworks/linux/library/l-async/)
"Synchronous non-blocking I/O: A less efficient variant of synchronous blocking". This is what we are doing? The good solutions IMO are: - Synchronous blocking I/O: simple and efficient - Asynchronous blocking I/O: not so good. use of "select" is not recommended - Asynchronous non-blocking I/O (AIO) which is a completely different API (portable?) which should fit well with FS's event model but with a lot of work Maybe we can make it configurable at build-time (as it is in mod_skinny) to allow performance testing? Perhaps the results will be very different from one platform to another. Mathieu
Can you produce some example code that we can test on other architectures? What exactly were you doing in mod_skinny? if you copied mod_event_socket it may be true your blocking approach is better but mod_event_socket has some high performance reasons to not block. We use the centos kernel and trying to make it work better on debian with a newer kernel could disrupt our stable user base. consider making a mod_debug or mod_testperf we can mess with on many boxes. On Fri, Sep 3, 2010 at 3:52 PM, Mathieu Parent <math.parent@gmail.com> wrote:
On Fri, Sep 3, 2010 at 8:22 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
What do you mean by high cpu exactly? when going blocking-mode, the user CPU gets lower and the kernel CPU is similar.
what are you using to measure it? top, vmstat
without mod_skinny mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 48 175552 161664 787636 0 0 8 10 184 361 6 8 86 1 1 0 48 171948 161672 791372 0 0 0 5 4141 9191 14 12 74 0 0 0 48 171700 161684 791496 0 0 0 11 4125 9081 13 9 78 0 0 0 48 175048 161688 787860 0 0 0 3 4344 9177 7 10 84 0 6 0 48 175048 161692 787860 0 0 0 1 4225 9035 9 11 80 0
with mod_skinny NONBLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 48 172668 162104 789052 0 0 8 10 190 374 6 8 86 1 1 0 48 173528 162108 788652 0 0 0 2 4792 10346 15 13 72 0 1 0 48 173528 162124 788664 0 0 0 17 4678 10326 9 12 78 0 0 0 48 173528 162124 788628 0 0 0 1 4788 10343 10 14 77 0 9 0 48 166956 162136 792764 0 0 0 10 4592 10535 16 13 71 0
with mod_skinny BLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 48 175288 161648 787832 0 0 8 10 182 358 6 8 86 1 1 0 48 175280 161648 787824 0 0 0 0 4236 9054 7 9 83 0 7 0 48 175404 161648 787804 0 0 0 3 4173 8927 7 10 83 0 0 0 48 175280 161652 787576 0 0 0 5 4288 9182 9 10 81 0 6 0 48 175404 161656 787580 0 0 0 4 4209 9001 8 10 82 0
So it seems that both %user and %system are lower. I have not done complete tests but mod_skinny is more responsive in BLOCKING-mode
what OS are you on? Linux netthieu 2.6.32-5-686 #1 SMP Thu Aug 12 13:38:27 UTC 2010 i686 GNU/Linux
this is Debian testing using FS built from recent git using debian packaging.
in my findings not more efficient, not at all reliable. Googling some, I found (from http://www.ibm.com/developerworks/linux/library/l-async/)
"Synchronous non-blocking I/O: A less efficient variant of synchronous blocking". This is what we are doing?
The good solutions IMO are: - Synchronous blocking I/O: simple and efficient - Asynchronous blocking I/O: not so good. use of "select" is not recommended - Asynchronous non-blocking I/O (AIO) which is a completely different API (portable?) which should fit well with FS's event model but with a lot of work
Maybe we can make it configurable at build-time (as it is in mod_skinny) to allow performance testing? Perhaps the results will be very different from one platform to another.
Mathieu
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Another idea to conserve CPU and sys-time usage - to use adaptive timeouts in the timewait intervals. General idea: requests to our system aren't random, but correlated. For example, night time there are too few requests, and so on. Simple code for demonstrate algorithm following. There is min_interval is ~4ms, max_interval is ~524ms: uint32_t interval = 1 << 25; // ~4ms while(usleep(interval >> 13)) { // Max 2^19us, or 524ms // wake up here if(need_to_do_something) { do_something(); interval = 1 << 25; // Reset interval } else { // Nothing to do -- *2 interval interval |= interval << 1; } } Anthony Minessale wrote:
Can you produce some example code that we can test on other architectures? What exactly were you doing in mod_skinny? if you copied mod_event_socket it may be true your blocking approach is better but mod_event_socket has some high performance reasons to not block.
We use the centos kernel and trying to make it work better on debian with a newer kernel could disrupt our stable user base.
consider making a mod_debug or mod_testperf we can mess with on many boxes.
On Fri, Sep 3, 2010 at 3:52 PM, Mathieu Parent<math.parent@gmail.com> wrote:
On Fri, Sep 3, 2010 at 8:22 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
What do you mean by high cpu exactly?
when going blocking-mode, the user CPU gets lower and the kernel CPU is similar.
what are you using to measure it?
top, vmstat
without mod_skinny mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 48 175552 161664 787636 0 0 8 10 184 361 6 8 86 1 1 0 48 171948 161672 791372 0 0 0 5 4141 9191 14 12 74 0 0 0 48 171700 161684 791496 0 0 0 11 4125 9081 13 9 78 0 0 0 48 175048 161688 787860 0 0 0 3 4344 9177 7 10 84 0 6 0 48 175048 161692 787860 0 0 0 1 4225 9035 9 11 80 0
with mod_skinny NONBLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 48 172668 162104 789052 0 0 8 10 190 374 6 8 86 1 1 0 48 173528 162108 788652 0 0 0 2 4792 10346 15 13 72 0 1 0 48 173528 162124 788664 0 0 0 17 4678 10326 9 12 78 0 0 0 48 173528 162124 788628 0 0 0 1 4788 10343 10 14 77 0 9 0 48 166956 162136 792764 0 0 0 10 4592 10535 16 13 71 0
with mod_skinny BLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 48 175288 161648 787832 0 0 8 10 182 358 6 8 86 1 1 0 48 175280 161648 787824 0 0 0 0 4236 9054 7 9 83 0 7 0 48 175404 161648 787804 0 0 0 3 4173 8927 7 10 83 0 0 0 48 175280 161652 787576 0 0 0 5 4288 9182 9 10 81 0 6 0 48 175404 161656 787580 0 0 0 4 4209 9001 8 10 82 0
So it seems that both %user and %system are lower. I have not done complete tests but mod_skinny is more responsive in BLOCKING-mode
what OS are you on?
Linux netthieu 2.6.32-5-686 #1 SMP Thu Aug 12 13:38:27 UTC 2010 i686 GNU/Linux
this is Debian testing using FS built from recent git using debian packaging.
in my findings not more efficient, not at all reliable.
Googling some, I found (from http://www.ibm.com/developerworks/linux/library/l-async/)
"Synchronous non-blocking I/O: A less efficient variant of synchronous blocking". This is what we are doing?
The good solutions IMO are: - Synchronous blocking I/O: simple and efficient - Asynchronous blocking I/O: not so good. use of "select" is not recommended - Asynchronous non-blocking I/O (AIO) which is a completely different API (portable?) which should fit well with FS's event model but with a lot of work
Maybe we can make it configurable at build-time (as it is in mod_skinny) to allow performance testing? Perhaps the results will be very different from one platform to another.
Mathieu
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
Careful you don't spend more time adjusting the interval than you save. ;) Steve on iPhone On 6 Sep 2010, at 18:05, Oleg Khovayko <khovayko@gmail.com> wrote:
Another idea to conserve CPU and sys-time usage - to use adaptive timeouts in the timewait intervals. General idea: requests to our system aren't random, but correlated. For example, night time there are too few requests, and so on.
Simple code for demonstrate algorithm following. There is min_interval is ~4ms, max_interval is ~524ms:
uint32_t interval = 1 << 25; // ~4ms
while(usleep(interval >> 13)) { // Max 2^19us, or 524ms // wake up here if(need_to_do_something) { do_something(); interval = 1 << 25; // Reset interval } else { // Nothing to do -- *2 interval interval |= interval << 1; } }
Anthony Minessale wrote:
Can you produce some example code that we can test on other architectures? What exactly were you doing in mod_skinny? if you copied mod_event_socket it may be true your blocking approach is better but mod_event_socket has some high performance reasons to not block.
We use the centos kernel and trying to make it work better on debian with a newer kernel could disrupt our stable user base.
consider making a mod_debug or mod_testperf we can mess with on many boxes.
On Fri, Sep 3, 2010 at 3:52 PM, Mathieu Parent<math.parent@gmail.com> wrote:
On Fri, Sep 3, 2010 at 8:22 PM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
What do you mean by high cpu exactly?
when going blocking-mode, the user CPU gets lower and the kernel CPU is similar.
what are you using to measure it?
top, vmstat
without mod_skinny mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 7 0 48 175552 161664 787636 0 0 8 10 184 361 6 8 86 1 1 0 48 171948 161672 791372 0 0 0 5 4141 9191 14 12 74 0 0 0 48 171700 161684 791496 0 0 0 11 4125 9081 13 9 78 0 0 0 48 175048 161688 787860 0 0 0 3 4344 9177 7 10 84 0 6 0 48 175048 161692 787860 0 0 0 1 4225 9035 9 11 80 0
with mod_skinny NONBLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 48 172668 162104 789052 0 0 8 10 190 374 6 8 86 1 1 0 48 173528 162108 788652 0 0 0 2 4792 10346 15 13 72 0 1 0 48 173528 162124 788664 0 0 0 17 4678 10326 9 12 78 0 0 0 48 173528 162124 788628 0 0 0 1 4788 10343 10 14 77 0 9 0 48 166956 162136 792764 0 0 0 10 4592 10535 16 13 71 0
with mod_skinny BLOCKED mathieu@netthieu:~/apps/freeswitch/freeswitch.git$ vmstat 5 5 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 48 175288 161648 787832 0 0 8 10 182 358 6 8 86 1 1 0 48 175280 161648 787824 0 0 0 0 4236 9054 7 9 83 0 7 0 48 175404 161648 787804 0 0 0 3 4173 8927 7 10 83 0 0 0 48 175280 161652 787576 0 0 0 5 4288 9182 9 10 81 0 6 0 48 175404 161656 787580 0 0 0 4 4209 9001 8 10 82 0
So it seems that both %user and %system are lower. I have not done complete tests but mod_skinny is more responsive in BLOCKING-mode
what OS are you on?
Linux netthieu 2.6.32-5-686 #1 SMP Thu Aug 12 13:38:27 UTC 2010 i686 GNU/Linux
this is Debian testing using FS built from recent git using debian packaging.
in my findings not more efficient, not at all reliable.
Googling some, I found (from http://www.ibm.com/developerworks/linux/library/l-async/)
"Synchronous non-blocking I/O: A less efficient variant of synchronous blocking". This is what we are doing?
The good solutions IMO are: - Synchronous blocking I/O: simple and efficient - Asynchronous blocking I/O: not so good. use of "select" is not recommended - Asynchronous non-blocking I/O (AIO) which is a completely different API (portable?) which should fit well with FS's event model but with a lot of work
Maybe we can make it configurable at build-time (as it is in mod_skinny) to allow performance testing? Perhaps the results will be very different from one platform to another.
Mathieu
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_______________________________________________ FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
participants (4)
-
Anthony Minessale
-
Mathieu Parent
-
Oleg Khovayko
-
Steven Ayre