[Freeswitch-users] High load kernel
Giacomo Vacca
giacomo.vacca at gmail.com
Mon May 2 19:20:58 MSD 2016
Hi Agustì,
does the thread free CPU as soon as the number of concurrent calls goes
down, or does it remain "stuck" polling?
I don't want to provide a red herring here but I had a slightly similar
issue, reported in https://freeswitch.org/jira/browse/FS-8726
The call thread that was saturating the CPU was tracked back by looking at
the file descriptor shown by strace, then with netstat identifying the RTP
port in use and then in the logs to find the session using that port.
Hope this can provide some additional ideas,
Giacomo
On 2 May 2016 at 17:03, Agustí Ubalde <aubalde at presenceco.com> wrote:
> Hi Marek,
>
> Here are the results of the checks:
>
> ======================================================================
> *# top*
> ======================================================================
> top - 10:16:15 up 10 days, 46 min, 3 users, load average: 1.13, 0.66,
> 0.27
> Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.0%us, 0.0%sy, 15.7%ni, 84.3%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu1 : 0.0%us, 0.7%sy, 9.3%ni, 90.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu2 : 0.0%us, 0.0%sy, 9.6%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu3 : 0.0%us, 1.0%sy, 6.6%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> *Cpu4 : 0.0%us, 72.7%sy, 27.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st* -----> HIGH CPU SYSTEM LOAD
> Cpu5 : 0.0%us, 0.0%sy, 14.2%ni, 85.8%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu6 : 0.0%us, 1.4%sy, 8.2%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Cpu7 : 0.0%us, 0.0%sy, 9.4%ni, 90.6%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 1921520k total, 995092k used, 926428k free, 185628k buffers
> Swap: 786428k total, 0k used, 786428k free, 533952k cached
> ======================================================================
>
> ======================================================================
> *# ps -AL | grep freeswitch*
> ======================================================================
> 27653 27653 ? 00:00:00 freeswitch
> 27653 27654 ? 00:00:00 freeswitch
> 27653 27656 ? 00:00:08 freeswitch
> 27653 27657 ? 00:00:01 freeswitch
> 27653 27660 ? 00:00:00 freeswitch
> 27653 27661 ? 00:00:01 freeswitch
> 27653 27662 ? 00:00:00 freeswitch
> 27653 27663 ? 00:00:00 freeswitch
> 27653 27666 ? 00:00:03 freeswitch
> 27653 27667 ? 00:00:00 freeswitch
> 27653 27668 ? 00:00:00 freeswitch
> 27653 27669 ? 00:00:00 freeswitch
> 27653 27670 ? 00:00:00 freeswitch
> 27653 27671 ? 00:00:00 freeswitch
> 27653 27672 ? 00:00:00 freeswitch
> 27653 27673 ? 00:00:00 freeswitch
> *27653 27674 ? 01:23:45 freeswitch* -----> PROCESS TO CHECK
> 27653 27675 ? 00:00:00 freeswitch
> 27653 27676 ? 00:00:00 freeswitch
> 27653 27677 ? 00:00:00 freeswitch
> 27653 27678 ? 00:00:00 freeswitch
> 27653 27679 ? 00:00:00 freeswitch
> 27653 27680 ? 00:00:00 freeswitch
> 27653 27681 ? 00:00:00 freeswitch
> 27653 27682 ? 00:00:00 freeswitch
> 27653 27683 ? 00:00:00 freeswitch
> 27653 27684 ? 00:00:00 freeswitch
> 27653 27685 ? 00:00:00 freeswitch
> 27653 27686 ? 00:00:00 freeswitch
> 27653 27692 ? 00:00:00 freeswitch
> 27653 27693 ? 00:00:04 freeswitch
> 27653 27694 ? 00:00:00 freeswitch
> 27653 27695 ? 00:00:00 freeswitch
> 27653 27719 ? 00:00:00 freeswitch
> 27653 27725 ? 00:01:00 freeswitch
> 27653 27726 ? 00:04:52 freeswitch
> 27653 27738 ? 00:04:42 freeswitch
> 27653 27739 ? 00:00:55 freeswitch
> 27653 27741 ? 00:04:40 freeswitch
> 27653 27743 ? 00:00:52 freeswitch
> 27653 27744 ? 00:04:39 freeswitch
> 27653 27745 ? 00:00:55 freeswitch
> 27653 27748 ? 00:00:58 freeswitch
> 27653 27749 ? 00:04:28 freeswitch
> 27653 27751 ? 00:04:38 freeswitch
> 27653 27752 ? 00:00:52 freeswitch
> 27653 27764 ? 00:00:52 freeswitch
> 27653 27765 ? 00:04:32 freeswitch
> 27653 27767 ? 00:04:37 freeswitch
> 27653 27768 ? 00:00:54 freeswitch
> 27653 27769 ? 00:04:36 freeswitch
> 27653 27770 ? 00:00:52 freeswitch
> 27653 27773 ? 00:04:38 freeswitch
> 27653 27774 ? 00:00:54 freeswitch
> 27653 27776 ? 00:04:37 freeswitch
> 27653 27777 ? 00:00:49 freeswitch
> 27653 27780 ? 00:04:36 freeswitch
> 27653 27781 ? 00:00:48 freeswitch
> 27653 27782 ? 00:01:15 freeswitch
> 27653 27783 ? 00:04:05 freeswitch
> 27653 27785 ? 00:04:26 freeswitch
> 27653 27786 ? 00:00:51 freeswitch
> 27653 27794 ? 00:01:00 freeswitch
> 27653 27795 ? 00:03:58 freeswitch
> 27653 27797 ? 00:04:24 freeswitch
> 27653 27798 ? 00:00:48 freeswitch
> 27653 27801 ? 00:04:14 freeswitch
> 27653 27802 ? 00:00:43 freeswitch
> 27653 27804 ? 00:01:25 freeswitch
> 27653 27805 ? 00:03:20 freeswitch
> 27653 27807 ? 00:01:58 freeswitch
> 27653 27809 ? 00:04:10 freeswitch
> 27653 27810 ? 00:00:44 freeswitch
> 27653 27813 ? 00:04:12 freeswitch
> 27653 27814 ? 00:00:44 freeswitch
> 27653 27823 ? 00:04:02 freeswitch
> 27653 27824 ? 00:00:36 freeswitch
> 27653 27838 ? 00:00:56 freeswitch
> 27653 27839 ? 00:02:58 freeswitch
> 27653 27870 ? 00:02:55 freeswitch
> 27653 27871 ? 00:00:32 freeswitch
> 27653 27893 ? 00:02:37 freeswitch
> 27653 27894 ? 00:00:06 freeswitch
> 27653 27927 ? 00:02:24 freeswitch
> 27653 27929 ? 00:00:26 freeswitch
> 27653 27948 ? 00:02:23 freeswitch
> 27653 27949 ? 00:00:26 freeswitch
> 27653 27955 ? 00:02:18 freeswitch
> 27653 27956 ? 00:00:25 freeswitch
> 27653 27957 ? 00:02:19 freeswitch
> 27653 27958 ? 00:00:26 freeswitch
> 27653 27963 ? 00:00:30 freeswitch
> 27653 27964 ? 00:01:59 freeswitch
> 27653 27968 ? 00:02:09 freeswitch
> 27653 27969 ? 00:00:25 freeswitch
> 27653 27970 ? 00:02:08 freeswitch
> 27653 27971 ? 00:00:24 freeswitch
> 27653 27983 ? 00:00:27 freeswitch
> 27653 27984 ? 00:01:48 freeswitch
> 27653 27989 ? 00:01:32 freeswitch
> 27653 27990 ? 00:00:06 freeswitch
> 27653 27995 ? 00:01:31 freeswitch
> 27653 27996 ? 00:00:17 freeswitch
> 27653 27997 ? 00:01:42 freeswitch
> 27653 27999 ? 00:00:19 freeswitch
> 27653 28001 ? 00:01:38 freeswitch
> 27653 28002 ? 00:00:13 freeswitch
> 27653 28006 ? 00:00:26 freeswitch
> 27653 28007 ? 00:00:45 freeswitch
> 27653 28010 ? 00:01:32 freeswitch
> 27653 28011 ? 00:00:16 freeswitch
> 27653 28022 ? 00:00:14 freeswitch
> 27653 28047 ? 00:01:00 freeswitch
> 27653 28048 ? 00:00:10 freeswitch
> ======================================================================
>
> ======================================================================
> *# strace -c -p 27674*
> ======================================================================
> Process 27674 attached - interrupt to quit
> ^CProcess 27674 detached
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> * 94.08 18.452455 0 63874778 epoll_wait*
> 5.90 1.156911 414 2797 nanosleep
> 0.01 0.001940 1 3041 140 read
> 0.01 0.001000 25 40 6 futex
> 0.00 0.000655 0 1408 21 write
> 0.00 0.000449 0 1389 sendto
> 0.00 0.000356 0 1348 recvfrom
> 0.00 0.000031 2 15 recvmsg
> 0.00 0.000031 1 33 9 shutdown
> 0.00 0.000029 1 38 mprotect
> 0.00 0.000025 3 10 socket
> 0.00 0.000023 2 10 bind
> 0.00 0.000000 0 38 close
> 0.00 0.000000 0 2 madvise
> 0.00 0.000000 0 5 5 connect
> 0.00 0.000000 0 30 accept
> 0.00 0.000000 0 10 getsockname
> 0.00 0.000000 0 140 setsockopt
> 0.00 0.000000 0 9 getsockopt
> 0.00 0.000000 0 208 fcntl
> 0.00 0.000000 0 68 epoll_ctl
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 19.613905 63885417 181 total
> ======================================================================
>
>
> Regards,
>
> *PRESENCE TECHNOLOGY*
> *Agustí Ubalde Bellot*
> Chief Developer
> C/ Comte Urgell 240 3A
> Barcelona 08036
> aubalde at presenceco.com
>
> Ph: +34 93 10 10 300
> Fx: +34 93 10 10 333
>
> *www.presenceco.com* <http://www.presenceco.com/>
>
> *Follow us on:*
>
> *[image: tw]* <http://twitter.com/presencetech>*[image: yt]*
> <http://www.youtube.com/PresenceTechnology>*[image: in]*
> <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]*
> <http://www.slideshare.net/PresenceTech>*[image: fb]*
> <http://www.facebook.com/presencetech>
>
> For additional information, please visit our website *www.presenceco.com*
> <http://www.presenceco.com/>
>
> 2016-05-02 13:38 GMT+02:00 Marek Červenka <cervajs at fpf.slu.cz>:
>
>> Dne 28.4.2016 v 20:15 Michael Jerris napsal(a):
>>
>> Strongly advise to be using latest release in 1.6, particularly for
>> anything webrtc related.
>> Strongly advice to use Debian 8, there were significant known performance
>> problems on CentOS 6
>>
>>
>> can you send us link with description of centos6 performance problems?
>> thanks
>>
>>
>>
>> On Apr 28, 2016, at 6:41 AM, Agustí Ubalde <aubalde at presenceco.com>
>> wrote:
>>
>> Which OS, FS revision, platform, hardware, ram, cpu, etc are you using?
>>
>>
>> - CentOS 6.5 64b virutalized (ESXi)
>> - Freeswitch 1.4.26
>> - 8vCPU - 4GB RAM
>>
>>
>> Which traffic are you handling?
>>
>>
>> - 60-70 active calls (WebRTC extensions)
>>
>>
>> What are exactly the simptoms?
>>
>>
>> - top command shows high system cpu load constantly (CPU 3)
>>
>>
>> How you get them? Is that reproducible?
>>
>>
>> - Always whit the same load (60-70 calls)
>>
>>
>> --
>> ---------------------------------------
>> Marek Cervenka
>> =======================================
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> *Presence Technology - DisclaimerThis message, its content and any file
> attached thereto is for the intended recipient only and is confidential and
> /or privileged. If you have received this e-mail in error or had access to
> it, you should note that the information in it is private and any use
> thereof is unauthorized. In such an event please notify us by e-mail or by
> telephone (+ 34 93 10 10 300). Any reproduction of this e-mail by
> whatsoever means and any transmission or dissemination thereof to other
> persons is prohibited. It should be deleted immediately from your system.
> Presence Technology reserves the right to take legal action against any
> persons unlawfully gaining access to the content of any external message it
> has emitted.*
>
> *For additional information, please visit our website **www.presenceco.com
> <http://www.presenceco.com/>*
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160502/7b8cb8f5/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list