[Freeswitch-users] High load kernel
Giacomo Vacca
giacomo.vacca at gmail.com
Fri May 6 11:50:07 MSD 2016
On 5 May 2016 at 10:39, Agustí Ubalde <aubalde at presenceco.com> wrote:
> Hi,
>
> These are the results of the following commands:
>
> *ps -LlFm -p `pidof freeswitch`*
>
> =====================================================================================
> F S UID PID PPID LWP C NLWP PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME
> CMD
>
> =====================================================================================
> 1 R 499 - - *3645* 10 - 99 19 - - - - 5 May02 -
> *05:56:26* -
>
> =====================================================================================
>
> *pstack `pidof freewitch`*
>
> =====================================================================================
> Thread 59 (Thread 0x7f4c196da700 (LWP *3645*)):
> #0 0x000000304d6e8e63 in epoll_wait () from /lib64/libc.so.6
> #1 0x00007f4c1af51ba5 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so
> #2 0x00007f4c1af51a28 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so
> #3 0x00007f4c1af4f715 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so
> #4 0x000000304da079d1 in start_thread () from /lib64/libpthread.so.0
> #5 0x000000304d6e886d in clone () from /lib64/libc.so.6
>
> =====================================================================================
>
> Any idea to get more information?
>
Run strace without -c to see the details of the epoll (what status is
returned?) and identify the FD associated to that activity. From there you
can identify on what port the socket is listening and from the logs
retrieve the call that may have caused the thread to remain looping.
If you can re-build FS take a look at the patch provided in
https://freeswitch.org/jira/browse/FS-8726, see if it can be applicable to
the version you're using and whether after the application it keeps
happening or not.
If you haven't done yet, I'd suggest then raising a JIRA with all the
collected info.
Good luck,
Giacomo
>
>
> Thanks,
>
> *PRESENCE TECHNOLOGY*
> *Agustí Ubalde Bellot*
> Chief Developer
> C/ Comte Urgell 240 3A
> Barcelona 08036
> aubalde at presenceco.com
>
> Ph: +34 93 10 10 300
> Fx: +34 93 10 10 333
>
> *www.presenceco.com* <http://www.presenceco.com/>
>
> *Follow us on:*
>
> *[image: tw]* <http://twitter.com/presencetech>*[image: yt]*
> <http://www.youtube.com/PresenceTechnology>*[image: in]*
> <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]*
> <http://www.slideshare.net/PresenceTech>*[image: fb]*
> <http://www.facebook.com/presencetech>
>
> For additional information, please visit our website *www.presenceco.com*
> <http://www.presenceco.com/>
>
> 2016-05-02 17:44 GMT+02:00 Agustí Ubalde <aubalde at presenceco.com>:
>
>> Hi Giacomo,
>>
>> It remain "stuck" polling until Freeswitch is restarted.
>>
>>
>> Regards,
>>
>> *PRESENCE TECHNOLOGY*
>> *Agustí Ubalde Bellot*
>> Chief Developer
>> C/ Comte Urgell 240 3A
>> Barcelona 08036
>> aubalde at presenceco.com
>>
>> Ph: +34 93 10 10 300
>> Fx: +34 93 10 10 333
>>
>> *www.presenceco.com* <http://www.presenceco.com/>
>>
>> *Follow us on:*
>>
>> *[image: tw]* <http://twitter.com/presencetech>*[image: yt]*
>> <http://www.youtube.com/PresenceTechnology>*[image: in]*
>> <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]*
>> <http://www.slideshare.net/PresenceTech>*[image: fb]*
>> <http://www.facebook.com/presencetech>
>>
>> For additional information, please visit our website *www.presenceco.com*
>> <http://www.presenceco.com/>
>>
>> 2016-05-02 17:20 GMT+02:00 Giacomo Vacca <giacomo.vacca at gmail.com>:
>>
>>> Hi Agustì,
>>> does the thread free CPU as soon as the number of concurrent calls goes
>>> down, or does it remain "stuck" polling?
>>>
>>> I don't want to provide a red herring here but I had a slightly similar
>>> issue, reported in https://freeswitch.org/jira/browse/FS-8726
>>>
>>> The call thread that was saturating the CPU was tracked back by looking
>>> at the file descriptor shown by strace, then with netstat identifying the
>>> RTP port in use and then in the logs to find the session using that port.
>>>
>>> Hope this can provide some additional ideas,
>>>
>>> Giacomo
>>>
>>> On 2 May 2016 at 17:03, Agustí Ubalde <aubalde at presenceco.com> wrote:
>>>
>>>> Hi Marek,
>>>>
>>>> Here are the results of the checks:
>>>>
>>>> ======================================================================
>>>> *# top*
>>>> ======================================================================
>>>> top - 10:16:15 up 10 days, 46 min, 3 users, load average: 1.13, 0.66,
>>>> 0.27
>>>> Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie
>>>> Cpu0 : 0.0%us, 0.0%sy, 15.7%ni, 84.3%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Cpu1 : 0.0%us, 0.7%sy, 9.3%ni, 90.0%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Cpu2 : 0.0%us, 0.0%sy, 9.6%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Cpu3 : 0.0%us, 1.0%sy, 6.6%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> *Cpu4 : 0.0%us, 72.7%sy, 27.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st* -----> HIGH CPU SYSTEM LOAD
>>>> Cpu5 : 0.0%us, 0.0%sy, 14.2%ni, 85.8%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Cpu6 : 0.0%us, 1.4%sy, 8.2%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Cpu7 : 0.0%us, 0.0%sy, 9.4%ni, 90.6%id, 0.0%wa, 0.0%hi, 0.0%si,
>>>> 0.0%st
>>>> Mem: 1921520k total, 995092k used, 926428k free, 185628k buffers
>>>> Swap: 786428k total, 0k used, 786428k free, 533952k cached
>>>> ======================================================================
>>>>
>>>> ======================================================================
>>>> *# ps -AL | grep freeswitch*
>>>> ======================================================================
>>>> 27653 27653 ? 00:00:00 freeswitch
>>>> 27653 27654 ? 00:00:00 freeswitch
>>>> 27653 27656 ? 00:00:08 freeswitch
>>>> 27653 27657 ? 00:00:01 freeswitch
>>>> 27653 27660 ? 00:00:00 freeswitch
>>>> 27653 27661 ? 00:00:01 freeswitch
>>>> 27653 27662 ? 00:00:00 freeswitch
>>>> 27653 27663 ? 00:00:00 freeswitch
>>>> 27653 27666 ? 00:00:03 freeswitch
>>>> 27653 27667 ? 00:00:00 freeswitch
>>>> 27653 27668 ? 00:00:00 freeswitch
>>>> 27653 27669 ? 00:00:00 freeswitch
>>>> 27653 27670 ? 00:00:00 freeswitch
>>>> 27653 27671 ? 00:00:00 freeswitch
>>>> 27653 27672 ? 00:00:00 freeswitch
>>>> 27653 27673 ? 00:00:00 freeswitch
>>>> *27653 27674 ? 01:23:45 freeswitch* -----> PROCESS TO CHECK
>>>> 27653 27675 ? 00:00:00 freeswitch
>>>> 27653 27676 ? 00:00:00 freeswitch
>>>> 27653 27677 ? 00:00:00 freeswitch
>>>> 27653 27678 ? 00:00:00 freeswitch
>>>> 27653 27679 ? 00:00:00 freeswitch
>>>> 27653 27680 ? 00:00:00 freeswitch
>>>> 27653 27681 ? 00:00:00 freeswitch
>>>> 27653 27682 ? 00:00:00 freeswitch
>>>> 27653 27683 ? 00:00:00 freeswitch
>>>> 27653 27684 ? 00:00:00 freeswitch
>>>> 27653 27685 ? 00:00:00 freeswitch
>>>> 27653 27686 ? 00:00:00 freeswitch
>>>> 27653 27692 ? 00:00:00 freeswitch
>>>> 27653 27693 ? 00:00:04 freeswitch
>>>> 27653 27694 ? 00:00:00 freeswitch
>>>> 27653 27695 ? 00:00:00 freeswitch
>>>> 27653 27719 ? 00:00:00 freeswitch
>>>> 27653 27725 ? 00:01:00 freeswitch
>>>> 27653 27726 ? 00:04:52 freeswitch
>>>> 27653 27738 ? 00:04:42 freeswitch
>>>> 27653 27739 ? 00:00:55 freeswitch
>>>> 27653 27741 ? 00:04:40 freeswitch
>>>> 27653 27743 ? 00:00:52 freeswitch
>>>> 27653 27744 ? 00:04:39 freeswitch
>>>> 27653 27745 ? 00:00:55 freeswitch
>>>> 27653 27748 ? 00:00:58 freeswitch
>>>> 27653 27749 ? 00:04:28 freeswitch
>>>> 27653 27751 ? 00:04:38 freeswitch
>>>> 27653 27752 ? 00:00:52 freeswitch
>>>> 27653 27764 ? 00:00:52 freeswitch
>>>> 27653 27765 ? 00:04:32 freeswitch
>>>> 27653 27767 ? 00:04:37 freeswitch
>>>> 27653 27768 ? 00:00:54 freeswitch
>>>> 27653 27769 ? 00:04:36 freeswitch
>>>> 27653 27770 ? 00:00:52 freeswitch
>>>> 27653 27773 ? 00:04:38 freeswitch
>>>> 27653 27774 ? 00:00:54 freeswitch
>>>> 27653 27776 ? 00:04:37 freeswitch
>>>> 27653 27777 ? 00:00:49 freeswitch
>>>> 27653 27780 ? 00:04:36 freeswitch
>>>> 27653 27781 ? 00:00:48 freeswitch
>>>> 27653 27782 ? 00:01:15 freeswitch
>>>> 27653 27783 ? 00:04:05 freeswitch
>>>> 27653 27785 ? 00:04:26 freeswitch
>>>> 27653 27786 ? 00:00:51 freeswitch
>>>> 27653 27794 ? 00:01:00 freeswitch
>>>> 27653 27795 ? 00:03:58 freeswitch
>>>> 27653 27797 ? 00:04:24 freeswitch
>>>> 27653 27798 ? 00:00:48 freeswitch
>>>> 27653 27801 ? 00:04:14 freeswitch
>>>> 27653 27802 ? 00:00:43 freeswitch
>>>> 27653 27804 ? 00:01:25 freeswitch
>>>> 27653 27805 ? 00:03:20 freeswitch
>>>> 27653 27807 ? 00:01:58 freeswitch
>>>> 27653 27809 ? 00:04:10 freeswitch
>>>> 27653 27810 ? 00:00:44 freeswitch
>>>> 27653 27813 ? 00:04:12 freeswitch
>>>> 27653 27814 ? 00:00:44 freeswitch
>>>> 27653 27823 ? 00:04:02 freeswitch
>>>> 27653 27824 ? 00:00:36 freeswitch
>>>> 27653 27838 ? 00:00:56 freeswitch
>>>> 27653 27839 ? 00:02:58 freeswitch
>>>> 27653 27870 ? 00:02:55 freeswitch
>>>> 27653 27871 ? 00:00:32 freeswitch
>>>> 27653 27893 ? 00:02:37 freeswitch
>>>> 27653 27894 ? 00:00:06 freeswitch
>>>> 27653 27927 ? 00:02:24 freeswitch
>>>> 27653 27929 ? 00:00:26 freeswitch
>>>> 27653 27948 ? 00:02:23 freeswitch
>>>> 27653 27949 ? 00:00:26 freeswitch
>>>> 27653 27955 ? 00:02:18 freeswitch
>>>> 27653 27956 ? 00:00:25 freeswitch
>>>> 27653 27957 ? 00:02:19 freeswitch
>>>> 27653 27958 ? 00:00:26 freeswitch
>>>> 27653 27963 ? 00:00:30 freeswitch
>>>> 27653 27964 ? 00:01:59 freeswitch
>>>> 27653 27968 ? 00:02:09 freeswitch
>>>> 27653 27969 ? 00:00:25 freeswitch
>>>> 27653 27970 ? 00:02:08 freeswitch
>>>> 27653 27971 ? 00:00:24 freeswitch
>>>> 27653 27983 ? 00:00:27 freeswitch
>>>> 27653 27984 ? 00:01:48 freeswitch
>>>> 27653 27989 ? 00:01:32 freeswitch
>>>> 27653 27990 ? 00:00:06 freeswitch
>>>> 27653 27995 ? 00:01:31 freeswitch
>>>> 27653 27996 ? 00:00:17 freeswitch
>>>> 27653 27997 ? 00:01:42 freeswitch
>>>> 27653 27999 ? 00:00:19 freeswitch
>>>> 27653 28001 ? 00:01:38 freeswitch
>>>> 27653 28002 ? 00:00:13 freeswitch
>>>> 27653 28006 ? 00:00:26 freeswitch
>>>> 27653 28007 ? 00:00:45 freeswitch
>>>> 27653 28010 ? 00:01:32 freeswitch
>>>> 27653 28011 ? 00:00:16 freeswitch
>>>> 27653 28022 ? 00:00:14 freeswitch
>>>> 27653 28047 ? 00:01:00 freeswitch
>>>> 27653 28048 ? 00:00:10 freeswitch
>>>> ======================================================================
>>>>
>>>> ======================================================================
>>>> *# strace -c -p 27674*
>>>> ======================================================================
>>>> Process 27674 attached - interrupt to quit
>>>> ^CProcess 27674 detached
>>>> % time seconds usecs/call calls errors syscall
>>>> ------ ----------- ----------- --------- --------- ----------------
>>>> * 94.08 18.452455 0 63874778 epoll_wait*
>>>> 5.90 1.156911 414 2797 nanosleep
>>>> 0.01 0.001940 1 3041 140 read
>>>> 0.01 0.001000 25 40 6 futex
>>>> 0.00 0.000655 0 1408 21 write
>>>> 0.00 0.000449 0 1389 sendto
>>>> 0.00 0.000356 0 1348 recvfrom
>>>> 0.00 0.000031 2 15 recvmsg
>>>> 0.00 0.000031 1 33 9 shutdown
>>>> 0.00 0.000029 1 38 mprotect
>>>> 0.00 0.000025 3 10 socket
>>>> 0.00 0.000023 2 10 bind
>>>> 0.00 0.000000 0 38 close
>>>> 0.00 0.000000 0 2 madvise
>>>> 0.00 0.000000 0 5 5 connect
>>>> 0.00 0.000000 0 30 accept
>>>> 0.00 0.000000 0 10 getsockname
>>>> 0.00 0.000000 0 140 setsockopt
>>>> 0.00 0.000000 0 9 getsockopt
>>>> 0.00 0.000000 0 208 fcntl
>>>> 0.00 0.000000 0 68 epoll_ctl
>>>> ------ ----------- ----------- --------- --------- ----------------
>>>> 100.00 19.613905 63885417 181 total
>>>> ======================================================================
>>>>
>>>>
>>>> Regards,
>>>>
>>>> *PRESENCE TECHNOLOGY*
>>>> *Agustí Ubalde Bellot*
>>>> Chief Developer
>>>> C/ Comte Urgell 240 3A
>>>> Barcelona 08036
>>>> aubalde at presenceco.com
>>>>
>>>> Ph: +34 93 10 10 300
>>>> Fx: +34 93 10 10 333
>>>>
>>>> *www.presenceco.com* <http://www.presenceco.com/>
>>>>
>>>> *Follow us on:*
>>>>
>>>> *[image: tw]* <http://twitter.com/presencetech>*[image: yt]*
>>>> <http://www.youtube.com/PresenceTechnology>*[image: in]*
>>>> <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]*
>>>> <http://www.slideshare.net/PresenceTech>*[image: fb]*
>>>> <http://www.facebook.com/presencetech>
>>>>
>>>> For additional information, please visit our website
>>>> *www.presenceco.com* <http://www.presenceco.com/>
>>>>
>>>> 2016-05-02 13:38 GMT+02:00 Marek Červenka <cervajs at fpf.slu.cz>:
>>>>
>>>>> Dne 28.4.2016 v 20:15 Michael Jerris napsal(a):
>>>>>
>>>>> Strongly advise to be using latest release in 1.6, particularly for
>>>>> anything webrtc related.
>>>>> Strongly advice to use Debian 8, there were significant known
>>>>> performance problems on CentOS 6
>>>>>
>>>>>
>>>>> can you send us link with description of centos6 performance problems?
>>>>> thanks
>>>>>
>>>>>
>>>>>
>>>>> On Apr 28, 2016, at 6:41 AM, Agustí Ubalde <aubalde at presenceco.com>
>>>>> wrote:
>>>>>
>>>>> Which OS, FS revision, platform, hardware, ram, cpu, etc are you using?
>>>>>
>>>>>
>>>>> - CentOS 6.5 64b virutalized (ESXi)
>>>>> - Freeswitch 1.4.26
>>>>> - 8vCPU - 4GB RAM
>>>>>
>>>>>
>>>>> Which traffic are you handling?
>>>>>
>>>>>
>>>>> - 60-70 active calls (WebRTC extensions)
>>>>>
>>>>>
>>>>> What are exactly the simptoms?
>>>>>
>>>>>
>>>>> - top command shows high system cpu load constantly (CPU 3)
>>>>>
>>>>>
>>>>> How you get them? Is that reproducible?
>>>>>
>>>>>
>>>>> - Always whit the same load (60-70 calls)
>>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------------
>>>>> Marek Cervenka
>>>>> =======================================
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> *Presence Technology - DisclaimerThis message, its content and any file
>>>> attached thereto is for the intended recipient only and is confidential and
>>>> /or privileged. If you have received this e-mail in error or had access to
>>>> it, you should note that the information in it is private and any use
>>>> thereof is unauthorized. In such an event please notify us by e-mail or by
>>>> telephone (+ 34 93 10 10 300). Any reproduction of this e-mail by
>>>> whatsoever means and any transmission or dissemination thereof to other
>>>> persons is prohibited. It should be deleted immediately from your system.
>>>> Presence Technology reserves the right to take legal action against any
>>>> persons unlawfully gaining access to the content of any external message it
>>>> has emitted.*
>>>>
>>>> *For additional information, please visit our website **www.presenceco.com
>>>> <http://www.presenceco.com/>*
>>>>
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>
>
> *Presence Technology - DisclaimerThis message, its content and any file
> attached thereto is for the intended recipient only and is confidential and
> /or privileged. If you have received this e-mail in error or had access to
> it, you should note that the information in it is private and any use
> thereof is unauthorized. In such an event please notify us by e-mail or by
> telephone (+ 34 93 10 10 300). Any reproduction of this e-mail by
> whatsoever means and any transmission or dissemination thereof to other
> persons is prohibited. It should be deleted immediately from your system.
> Presence Technology reserves the right to take legal action against any
> persons unlawfully gaining access to the content of any external message it
> has emitted.*
>
> *For additional information, please visit our website **www.presenceco.com
> <http://www.presenceco.com/>*
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160506/4804ddec/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list