High latencies may cause the problem? Regards, *PRESENCE TECHNOLOGY* *Agustí Ubalde Bellot* Chief Developer C/ Comte Urgell 240 3A Barcelona 08036 aubalde@presenceco.com Ph: +34 93 10 10 300 Fx: +34 93 10 10 333 *www.presenceco.com* <http://www.presenceco.com/> *Follow us on:* *[image: tw]* <http://twitter.com/presencetech>*[image: yt]* <http://www.youtube.com/PresenceTechnology>*[image: in]* <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]* <http://www.slideshare.net/PresenceTech>*[image: fb]* <http://www.facebook.com/presencetech> For additional information, please visit our website *www.presenceco.com* <http://www.presenceco.com/> 2016-05-05 10:39 GMT+02:00 Agustí Ubalde <aubalde@presenceco.com>:
Hi,
These are the results of the following commands:
*ps -LlFm -p `pidof freeswitch`*
===================================================================================== F S UID PID PPID LWP C NLWP PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
===================================================================================== 1 R 499 - - *3645* 10 - 99 19 - - - - 5 May02 - *05:56:26* -
=====================================================================================
*pstack `pidof freewitch`*
===================================================================================== Thread 59 (Thread 0x7f4c196da700 (LWP *3645*)): #0 0x000000304d6e8e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f4c1af51ba5 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so #2 0x00007f4c1af51a28 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so #3 0x00007f4c1af4f715 in ?? () from /usr/local/freeswitch/mod/mod_sofia.so #4 0x000000304da079d1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000304d6e886d in clone () from /lib64/libc.so.6
=====================================================================================
Any idea to get more information?
Thanks,
*PRESENCE TECHNOLOGY* *Agustí Ubalde Bellot* Chief Developer C/ Comte Urgell 240 3A Barcelona 08036 aubalde@presenceco.com
Ph: +34 93 10 10 300 Fx: +34 93 10 10 333
*www.presenceco.com* <http://www.presenceco.com/>
*Follow us on:*
*[image: tw]* <http://twitter.com/presencetech>*[image: yt]* <http://www.youtube.com/PresenceTechnology>*[image: in]* <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]* <http://www.slideshare.net/PresenceTech>*[image: fb]* <http://www.facebook.com/presencetech>
For additional information, please visit our website *www.presenceco.com* <http://www.presenceco.com/>
2016-05-02 17:44 GMT+02:00 Agustí Ubalde <aubalde@presenceco.com>:
Hi Giacomo,
It remain "stuck" polling until Freeswitch is restarted.
Regards,
*PRESENCE TECHNOLOGY* *Agustí Ubalde Bellot* Chief Developer C/ Comte Urgell 240 3A Barcelona 08036 aubalde@presenceco.com
Ph: +34 93 10 10 300 Fx: +34 93 10 10 333
*www.presenceco.com* <http://www.presenceco.com/>
*Follow us on:*
*[image: tw]* <http://twitter.com/presencetech>*[image: yt]* <http://www.youtube.com/PresenceTechnology>*[image: in]* <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]* <http://www.slideshare.net/PresenceTech>*[image: fb]* <http://www.facebook.com/presencetech>
For additional information, please visit our website *www.presenceco.com* <http://www.presenceco.com/>
2016-05-02 17:20 GMT+02:00 Giacomo Vacca <giacomo.vacca@gmail.com>:
Hi Agustì, does the thread free CPU as soon as the number of concurrent calls goes down, or does it remain "stuck" polling?
I don't want to provide a red herring here but I had a slightly similar issue, reported in https://freeswitch.org/jira/browse/FS-8726
The call thread that was saturating the CPU was tracked back by looking at the file descriptor shown by strace, then with netstat identifying the RTP port in use and then in the logs to find the session using that port.
Hope this can provide some additional ideas,
Giacomo
On 2 May 2016 at 17:03, Agustí Ubalde <aubalde@presenceco.com> wrote:
Hi Marek,
Here are the results of the checks:
====================================================================== *# top* ====================================================================== top - 10:16:15 up 10 days, 46 min, 3 users, load average: 1.13, 0.66, 0.27 Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 15.7%ni, 84.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.7%sy, 9.3%ni, 90.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 9.6%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 1.0%sy, 6.6%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st *Cpu4 : 0.0%us, 72.7%sy, 27.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st* -----> HIGH CPU SYSTEM LOAD Cpu5 : 0.0%us, 0.0%sy, 14.2%ni, 85.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 1.4%sy, 8.2%ni, 90.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 9.4%ni, 90.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1921520k total, 995092k used, 926428k free, 185628k buffers Swap: 786428k total, 0k used, 786428k free, 533952k cached ======================================================================
====================================================================== *# ps -AL | grep freeswitch* ====================================================================== 27653 27653 ? 00:00:00 freeswitch 27653 27654 ? 00:00:00 freeswitch 27653 27656 ? 00:00:08 freeswitch 27653 27657 ? 00:00:01 freeswitch 27653 27660 ? 00:00:00 freeswitch 27653 27661 ? 00:00:01 freeswitch 27653 27662 ? 00:00:00 freeswitch 27653 27663 ? 00:00:00 freeswitch 27653 27666 ? 00:00:03 freeswitch 27653 27667 ? 00:00:00 freeswitch 27653 27668 ? 00:00:00 freeswitch 27653 27669 ? 00:00:00 freeswitch 27653 27670 ? 00:00:00 freeswitch 27653 27671 ? 00:00:00 freeswitch 27653 27672 ? 00:00:00 freeswitch 27653 27673 ? 00:00:00 freeswitch *27653 27674 ? 01:23:45 freeswitch* -----> PROCESS TO CHECK 27653 27675 ? 00:00:00 freeswitch 27653 27676 ? 00:00:00 freeswitch 27653 27677 ? 00:00:00 freeswitch 27653 27678 ? 00:00:00 freeswitch 27653 27679 ? 00:00:00 freeswitch 27653 27680 ? 00:00:00 freeswitch 27653 27681 ? 00:00:00 freeswitch 27653 27682 ? 00:00:00 freeswitch 27653 27683 ? 00:00:00 freeswitch 27653 27684 ? 00:00:00 freeswitch 27653 27685 ? 00:00:00 freeswitch 27653 27686 ? 00:00:00 freeswitch 27653 27692 ? 00:00:00 freeswitch 27653 27693 ? 00:00:04 freeswitch 27653 27694 ? 00:00:00 freeswitch 27653 27695 ? 00:00:00 freeswitch 27653 27719 ? 00:00:00 freeswitch 27653 27725 ? 00:01:00 freeswitch 27653 27726 ? 00:04:52 freeswitch 27653 27738 ? 00:04:42 freeswitch 27653 27739 ? 00:00:55 freeswitch 27653 27741 ? 00:04:40 freeswitch 27653 27743 ? 00:00:52 freeswitch 27653 27744 ? 00:04:39 freeswitch 27653 27745 ? 00:00:55 freeswitch 27653 27748 ? 00:00:58 freeswitch 27653 27749 ? 00:04:28 freeswitch 27653 27751 ? 00:04:38 freeswitch 27653 27752 ? 00:00:52 freeswitch 27653 27764 ? 00:00:52 freeswitch 27653 27765 ? 00:04:32 freeswitch 27653 27767 ? 00:04:37 freeswitch 27653 27768 ? 00:00:54 freeswitch 27653 27769 ? 00:04:36 freeswitch 27653 27770 ? 00:00:52 freeswitch 27653 27773 ? 00:04:38 freeswitch 27653 27774 ? 00:00:54 freeswitch 27653 27776 ? 00:04:37 freeswitch 27653 27777 ? 00:00:49 freeswitch 27653 27780 ? 00:04:36 freeswitch 27653 27781 ? 00:00:48 freeswitch 27653 27782 ? 00:01:15 freeswitch 27653 27783 ? 00:04:05 freeswitch 27653 27785 ? 00:04:26 freeswitch 27653 27786 ? 00:00:51 freeswitch 27653 27794 ? 00:01:00 freeswitch 27653 27795 ? 00:03:58 freeswitch 27653 27797 ? 00:04:24 freeswitch 27653 27798 ? 00:00:48 freeswitch 27653 27801 ? 00:04:14 freeswitch 27653 27802 ? 00:00:43 freeswitch 27653 27804 ? 00:01:25 freeswitch 27653 27805 ? 00:03:20 freeswitch 27653 27807 ? 00:01:58 freeswitch 27653 27809 ? 00:04:10 freeswitch 27653 27810 ? 00:00:44 freeswitch 27653 27813 ? 00:04:12 freeswitch 27653 27814 ? 00:00:44 freeswitch 27653 27823 ? 00:04:02 freeswitch 27653 27824 ? 00:00:36 freeswitch 27653 27838 ? 00:00:56 freeswitch 27653 27839 ? 00:02:58 freeswitch 27653 27870 ? 00:02:55 freeswitch 27653 27871 ? 00:00:32 freeswitch 27653 27893 ? 00:02:37 freeswitch 27653 27894 ? 00:00:06 freeswitch 27653 27927 ? 00:02:24 freeswitch 27653 27929 ? 00:00:26 freeswitch 27653 27948 ? 00:02:23 freeswitch 27653 27949 ? 00:00:26 freeswitch 27653 27955 ? 00:02:18 freeswitch 27653 27956 ? 00:00:25 freeswitch 27653 27957 ? 00:02:19 freeswitch 27653 27958 ? 00:00:26 freeswitch 27653 27963 ? 00:00:30 freeswitch 27653 27964 ? 00:01:59 freeswitch 27653 27968 ? 00:02:09 freeswitch 27653 27969 ? 00:00:25 freeswitch 27653 27970 ? 00:02:08 freeswitch 27653 27971 ? 00:00:24 freeswitch 27653 27983 ? 00:00:27 freeswitch 27653 27984 ? 00:01:48 freeswitch 27653 27989 ? 00:01:32 freeswitch 27653 27990 ? 00:00:06 freeswitch 27653 27995 ? 00:01:31 freeswitch 27653 27996 ? 00:00:17 freeswitch 27653 27997 ? 00:01:42 freeswitch 27653 27999 ? 00:00:19 freeswitch 27653 28001 ? 00:01:38 freeswitch 27653 28002 ? 00:00:13 freeswitch 27653 28006 ? 00:00:26 freeswitch 27653 28007 ? 00:00:45 freeswitch 27653 28010 ? 00:01:32 freeswitch 27653 28011 ? 00:00:16 freeswitch 27653 28022 ? 00:00:14 freeswitch 27653 28047 ? 00:01:00 freeswitch 27653 28048 ? 00:00:10 freeswitch ======================================================================
====================================================================== *# strace -c -p 27674* ====================================================================== Process 27674 attached - interrupt to quit ^CProcess 27674 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- * 94.08 18.452455 0 63874778 epoll_wait* 5.90 1.156911 414 2797 nanosleep 0.01 0.001940 1 3041 140 read 0.01 0.001000 25 40 6 futex 0.00 0.000655 0 1408 21 write 0.00 0.000449 0 1389 sendto 0.00 0.000356 0 1348 recvfrom 0.00 0.000031 2 15 recvmsg 0.00 0.000031 1 33 9 shutdown 0.00 0.000029 1 38 mprotect 0.00 0.000025 3 10 socket 0.00 0.000023 2 10 bind 0.00 0.000000 0 38 close 0.00 0.000000 0 2 madvise 0.00 0.000000 0 5 5 connect 0.00 0.000000 0 30 accept 0.00 0.000000 0 10 getsockname 0.00 0.000000 0 140 setsockopt 0.00 0.000000 0 9 getsockopt 0.00 0.000000 0 208 fcntl 0.00 0.000000 0 68 epoll_ctl ------ ----------- ----------- --------- --------- ---------------- 100.00 19.613905 63885417 181 total ======================================================================
Regards,
*PRESENCE TECHNOLOGY* *Agustí Ubalde Bellot* Chief Developer C/ Comte Urgell 240 3A Barcelona 08036 aubalde@presenceco.com
Ph: +34 93 10 10 300 Fx: +34 93 10 10 333
*www.presenceco.com* <http://www.presenceco.com/>
*Follow us on:*
*[image: tw]* <http://twitter.com/presencetech>*[image: yt]* <http://www.youtube.com/PresenceTechnology>*[image: in]* <http://www.linkedin.com/groups?mostPopular=&gid=2898558>*[image: ss]* <http://www.slideshare.net/PresenceTech>*[image: fb]* <http://www.facebook.com/presencetech>
For additional information, please visit our website *www.presenceco.com* <http://www.presenceco.com/>
2016-05-02 13:38 GMT+02:00 Marek Červenka <cervajs@fpf.slu.cz>:
Dne 28.4.2016 v 20:15 Michael Jerris napsal(a):
Strongly advise to be using latest release in 1.6, particularly for anything webrtc related. Strongly advice to use Debian 8, there were significant known performance problems on CentOS 6
can you send us link with description of centos6 performance problems? thanks
On Apr 28, 2016, at 6:41 AM, Agustí Ubalde <aubalde@presenceco.com> wrote:
Which OS, FS revision, platform, hardware, ram, cpu, etc are you using?
- CentOS 6.5 64b virutalized (ESXi) - Freeswitch 1.4.26 - 8vCPU - 4GB RAM
Which traffic are you handling?
- 60-70 active calls (WebRTC extensions)
What are exactly the simptoms?
- top command shows high system cpu load constantly (CPU 3)
How you get them? Is that reproducible?
- Always whit the same load (60-70 calls)
-- --------------------------------------- Marek Cervenka =======================================
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
*Presence Technology - DisclaimerThis message, its content and any file attached thereto is for the intended recipient only and is confidential and /or privileged. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorized. In such an event please notify us by e-mail or by telephone (+ 34 93 10 10 300). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. It should be deleted immediately from your system. Presence Technology reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted.*
*For additional information, please visit our website **www.presenceco.com <http://www.presenceco.com/>*
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- *Presence Technology - DisclaimerThis message, its content and any file attached thereto is for the intended recipient only and is confidential and /or privileged. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorized. In such an event please notify us by e-mail or by telephone (+ 34 93 10 10 300). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. It should be deleted immediately from your system. Presence Technology reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted.* *For additional information, please visit our website **www.presenceco.com <http://www.presenceco.com/>*