[Freeswitch-users] FreeSWITCH 100% CPU utilization

Shaun Stokes shaun.stokes at itec-support.co.uk
Wed Apr 12 13:53:30 MSD 2017


Although the documentation may be out of date there are situations where using more than 1 SIP profile is beneficial and can help overcome what could be a bottleneck.

We’ve found that spreading the load with roughly 750 registrations per SIP profile reduces packet loss that would otherwise cause registrations to occasionally fail to re-register, we didn’t have CPU spikes so slightly different symptoms. We are using presence, NDLB-force-report, nat-options-ping and 10 to 20 minute (sip-expires-max-deviation) registration expiry.

From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Anthony Minessale
Sent: 11 April 2017 16:04
To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
Subject: Re: [Freeswitch-users] FreeSWITCH 100% CPU utilization

Thank you.

That whole section is actually out of date and inaccurate.  I have removed it to prevent further confusion.


On Tue, Apr 11, 2017 at 9:29 AM, Örn Arnarson <orn at arnarson.net<mailto:orn at arnarson.net>> wrote:
Hello again,

It was on the old wiki that I saw it, but the same info is present on the new wiki:
https://freeswitch.org/confluence/display/FREESWITCH/Performance+Testing+and+Configurations
Specifically, this part:

o libsofia only handles 1 thread per profile, so if that is your bottle neck use more profiles

When one is not familiar with the internals of FreeSWITCH, as I'm not, one might incorrectly reach the same conclusion as I did. Not sure how to prevent that, though.

Just above that, it does mention the multi-threaded handling of FS where each call leg has its own thread, so if one reads the whole page (which I clearly didn't), this confusion is probably avoided.

Regards,
Örn

On Sun, Apr 9, 2017 at 1:50 AM, Anthony Minessale <anthony.minessale at gmail.com<mailto:anthony.minessale at gmail.com>> wrote:
What can happen is that the incorrect info can also get archived and its hard to undo so there is a similar duty to debunk anything that may lead to confusion for those who read it in years to come.  Maybe supply the wiki page you saw that on and we can correct it.  Hopefully its on the old wiki.

We can't do much to debug 1.2 so you are correct.  We will have to update to latest to get any clean idea of what you problem is.  (If it persists)
Rest assured that a properly functioning FS will do its best to distribute the load across all CPU when it can.


On Sat, Apr 8, 2017 at 8:03 PM, Örn Arnarson <orn at arnarson.net<mailto:orn at arnarson.net>> wrote:
Hi Anthony,

Thanks for your response.

You make a fair point. I was making some assumptions.

I wasn't really reporting an issue, though. At the time of the post I was asking questions about ways to leverage more of the CPU cores due to a misconception I had on how that was being done. Partially, the wording on the wiki is what led me to that incorrect assumption.

Subsequently, FreeSWITCH stopped running, for one reason or another, and upon restarting, the issue was gone. I felt a duty to report that for the archives, should someone else encounter a similar problem in the future.

For the record/archives, both the OS and the FreeSWITCH versions are ancient; Ubuntu 12.04.5 LTS, with FreeSWITCH version 1.2.14. They are both earmarked for upgrades later this year.

Sorry for wasting your guys' time.

Regards,
Örn

On Tue, Apr 4, 2017 at 12:02 AM, Anthony Minessale <anthony.minessale at gmail.com<mailto:anthony.minessale at gmail.com>> wrote:
You are making guesses and assumptions about the architecture and enabling FUD.
You should report issues to JIRA and not here and you should supply pertinent info such as version of the OS and cpu etc.



On Mon, Apr 3, 2017 at 5:21 PM, Brian West <brian at freeswitch.org<mailto:brian at freeswitch.org>> wrote:
Sure it's not getting attached? Or your DB?

On Mon, Apr 3, 2017 at 12:02 PM Örn Arnarson <orn at arnarson.net<mailto:orn at arnarson.net>> wrote:
Thanks again.

I did another strace, which for some reason crashed my FreeSWITCH (whoops), but after the restart, I'm seeing around 10% CPU utilization on that thread.

It seems that there was something odd going on with the main signalling thread.

Regards,
Örn

On Mon, Apr 3, 2017 at 4:57 PM, Brian West <brian at freeswitch.org<mailto:brian at freeswitch.org>> wrote:
RTP is in the Session thread and each session gets its own thread.  Two per bridged call.

On Mon, Apr 3, 2017 at 11:55 AM, Örn Arnarson <orn at arnarson.net<mailto:orn at arnarson.net>> wrote:
Thanks for your reply Brian.

Does RTP only have a single thread? Is it identifiable somehow? The 100 (120%, really) thread is likely the RTP thread then. In fact, after an strace on the thread and its forks it seems to be the case.

Presence is disabled and I'm receiving about 20 SIP messages per second combined on all profiles. Not quite getting the extensive processing time.

Regards,
Örn

On Mon, Apr 3, 2017 at 4:49 PM, Brian West <brian at freeswitch.org<mailto:brian at freeswitch.org>> wrote:
RTP already has its own thread,  The signaling thread in Sofia is the only bottle neck, Sounds like your presence or registration traffic is clogging up your system.

/b


On Mon, Apr 3, 2017 at 11:42 AM, Örn Arnarson <orn at arnarson.net<mailto:orn at arnarson.net>> wrote:
Hello FS users,

I'm running a FS server with 8 CPU cores and around 3000 users, ~50 concurrent calls.
FS is proxying the media.

I'm running into about a 1-1.5 second delay when forwarding SIP responses back to the UAC (200 OK for example).

It seems that almost all of the processing is being done by one CPU core, while the other 7 sit idle.

I found this on the old wiki:
libsofia only handles 1 thread per profile, so if that is your bottle neck use more profiles

There's no mention of this on the new one. Has this changed at all?

In any case, I'm assuming that SIP and RTP co-exist on the thread, seeing as I don't have that many SIP requests per second, and yet it is taking more than 1 second to forward the requests.

I guess my question is; Is it possible to offload the RTP handling off to different cores somehow? Or force FreeSWITCH to use more than one thread per sofia profile?

Regards,
Örn

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--

Brian West
brian at freeswitch.org<mailto:brian at freeswitch.org>

Twitter: @FreeSWITCH , @briankwest

http://www.freeswitchbook.com
http://www.freeswitchcookbook.com

Allison prompts for FreeSWITCH:

https://www.gofundme.com/allison-prompts-for-freeswitch


Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch>

T:+19184209001<tel:+1%20918-420-9001> | F:+19184209002<tel:+1%20918-420-9002> | M:+1918424WEST (9378)
Skype:briankwest

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--

Brian West
brian at freeswitch.org<mailto:brian at freeswitch.org>

Twitter: @FreeSWITCH , @briankwest

http://www.freeswitchbook.com
http://www.freeswitchcookbook.com

Allison prompts for FreeSWITCH:

https://www.gofundme.com/allison-prompts-for-freeswitch


Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch>

T:+19184209001<tel:+1%20918-420-9001> | F:+19184209002<tel:+1%20918-420-9002> | M:+1918424WEST (9378)
Skype:briankwest

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--

Brian West
brian at freeswitch.org<mailto:brian at freeswitch.org>

Twitter: @FreeSWITCH , @briankwest

http://www.freeswitchbook.com
http://www.freeswitchcookbook.com

Allison prompts for FreeSWITCH:

https://www.gofundme.com/allison-prompts-for-freeswitch


Got Bugs? Report them here<https://freeswitch.org/jira>! | Reddit: /r/freeswitch<https://www.reddit.com/r/freeswitch>

T:+19184209001<tel:(918)%20420-9001> | F:+19184209002<tel:(918)%20420-9002> | M:+1918424WEST (9378)
Skype:briankwest

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/http://cluecon.com/http://twitter.com/FreeSWITCH
☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org<mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900<tel:+1%20919-386-9900>

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/http://cluecon.com/http://twitter.com/FreeSWITCH
☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org<mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900<tel:+1%20919-386-9900>

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting at freeswitch.org<mailto:consulting at freeswitch.org>
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users at lists.freeswitch.org<mailto:FreeSWITCH-users at lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/http://cluecon.com/http://twitter.com/FreeSWITCH
☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org<mailto:sip%3A888 at conference.freeswitch.org>  ☎ +19193869900

https://www.youtube.com/watch?v=9XXgW34t40s
https://www.youtube.com/watch?v=NLaDpGQuZDA

______________________________________________________________________
This message has been checked for all known viruses by MessageLabs Virus Scanning Service.
______________________________________________________________________
[http://www.itec-support.co.uk/wp-content/uploads/2016/07/email_logo.jpg]
Shaun Stokes - Infrastructure Analyst

T :     01453 700713
E :     shaun.stokes at itec-support.co.uk
W :     www.itec-support.co.uk

Registered Address :- ITEC Support, Suite 2 Prospect House, Bath Road, Stroud, Gloucestershire GL5 3QF
Company No. 06908001

CONFIDENTIALITY NOTICE
This communication and the information it contains are intended for the person or organisation to which it is addressed. Its contents are confidential and may be protected in law. Unauthorised use, copying or disclosure of any of it may be unlawful. If you are not the intended recipient, please contact us immediately.
The contents of any attachments in this e-mail may contain software viruses, which could damage your own computer system. While ITEC Support has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checking procedure before opening any attachment.

______________________________________________________________________
This message has been checked for all known viruses by  MessageLabs Virus Scanning Service.
______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20170412/fe2ca096/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list