[Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences

Anthony Minessale anthony.minessale at gmail.com
Sat Aug 2 02:49:37 MSD 2014 

Its almost certain that you should do a fresh clean build.  Once you start
moving around in versions you can cause lots of problems with inconsistent
binaries etc.


On Fri, Aug 1, 2014 at 3:18 PM, Oleg Stolyar <olegstolyar at gmail.com> wrote:

> Talk about "inconsistent" (and embarrassing for me for accusing the new
> version)
>
> Now the older version that was rock solid for months crashed on me also.
>  This time I have the backtrace.  It's here:
> https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt.
>
> Please let me know if you'd like me to file a Jira and attach it to it or
> if you need this reproduced on the latest master before you can look into
> it.
>
> Really appreciate all the help and advice!
>
>
> On Thu, Jul 31, 2014 at 7:27 PM, Oleg Stolyar <olegstolyar at gmail.com>
> wrote:
>
>> Running a clean build is a great idea!  I will do that and report my
>> findings - it may be after ClueCon though.
>>
>>
>> On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale <
>> anthony.minessale at gmail.com> wrote:
>>
>>> I actually think you need to start over.  Nothing you say is consistent
>>> and I have a rule when doing debugging that when things don't add up, start
>>> over.
>>>
>>> Run latest GIT with clean build (remove all old binaries from bin lib
>>> and mod in FS install target)
>>>
>>> Start from there getting BT, evaluating CPU etc.  Jumping around
>>> versions is the leading way to cause problems that cannot be explained.
>>>
>>>
>>>
>>> On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>> wrote:
>>>
>>>> Yes, I enabled core dump and will get the backtrace if it crashes
>>>> again.  I need to make preparations for the rest of my system to handle
>>>> FreeSWITCH crashes better before I put the new code back in production, so
>>>> it may take a while.
>>>>
>>>> If I find the time I will also try to bisect git to find the commit
>>>> that caused this as Anthony suggested.
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike at jerris.com>
>>>> wrote:
>>>>
>>>>> There is no reason at all to think it has anything to do with the
>>>>> crash.  For the crash, simply get a backtrace of it so we can take a look
>>>>> at what is going on with the crash.
>>>>>
>>>>> On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Well, it gets a little complicated.  I need to enable multiple
>>>>> simultaneous eavesdroppers and the ability to include one extra person into
>>>>> the conversation.  Let's call it Leg C.  Then Leg B should be able to drop
>>>>> off leaving Leg A and Leg B talking.  Additionally I need the ability for
>>>>> Leg B and Leg C to talk to each other putting Leg A on hold.
>>>>>
>>>>> I considered using the eavesdrop tool for this which would give me the
>>>>> ability for the eavesdropper to talk to Leg A, Leg B or both.  However, I
>>>>> was not sure that having Leg A drop off would work and ultimately decided
>>>>> that the simplicity of implementing this with conferences  is worth the CPU
>>>>> hit.
>>>>>
>>>>> Again, the only reason I am concerned with the current additional CPU
>>>>> jump is because it's unexplained and I suspect it has something to do with
>>>>> the crash I experienced.
>>>>>
>>>>>
>>>>> On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Just one extra?
>>>>>>
>>>>>> https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
>>>>>>
>>>>>>
>>>>>> On 31 July 2014 15:04, Oleg Stolyar <olegstolyar at gmail.com> wrote:
>>>>>>
>>>>>>> That's a whole different discussion.  Mostly because I need the
>>>>>>> ability to easily add another person to the call.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <
>>>>>>> rtreleaven at bunnykick.ca> wrote:
>>>>>>>
>>>>>>>> Why do you have two person conferences instead of bridged calls?
>>>>>>>>
>>>>>>>> Sent from my BlackBerry® PlayBook™
>>>>>>>> www.blackberry.com
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> *From:* "Oleg Stolyar" <olegstolyar at gmail.com>
>>>>>>>> *To:* "FreeSWITCH Users Help" <
>>>>>>>> freeswitch-users at lists.freeswitch.org>
>>>>>>>> *Sent:* 31 July, 2014 9:40 AM
>>>>>>>> *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4
>>>>>>>> consume more CPU for conferences
>>>>>>>>
>>>>>>>> I am not sure we are on the same page with the numbers.  The
>>>>>>>> relative increase is about 30%-40%
>>>>>>>>
>>>>>>>> The 0.3% is the absolute increase per conference.  CPU usage per
>>>>>>>> single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
>>>>>>>>
>>>>>>>> I am using AWS m3.xlarge instances with CentOS.  It's a 4-core
>>>>>>>> system, so the increase in CPU of usage of 30% would reduce the number of
>>>>>>>> calls each instance can take.  The increase itself would not necessarily be
>>>>>>>> a huge deal except if it's unexplained, it may be an indicator that
>>>>>>>> something is wrong.  As I said in my original email, the system running the
>>>>>>>> recent code also crashed less than 24 hours after starting to take live
>>>>>>>> calls.  I wanted to call out the increase as it seems likely to me
>>>>>>>> (although by no means certain) that the two are related.
>>>>>>>>
>>>>>>>> The previous FreeSWITCH version I used have always been rock solid
>>>>>>>> - this was the first time FreeSWITCH crashed on me.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian at freeswitch.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> What kind of system are you running on where 0.2 to 0.3% CPU
>>>>>>>>> increase would really matter?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <
>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Of course, and it did when I first made the rate change.  It went
>>>>>>>>>> from 1% CPU to the 1.2%-1.3% in the master from May.  The problem I am
>>>>>>>>>> running into now is that there is another jump in CPU with no additional
>>>>>>>>>> config change.
>>>>>>>>>>
>>>>>>>>>> If nothing comes to mind, I will patch the May master with the
>>>>>>>>>> fix for WebRTC password size to fix my immediate issue and then try to find
>>>>>>>>>> the time to bisect git and pinpoint what happened.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <
>>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> It is very likely to add to the cpu to transcode everything to
>>>>>>>>>>> 12000hz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <
>>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Anthony, thank you for responding!
>>>>>>>>>>>>
>>>>>>>>>>>> I apologize for the previous email - I retracted it too soon -
>>>>>>>>>>>> the CPU simply jumped down for a bit then went back up.
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the full versions for both the master where it
>>>>>>>>>>>> happened and v1.4.  Is this enough?
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH Version
>>>>>>>>>>>> 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29
>>>>>>>>>>>> 13:22:47Z 64bit)
>>>>>>>>>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit
>>>>>>>>>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
>>>>>>>>>>>>
>>>>>>>>>>>> It does not occur on this master:
>>>>>>>>>>>> FreeSWITCH Version
>>>>>>>>>>>> 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22
>>>>>>>>>>>> 17:56:48Z 64bit)
>>>>>>>>>>>>
>>>>>>>>>>>> I figured I would try this email before bisecting git but it
>>>>>>>>>>>> will likely be my next step.
>>>>>>>>>>>>
>>>>>>>>>>>> On the subject of the 12000 rate related bug - I'd love to
>>>>>>>>>>>> discuss this more with you - perhaps I'll try to get a little bit of your
>>>>>>>>>>>> time at ClueCon to just show you the problem.  It actually happens on every
>>>>>>>>>>>> platform I tested it on except Debian 7.  Just in the last couple of days I
>>>>>>>>>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04.  I only mentioned it in
>>>>>>>>>>>> this thread in case the 12000 sample rate conference on 8000 codecs is
>>>>>>>>>>>> relevant to the CPU increase.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <
>>>>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not really sure how to pinpoint .8% cpu increase.  You
>>>>>>>>>>>>> would maybe have to git bisect?  I believe you already said this in another
>>>>>>>>>>>>> email then retracted it.  So with all the back and forth it starts to sound
>>>>>>>>>>>>> like you have misinformation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can't use vague references like "latest" which changes
>>>>>>>>>>>>> every 10 min.
>>>>>>>>>>>>> You need to try the most current master (TODAY then recite the
>>>>>>>>>>>>> hash) with a clean build and have a valid backtrace.
>>>>>>>>>>>>> The things you are saying are inconsistent so I think you need
>>>>>>>>>>>>> to rule out build skew from jumping around so many versions.
>>>>>>>>>>>>> Your 12000 thing also makes no sense at all and does not occur
>>>>>>>>>>>>> for anyone else which suggests you have environmental problem related to
>>>>>>>>>>>>> the way your system is setup.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <
>>>>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have my FreeSWITCH running a bunch of conferences with two
>>>>>>>>>>>>>> or three users using PCMU/8000 from all the user agents.  I have the
>>>>>>>>>>>>>> conference sample rate set to 12000 to get around a strange bug I reported
>>>>>>>>>>>>>> a couple of months ago.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In my old installation with master from May 22 2014, the
>>>>>>>>>>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the latest master AND in the latest v1.4 it went up to
>>>>>>>>>>>>>> about 1.6%-2.0% CPU per two-person conference.  CPU usage by the FreeSWITCH
>>>>>>>>>>>>>> process also jumps up and down more.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Additionally (not sure if it's related) the latest master
>>>>>>>>>>>>>> crashed less than 24 hours after it started taking live traffic.  I ma
>>>>>>>>>>>>>> trying to get the core dump and see if v1.4 also crashes but including the
>>>>>>>>>>>>>> info here in case it gives someone an idea of what's going on with the CPU.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any ideas as to what's causing the increased CPU usage would
>>>>>>>>>>>>>> be greatly appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ___
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>
>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>> http://twitter.com/FreeSWITCH
>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>> <http://freeswitch.org/g+>*
>>>
>>> ClueCon Weekly Development Call
>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>


-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140801/f96d42a0/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list