[Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences

Thu Jul 31 22:22:36 MSD 2014

I actually think you need to start over.  Nothing you say is consistent and
I have a rule when doing debugging that when things don't add up, start
over.

Run latest GIT with clean build (remove all old binaries from bin lib and
mod in FS install target)

Start from there getting BT, evaluating CPU etc.  Jumping around versions
is the leading way to cause problems that cannot be explained.

On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar at gmail.com>
wrote:

> Yes, I enabled core dump and will get the backtrace if it crashes again.
>  I need to make preparations for the rest of my system to handle FreeSWITCH
> crashes better before I put the new code back in production, so it may take
> a while.
>
> If I find the time I will also try to bisect git to find the commit that
> caused this as Anthony suggested.
>
>
> On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike at jerris.com> wrote:
>
>> There is no reason at all to think it has anything to do with the crash.
>>  For the crash, simply get a backtrace of it so we can take a look at what
>> is going on with the crash.
>>
>> On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar at gmail.com> wrote:
>>
>> Well, it gets a little complicated.  I need to enable multiple
>> simultaneous eavesdroppers and the ability to include one extra person into
>> the conversation.  Let's call it Leg C.  Then Leg B should be able to drop
>> off leaving Leg A and Leg B talking.  Additionally I need the ability for
>> Leg B and Leg C to talk to each other putting Leg A on hold.
>>
>> I considered using the eavesdrop tool for this which would give me the
>> ability for the eavesdropper to talk to Leg A, Leg B or both.  However, I
>> was not sure that having Leg A drop off would work and ultimately decided
>> that the simplicity of implementing this with conferences  is worth the CPU
>> hit.
>>
>> Again, the only reason I am concerned with the current additional CPU
>> jump is because it's unexplained and I suspect it has something to do with
>> the crash I experienced.
>>
>>
>> On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre at gmail.com> wrote:
>>
>>> Just one extra?
>>>
>>> https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
>>>
>>>
>>> On 31 July 2014 15:04, Oleg Stolyar <olegstolyar at gmail.com> wrote:
>>>
>>>> That's a whole different discussion.  Mostly because I need the ability
>>>> to easily add another person to the call.
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <
>>>> rtreleaven at bunnykick.ca> wrote:
>>>>
>>>>> Why do you have two person conferences instead of bridged calls?
>>>>>
>>>>> Sent from my BlackBerry® PlayBook™
>>>>> www.blackberry.com
>>>>>
>>>>> ------------------------------
>>>>> *From:* "Oleg Stolyar" <olegstolyar at gmail.com>
>>>>> *To:* "FreeSWITCH Users Help" <freeswitch-users at lists.freeswitch.org>
>>>>> *Sent:* 31 July, 2014 9:40 AM
>>>>> *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4
>>>>> consume more CPU for conferences
>>>>>
>>>>> I am not sure we are on the same page with the numbers.  The relative
>>>>> increase is about 30%-40%
>>>>>
>>>>> The 0.3% is the absolute increase per conference.  CPU usage per
>>>>> single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
>>>>>
>>>>> I am using AWS m3.xlarge instances with CentOS.  It's a 4-core system,
>>>>> so the increase in CPU of usage of 30% would reduce the number of calls
>>>>> each instance can take.  The increase itself would not necessarily be a
>>>>> huge deal except if it's unexplained, it may be an indicator that something
>>>>> is wrong.  As I said in my original email, the system running the recent
>>>>> code also crashed less than 24 hours after starting to take live calls.  I
>>>>> wanted to call out the increase as it seems likely to me (although by no
>>>>> means certain) that the two are related.
>>>>>
>>>>> The previous FreeSWITCH version I used have always been rock solid -
>>>>> this was the first time FreeSWITCH crashed on me.
>>>>>
>>>>>
>>>>> On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian at freeswitch.org>
>>>>> wrote:
>>>>>
>>>>>> What kind of system are you running on where 0.2 to 0.3% CPU increase
>>>>>> would really matter?
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Of course, and it did when I first made the rate change.  It went
>>>>>>> from 1% CPU to the 1.2%-1.3% in the master from May.  The problem I am
>>>>>>> running into now is that there is another jump in CPU with no additional
>>>>>>> config change.
>>>>>>>
>>>>>>> If nothing comes to mind, I will patch the May master with the fix
>>>>>>> for WebRTC password size to fix my immediate issue and then try to find the
>>>>>>> time to bisect git and pinpoint what happened.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <
>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>
>>>>>>>> It is very likely to add to the cpu to transcode everything to
>>>>>>>> 12000hz
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <
>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Anthony, thank you for responding!
>>>>>>>>>
>>>>>>>>> I apologize for the previous email - I retracted it too soon - the
>>>>>>>>> CPU simply jumped down for a bit then went back up.
>>>>>>>>>
>>>>>>>>> Here are the full versions for both the master where it happened
>>>>>>>>> and v1.4.  Is this enough?
>>>>>>>>>
>>>>>>>>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit
>>>>>>>>> (git af64da8 2014-07-29 13:22:47Z 64bit)
>>>>>>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit
>>>>>>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
>>>>>>>>>
>>>>>>>>> It does not occur on this master:
>>>>>>>>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit
>>>>>>>>> (git 9d99c34 2014-05-22 17:56:48Z 64bit)
>>>>>>>>>
>>>>>>>>> I figured I would try this email before bisecting git but it will
>>>>>>>>> likely be my next step.
>>>>>>>>>
>>>>>>>>> On the subject of the 12000 rate related bug - I'd love to discuss
>>>>>>>>> this more with you - perhaps I'll try to get a little bit of your time at
>>>>>>>>> ClueCon to just show you the problem.  It actually happens on every
>>>>>>>>> platform I tested it on except Debian 7.  Just in the last couple of days I
>>>>>>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04.  I only mentioned it in
>>>>>>>>> this thread in case the 12000 sample rate conference on 8000 codecs is
>>>>>>>>> relevant to the CPU increase.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <
>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm not really sure how to pinpoint .8% cpu increase.  You would
>>>>>>>>>> maybe have to git bisect?  I believe you already said this in another email
>>>>>>>>>> then retracted it.  So with all the back and forth it starts to sound like
>>>>>>>>>> you have misinformation.
>>>>>>>>>>
>>>>>>>>>> You can't use vague references like "latest" which changes every
>>>>>>>>>> 10 min.
>>>>>>>>>> You need to try the most current master (TODAY then recite the
>>>>>>>>>> hash) with a clean build and have a valid backtrace.
>>>>>>>>>> The things you are saying are inconsistent so I think you need to
>>>>>>>>>> rule out build skew from jumping around so many versions.
>>>>>>>>>> Your 12000 thing also makes no sense at all and does not occur
>>>>>>>>>> for anyone else which suggests you have environmental problem related to
>>>>>>>>>> the way your system is setup.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <
>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> I have my FreeSWITCH running a bunch of conferences with two or
>>>>>>>>>>> three users using PCMU/8000 from all the user agents.  I have the
>>>>>>>>>>> conference sample rate set to 12000 to get around a strange bug I reported
>>>>>>>>>>> a couple of months ago.
>>>>>>>>>>>
>>>>>>>>>>> In my old installation with master from May 22 2014, the servers
>>>>>>>>>>> consumed about 1.2%-1.3% CPU per two-person conference.
>>>>>>>>>>>
>>>>>>>>>>> In the latest master AND in the latest v1.4 it went up to about
>>>>>>>>>>> 1.6%-2.0% CPU per two-person conference.  CPU usage by the FreeSWITCH
>>>>>>>>>>> process also jumps up and down more.
>>>>>>>>>>>
>>>>>>>>>>> Additionally (not sure if it's related) the latest master
>>>>>>>>>>> crashed less than 24 hours after it started taking live traffic.  I ma
>>>>>>>>>>> trying to get the core dump and see if v1.4 also crashes but including the
>>>>>>>>>>> info here in case it gives someone an idea of what's going on with the CPU.
>>>>>>>>>>>
>>>>>>>>>>> Any ideas as to what's causing the increased CPU usage would be
>>>>>>>>>>> greatly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> ___
>>>>>>>>>>
>>>>>>>>>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*

ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140731/99faf115/attachment-0001.html 