[Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences

Fri Aug 1 06:27:27 MSD 2014

Running a clean build is a great idea!  I will do that and report my
findings - it may be after ClueCon though.

On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:

> I actually think you need to start over.  Nothing you say is consistent
> and I have a rule when doing debugging that when things don't add up, start
> over.
>
> Run latest GIT with clean build (remove all old binaries from bin lib and
> mod in FS install target)
>
> Start from there getting BT, evaluating CPU etc.  Jumping around versions
> is the leading way to cause problems that cannot be explained.
>
>
>
> On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar at gmail.com>
> wrote:
>
>> Yes, I enabled core dump and will get the backtrace if it crashes again.
>>  I need to make preparations for the rest of my system to handle FreeSWITCH
>> crashes better before I put the new code back in production, so it may take
>> a while.
>>
>> If I find the time I will also try to bisect git to find the commit that
>> caused this as Anthony suggested.
>>
>>
>> On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike at jerris.com> wrote:
>>
>>> There is no reason at all to think it has anything to do with the crash.
>>>  For the crash, simply get a backtrace of it so we can take a look at what
>>> is going on with the crash.
>>>
>>> On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar at gmail.com>
>>> wrote:
>>>
>>> Well, it gets a little complicated.  I need to enable multiple
>>> simultaneous eavesdroppers and the ability to include one extra person into
>>> the conversation.  Let's call it Leg C.  Then Leg B should be able to drop
>>> off leaving Leg A and Leg B talking.  Additionally I need the ability for
>>> Leg B and Leg C to talk to each other putting Leg A on hold.
>>>
>>> I considered using the eavesdrop tool for this which would give me the
>>> ability for the eavesdropper to talk to Leg A, Leg B or both.  However, I
>>> was not sure that having Leg A drop off would work and ultimately decided
>>> that the simplicity of implementing this with conferences  is worth the CPU
>>> hit.
>>>
>>> Again, the only reason I am concerned with the current additional CPU
>>> jump is because it's unexplained and I suspect it has something to do with
>>> the crash I experienced.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre at gmail.com>
>>> wrote:
>>>
>>>> Just one extra?
>>>>
>>>> https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
>>>>
>>>>
>>>> On 31 July 2014 15:04, Oleg Stolyar <olegstolyar at gmail.com> wrote:
>>>>
>>>>> That's a whole different discussion.  Mostly because I need the
>>>>> ability to easily add another person to the call.
>>>>>
>>>>>
>>>>> On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <
>>>>> rtreleaven at bunnykick.ca> wrote:
>>>>>
>>>>>> Why do you have two person conferences instead of bridged calls?
>>>>>>
>>>>>> Sent from my BlackBerry® PlayBook™
>>>>>> www.blackberry.com
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* "Oleg Stolyar" <olegstolyar at gmail.com>
>>>>>> *To:* "FreeSWITCH Users Help" <freeswitch-users at lists.freeswitch.org>
>>>>>> *Sent:* 31 July, 2014 9:40 AM
>>>>>> *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4
>>>>>> consume more CPU for conferences
>>>>>>
>>>>>> I am not sure we are on the same page with the numbers.  The relative
>>>>>> increase is about 30%-40%
>>>>>>
>>>>>> The 0.3% is the absolute increase per conference.  CPU usage per
>>>>>> single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
>>>>>>
>>>>>> I am using AWS m3.xlarge instances with CentOS.  It's a 4-core
>>>>>> system, so the increase in CPU of usage of 30% would reduce the number of
>>>>>> calls each instance can take.  The increase itself would not necessarily be
>>>>>> a huge deal except if it's unexplained, it may be an indicator that
>>>>>> something is wrong.  As I said in my original email, the system running the
>>>>>> recent code also crashed less than 24 hours after starting to take live
>>>>>> calls.  I wanted to call out the increase as it seems likely to me
>>>>>> (although by no means certain) that the two are related.
>>>>>>
>>>>>> The previous FreeSWITCH version I used have always been rock solid -
>>>>>> this was the first time FreeSWITCH crashed on me.
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian at freeswitch.org>
>>>>>> wrote:
>>>>>>
>>>>>>> What kind of system are you running on where 0.2 to 0.3% CPU
>>>>>>> increase would really matter?
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <olegstolyar at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Of course, and it did when I first made the rate change.  It went
>>>>>>>> from 1% CPU to the 1.2%-1.3% in the master from May.  The problem I am
>>>>>>>> running into now is that there is another jump in CPU with no additional
>>>>>>>> config change.
>>>>>>>>
>>>>>>>> If nothing comes to mind, I will patch the May master with the fix
>>>>>>>> for WebRTC password size to fix my immediate issue and then try to find the
>>>>>>>> time to bisect git and pinpoint what happened.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <
>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> It is very likely to add to the cpu to transcode everything to
>>>>>>>>> 12000hz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <
>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Anthony, thank you for responding!
>>>>>>>>>>
>>>>>>>>>> I apologize for the previous email - I retracted it too soon -
>>>>>>>>>> the CPU simply jumped down for a bit then went back up.
>>>>>>>>>>
>>>>>>>>>> Here are the full versions for both the master where it happened
>>>>>>>>>> and v1.4.  Is this enough?
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit
>>>>>>>>>> (git af64da8 2014-07-29 13:22:47Z 64bit)
>>>>>>>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit
>>>>>>>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
>>>>>>>>>>
>>>>>>>>>> It does not occur on this master:
>>>>>>>>>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit
>>>>>>>>>> (git 9d99c34 2014-05-22 17:56:48Z 64bit)
>>>>>>>>>>
>>>>>>>>>> I figured I would try this email before bisecting git but it will
>>>>>>>>>> likely be my next step.
>>>>>>>>>>
>>>>>>>>>> On the subject of the 12000 rate related bug - I'd love to
>>>>>>>>>> discuss this more with you - perhaps I'll try to get a little bit of your
>>>>>>>>>> time at ClueCon to just show you the problem.  It actually happens on every
>>>>>>>>>> platform I tested it on except Debian 7.  Just in the last couple of days I
>>>>>>>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04.  I only mentioned it in
>>>>>>>>>> this thread in case the 12000 sample rate conference on 8000 codecs is
>>>>>>>>>> relevant to the CPU increase.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <
>>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm not really sure how to pinpoint .8% cpu increase.  You would
>>>>>>>>>>> maybe have to git bisect?  I believe you already said this in another email
>>>>>>>>>>> then retracted it.  So with all the back and forth it starts to sound like
>>>>>>>>>>> you have misinformation.
>>>>>>>>>>>
>>>>>>>>>>> You can't use vague references like "latest" which changes every
>>>>>>>>>>> 10 min.
>>>>>>>>>>> You need to try the most current master (TODAY then recite the
>>>>>>>>>>> hash) with a clean build and have a valid backtrace.
>>>>>>>>>>> The things you are saying are inconsistent so I think you need
>>>>>>>>>>> to rule out build skew from jumping around so many versions.
>>>>>>>>>>> Your 12000 thing also makes no sense at all and does not occur
>>>>>>>>>>> for anyone else which suggests you have environmental problem related to
>>>>>>>>>>> the way your system is setup.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <
>>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>
>>>>>>>>>>>> I have my FreeSWITCH running a bunch of conferences with two or
>>>>>>>>>>>> three users using PCMU/8000 from all the user agents.  I have the
>>>>>>>>>>>> conference sample rate set to 12000 to get around a strange bug I reported
>>>>>>>>>>>> a couple of months ago.
>>>>>>>>>>>>
>>>>>>>>>>>> In my old installation with master from May 22 2014, the
>>>>>>>>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference.
>>>>>>>>>>>>
>>>>>>>>>>>> In the latest master AND in the latest v1.4 it went up to about
>>>>>>>>>>>> 1.6%-2.0% CPU per two-person conference.  CPU usage by the FreeSWITCH
>>>>>>>>>>>> process also jumps up and down more.
>>>>>>>>>>>>
>>>>>>>>>>>> Additionally (not sure if it's related) the latest master
>>>>>>>>>>>> crashed less than 24 hours after it started taking live traffic.  I ma
>>>>>>>>>>>> trying to get the core dump and see if v1.4 also crashes but including the
>>>>>>>>>>>> info here in case it gives someone an idea of what's going on with the CPU.
>>>>>>>>>>>>
>>>>>>>>>>>> Any ideas as to what's causing the increased CPU usage would be
>>>>>>>>>>>> greatly appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> ___
>>>>>>>>>>>
>>>>>>>>>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140731/a8547a64/attachment-0001.html 