[Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences
Oleg Stolyar
olegstolyar at gmail.com
Sat Aug 2 00:18:27 MSD 2014
Talk about "inconsistent" (and embarrassing for me for accusing the new
version)
Now the older version that was rock solid for months crashed on me also.
This time I have the backtrace. It's here:
https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt.
Please let me know if you'd like me to file a Jira and attach it to it or
if you need this reproduced on the latest master before you can look into
it.
Really appreciate all the help and advice!
On Thu, Jul 31, 2014 at 7:27 PM, Oleg Stolyar <olegstolyar at gmail.com> wrote:
> Running a clean build is a great idea! I will do that and report my
> findings - it may be after ClueCon though.
>
>
> On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale <
> anthony.minessale at gmail.com> wrote:
>
>> I actually think you need to start over. Nothing you say is consistent
>> and I have a rule when doing debugging that when things don't add up, start
>> over.
>>
>> Run latest GIT with clean build (remove all old binaries from bin lib and
>> mod in FS install target)
>>
>> Start from there getting BT, evaluating CPU etc. Jumping around versions
>> is the leading way to cause problems that cannot be explained.
>>
>>
>>
>> On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <olegstolyar at gmail.com>
>> wrote:
>>
>>> Yes, I enabled core dump and will get the backtrace if it crashes again.
>>> I need to make preparations for the rest of my system to handle FreeSWITCH
>>> crashes better before I put the new code back in production, so it may take
>>> a while.
>>>
>>> If I find the time I will also try to bisect git to find the commit that
>>> caused this as Anthony suggested.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <mike at jerris.com> wrote:
>>>
>>>> There is no reason at all to think it has anything to do with the
>>>> crash. For the crash, simply get a backtrace of it so we can take a look
>>>> at what is going on with the crash.
>>>>
>>>> On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <olegstolyar at gmail.com>
>>>> wrote:
>>>>
>>>> Well, it gets a little complicated. I need to enable multiple
>>>> simultaneous eavesdroppers and the ability to include one extra person into
>>>> the conversation. Let's call it Leg C. Then Leg B should be able to drop
>>>> off leaving Leg A and Leg B talking. Additionally I need the ability for
>>>> Leg B and Leg C to talk to each other putting Leg A on hold.
>>>>
>>>> I considered using the eavesdrop tool for this which would give me the
>>>> ability for the eavesdropper to talk to Leg A, Leg B or both. However, I
>>>> was not sure that having Leg A drop off would work and ultimately decided
>>>> that the simplicity of implementing this with conferences is worth the CPU
>>>> hit.
>>>>
>>>> Again, the only reason I am concerned with the current additional CPU
>>>> jump is because it's unexplained and I suspect it has something to do with
>>>> the crash I experienced.
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <steveayre at gmail.com>
>>>> wrote:
>>>>
>>>>> Just one extra?
>>>>>
>>>>> https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way
>>>>>
>>>>>
>>>>> On 31 July 2014 15:04, Oleg Stolyar <olegstolyar at gmail.com> wrote:
>>>>>
>>>>>> That's a whole different discussion. Mostly because I need the
>>>>>> ability to easily add another person to the call.
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <
>>>>>> rtreleaven at bunnykick.ca> wrote:
>>>>>>
>>>>>>> Why do you have two person conferences instead of bridged calls?
>>>>>>>
>>>>>>> Sent from my BlackBerry® PlayBook™
>>>>>>> www.blackberry.com
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> *From:* "Oleg Stolyar" <olegstolyar at gmail.com>
>>>>>>> *To:* "FreeSWITCH Users Help" <freeswitch-users at lists.freeswitch.org
>>>>>>> >
>>>>>>> *Sent:* 31 July, 2014 9:40 AM
>>>>>>> *Subject:* Re: [Freeswitch-users] Latest master and latest v1.4
>>>>>>> consume more CPU for conferences
>>>>>>>
>>>>>>> I am not sure we are on the same page with the numbers. The
>>>>>>> relative increase is about 30%-40%
>>>>>>>
>>>>>>> The 0.3% is the absolute increase per conference. CPU usage per
>>>>>>> single two-member conference went from 1.2%-1.3% to 1.6%-2.0%
>>>>>>>
>>>>>>> I am using AWS m3.xlarge instances with CentOS. It's a 4-core
>>>>>>> system, so the increase in CPU of usage of 30% would reduce the number of
>>>>>>> calls each instance can take. The increase itself would not necessarily be
>>>>>>> a huge deal except if it's unexplained, it may be an indicator that
>>>>>>> something is wrong. As I said in my original email, the system running the
>>>>>>> recent code also crashed less than 24 hours after starting to take live
>>>>>>> calls. I wanted to call out the increase as it seems likely to me
>>>>>>> (although by no means certain) that the two are related.
>>>>>>>
>>>>>>> The previous FreeSWITCH version I used have always been rock solid -
>>>>>>> this was the first time FreeSWITCH crashed on me.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 31, 2014 at 6:21 AM, Brian West <brian at freeswitch.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> What kind of system are you running on where 0.2 to 0.3% CPU
>>>>>>>> increase would really matter?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <
>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Of course, and it did when I first made the rate change. It went
>>>>>>>>> from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am
>>>>>>>>> running into now is that there is another jump in CPU with no additional
>>>>>>>>> config change.
>>>>>>>>>
>>>>>>>>> If nothing comes to mind, I will patch the May master with the fix
>>>>>>>>> for WebRTC password size to fix my immediate issue and then try to find the
>>>>>>>>> time to bisect git and pinpoint what happened.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <
>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> It is very likely to add to the cpu to transcode everything to
>>>>>>>>>> 12000hz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <
>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Anthony, thank you for responding!
>>>>>>>>>>>
>>>>>>>>>>> I apologize for the previous email - I retracted it too soon -
>>>>>>>>>>> the CPU simply jumped down for a bit then went back up.
>>>>>>>>>>>
>>>>>>>>>>> Here are the full versions for both the master where it happened
>>>>>>>>>>> and v1.4. Is this enough?
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit
>>>>>>>>>>> (git af64da8 2014-07-29 13:22:47Z 64bit)
>>>>>>>>>>> FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit
>>>>>>>>>>> (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)
>>>>>>>>>>>
>>>>>>>>>>> It does not occur on this master:
>>>>>>>>>>> FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit
>>>>>>>>>>> (git 9d99c34 2014-05-22 17:56:48Z 64bit)
>>>>>>>>>>>
>>>>>>>>>>> I figured I would try this email before bisecting git but it
>>>>>>>>>>> will likely be my next step.
>>>>>>>>>>>
>>>>>>>>>>> On the subject of the 12000 rate related bug - I'd love to
>>>>>>>>>>> discuss this more with you - perhaps I'll try to get a little bit of your
>>>>>>>>>>> time at ClueCon to just show you the problem. It actually happens on every
>>>>>>>>>>> platform I tested it on except Debian 7. Just in the last couple of days I
>>>>>>>>>>> reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in
>>>>>>>>>>> this thread in case the 12000 sample rate conference on 8000 codecs is
>>>>>>>>>>> relevant to the CPU increase.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <
>>>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'm not really sure how to pinpoint .8% cpu increase. You
>>>>>>>>>>>> would maybe have to git bisect? I believe you already said this in another
>>>>>>>>>>>> email then retracted it. So with all the back and forth it starts to sound
>>>>>>>>>>>> like you have misinformation.
>>>>>>>>>>>>
>>>>>>>>>>>> You can't use vague references like "latest" which changes
>>>>>>>>>>>> every 10 min.
>>>>>>>>>>>> You need to try the most current master (TODAY then recite the
>>>>>>>>>>>> hash) with a clean build and have a valid backtrace.
>>>>>>>>>>>> The things you are saying are inconsistent so I think you need
>>>>>>>>>>>> to rule out build skew from jumping around so many versions.
>>>>>>>>>>>> Your 12000 thing also makes no sense at all and does not occur
>>>>>>>>>>>> for anyone else which suggests you have environmental problem related to
>>>>>>>>>>>> the way your system is setup.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <
>>>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have my FreeSWITCH running a bunch of conferences with two
>>>>>>>>>>>>> or three users using PCMU/8000 from all the user agents. I have the
>>>>>>>>>>>>> conference sample rate set to 12000 to get around a strange bug I reported
>>>>>>>>>>>>> a couple of months ago.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In my old installation with master from May 22 2014, the
>>>>>>>>>>>>> servers consumed about 1.2%-1.3% CPU per two-person conference.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the latest master AND in the latest v1.4 it went up to
>>>>>>>>>>>>> about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH
>>>>>>>>>>>>> process also jumps up and down more.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Additionally (not sure if it's related) the latest master
>>>>>>>>>>>>> crashed less than 24 hours after it started taking live traffic. I ma
>>>>>>>>>>>>> trying to get the core dump and see if v1.4 also crashes but including the
>>>>>>>>>>>>> info here in case it gives someone an idea of what's going on with the CPU.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas as to what's causing the increased CPU usage would
>>>>>>>>>>>>> be greatly appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ___
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>
>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>> http://twitter.com/FreeSWITCH
>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>> <http://freeswitch.org/g+>*
>>
>> ClueCon Weekly Development Call
>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140801/eb7bddcc/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list