<div dir="ltr">Its almost certain that you should do a fresh clean build. Once you start moving around in versions you can cause lots of problems with inconsistent binaries etc. </div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Aug 1, 2014 at 3:18 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>Talk about "inconsistent" (and embarrassing for me for accusing the new version)<br></div><div><br></div><div>Now the older version that was rock solid for months crashed on me also. This time I have the backtrace. It's here: <a href="https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt" target="_blank">https://www.dropbox.com/s/y0u48n61rul0x30/20140801T200335Z-bt.txt</a>. </div>
<div><br></div><div>Please let me know if you'd like me to file a Jira and attach it to it or if you need this reproduced on the latest master before you can look into it.</div><div><br></div><div>Really appreciate all the help and advice!</div>
</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 31, 2014 at 7:27 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Running a clean build is a great idea! I will do that and report my findings - it may be after ClueCon though.</div>
<div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 31, 2014 at 11:22 AM, Anthony Minessale <span dir="ltr"><<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I actually think you need to start over. Nothing you say is consistent and I have a rule when doing debugging that when things don't add up, start over.<div>
<br></div><div>Run latest GIT with clean build (remove all old binaries from bin lib and mod in FS install target)</div>
<div><br></div><div>Start from there getting BT, evaluating CPU etc. Jumping around versions is the leading way to cause problems that cannot be explained.</div><div><br></div></div><div class="gmail_extra"><div><div>
<br><br><div class="gmail_quote">
On Thu, Jul 31, 2014 at 12:06 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Yes, I enabled core dump and will get the backtrace if it crashes again. I need to make preparations for the rest of my system to handle FreeSWITCH crashes better before I put the new code back in production, so it may take a while.<div>
<br></div><div>If I find the time I will also try to bisect git to find the commit that caused this as Anthony suggested.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Thu, Jul 31, 2014 at 9:54 AM, Michael Jerris <span dir="ltr"><<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div style="word-wrap:break-word">There is no reason at all to think it has anything to do with the crash. For the crash, simply get a backtrace of it so we can take a look at what is going on with the crash.<div>
<div><div><br><div><div>On Jul 31, 2014, at 11:54 AM, Oleg Stolyar <<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">
Well, it gets a little complicated. I need to enable multiple simultaneous eavesdroppers and the ability to include one extra person into the conversation. Let's call it Leg C. Then Leg B should be able to drop off leaving Leg A and Leg B talking. Additionally I need the ability for Leg B and Leg C to talk to each other putting Leg A on hold.<div>
<br></div><div>I considered using the eavesdrop tool for this which would give me the ability for the eavesdropper to talk to Leg A, Leg B or both. However, I was not sure that having Leg A drop off would work and ultimately decided that the simplicity of implementing this with conferences is worth the CPU hit.</div>
<div><br></div><div>Again, the only reason I am concerned with the current additional CPU jump is because it's unexplained and I suspect it has something to do with the crash I experienced.</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Thu, Jul 31, 2014 at 8:35 AM, Steven Ayre <span dir="ltr"><<a href="mailto:steveayre@gmail.com" target="_blank">steveayre@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Just one extra?<div><br></div><div><a href="https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way" target="_blank">https://wiki.freeswitch.org/wiki/Misc._Dialplan_Tools_three_way</a><br></div></div>
<div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 31 July 2014 15:04, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">That's a whole different discussion. Mostly because I need the ability to easily add another person to the call.</div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Jul 31, 2014 at 6:48 AM, Russell Treleaven <span dir="ltr"><<a href="mailto:rtreleaven@bunnykick.ca" target="_blank">rtreleaven@bunnykick.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>Why do you have two person conferences instead of bridged calls? <br>
<br><div>Sent from my BlackBerry® PlayBook™<br>
<a href="http://www.blackberry.com/" target="_blank">www.blackberry.com</a></div>
<br><hr><div><strong>From:</strong> "Oleg Stolyar" <<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>><br><strong>To:</strong> "FreeSWITCH Users Help" <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
<strong>Sent:</strong> 31 July, 2014 9:40 AM<br><strong>Subject:</strong> Re: [Freeswitch-users] Latest master and latest v1.4 consume more CPU for conferences<br></div><div><br><div dir="ltr"><div>I am not sure we are on the same page with the numbers. The relative increase is about 30%-40%</div>
<div><br></div><div>The 0.3% is the absolute increase per conference. CPU usage per single two-member conference went from 1.2%-1.3% to 1.6%-2.0%</div>
<div><br></div>I am using AWS m3.xlarge instances with CentOS. It's a 4-core system, so the increase in CPU of usage of 30% would reduce the number of calls each instance can take. The increase itself would not necessarily be a huge deal except if it's unexplained, it may be an indicator that something is wrong. As I said in my original email, the system running the recent code also crashed less than 24 hours after starting to take live calls. I wanted to call out the increase as it seems likely to me (although by no means certain) that the two are related. <div>
<br></div><div>The previous FreeSWITCH version I used have always been rock solid - this was the first time FreeSWITCH crashed on me.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 31, 2014 at 6:21 AM, Brian West <span dir="ltr"><<a href="mailto:brian@freeswitch.org" target="_blank">brian@freeswitch.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">What kind of system are you running on where 0.2 to 0.3% CPU increase would really matter?</div>
<div class="gmail_extra">
<div><br><br><div class="gmail_quote">On Wed, Jul 30, 2014 at 9:39 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Of course, and it did when I first made the rate change. It went from 1% CPU to the 1.2%-1.3% in the master from May. The problem I am running into now is that there is another jump in CPU with no additional config change.<div>
<br></div><div>If nothing comes to mind, I will patch the May master with the fix for WebRTC password size to fix my immediate issue and then try to find the time to bisect git and pinpoint what happened.</div></div><div>
<div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jul 30, 2014 at 7:26 PM, Anthony Minessale <span dir="ltr"><<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">It is very likely to add to the cpu to transcode everything to 12000hz<div><br></div></div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 30, 2014 at 6:12 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Anthony, thank you for responding!<div>
<br></div><div>I apologize for the previous email - I retracted it too soon - the CPU simply jumped down for a bit then went back up.</div>
<div><br></div><div>Here are the full versions for both the master where it happened and v1.4. Is this enough?</div>
<div><br></div><div>FreeSWITCH Version 1.5.14b+git~20140729T132247Z~af64da86f2~64bit (git af64da8 2014-07-29 13:22:47Z 64bit)<br></div><div>FreeSWITCH Version 1.4.7+git~20140724T230506Z~9d1c0f7f3d~64bit (git 9d1c0f7 2014-07-24 23:05:06Z 64bit)<br>
</div><div><br></div><div>It does not occur on this master:</div><div>FreeSWITCH Version 1.5.13b+git~20140522T175648Z~9d99c34682~64bit (git 9d99c34 2014-05-22 17:56:48Z 64bit)</div><div><br></div><div>I figured I would try this email before bisecting git but it will likely be my next step.</div>
<div><br></div><div>On the subject of the 12000 rate related bug - I'd love to discuss this more with you - perhaps I'll try to get a little bit of your time at ClueCon to just show you the problem. It actually happens on every platform I tested it on except Debian 7. Just in the last couple of days I reproduced it on Ubuntu 12.04 and Ubuntu 14.04. I only mentioned it in this thread in case the 12000 sample rate conference on 8000 codecs is relevant to the CPU increase.</div>
<div><br></div></div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 30, 2014 at 3:46 PM, Anthony Minessale <span dir="ltr"><<a href="mailto:anthony.minessale@gmail.com" target="_blank">anthony.minessale@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>I'm not really sure how to pinpoint .8% cpu increase. You would maybe have to git bisect? I believe you already said this in another email then retracted it. So with all the back and forth it starts to sound like you have misinformation.</div>
<div><br></div>You can't use vague references like "latest" which changes every 10 min. <div>You need to try the most current master (TODAY then recite the hash) with a clean build and have a valid backtrace.</div>
<div>The things you are saying are inconsistent so I think you need to rule out build skew from jumping around so many versions.</div><div>Your 12000 thing also makes no sense at all and does not occur for anyone else which suggests you have environmental problem related to the way your system is setup.</div>
<div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Wed, Jul 30, 2014 at 5:31 PM, Oleg Stolyar <span dir="ltr"><<a href="mailto:olegstolyar@gmail.com" target="_blank">olegstolyar@gmail.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div dir="ltr">Hi guys,<div><br></div><div>I have my FreeSWITCH running a bunch of conferences with two or three users using PCMU/8000 from all the user agents. I have the conference sample rate set to 12000 to get around a strange bug I reported a couple of months ago.</div>
<div><br></div><div>In my old installation with master from May 22 2014, the servers consumed about 1.2%-1.3% CPU per two-person conference.</div><div><br></div><div>In the latest master AND in the latest v1.4 it went up to about 1.6%-2.0% CPU per two-person conference. CPU usage by the FreeSWITCH process also jumps up and down more.</div>
<div><br></div><div>Additionally (not sure if it's related) the latest master crashed less than 24 hours after it started taking live traffic. I ma trying to get the core dump and see if v1.4 also crashes but including the info here in case it gives someone an idea of what's going on with the CPU.</div>
<div><br></div><div>Any ideas as to what's causing the increased CPU usage would be greatly appreciated.</div></div>
<br></div>___</blockquote></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div>
</div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div><br></div></div><div>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></div></blockquote></div><br></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><div>-- <br><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div>
<div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br>
</div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ <a href="tel:%2B19193869900" value="+19193869900" target="_blank">+19193869900</a> </div><div><br>
</div></div></div></div>
</div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬<div><br><div>☞ <a href="http://freeswitch.org/" target="_blank">http://freeswitch.org/</a> ☞ <a href="http://cluecon.com/" target="_blank">http://cluecon.com/</a> ☞ <a href="http://twitter.com/FreeSWITCH" target="_blank">http://twitter.com/FreeSWITCH</a></div>
<div><div>☞ <a href="http://irc.freenode.net" target="_blank">irc.freenode.net</a> #freeswitch ☞ <u><a href="http://freeswitch.org/g+" target="_blank">http://freeswitch.org/g+</a></u><br><br></div><div>ClueCon Weekly Development Call <br>
</div><div>☎ <a href="mailto:sip%3A888@conference.freeswitch.org" target="_blank">sip:888@conference.freeswitch.org</a> ☎ +19193869900 </div><div><br></div></div></div></div>
</div>