<div dir="ltr">It didn't explictly say 'pause'... what i noticed grep-ing the log for the uuid of a call's that had hung was that it was going through a sequence of log messages that are usually <1ms apart and instead were ~30s apart.<div>
<br></div><div>-Steve</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 20 November 2013 21:56, Andre Demattia <span dir="ltr"><<a href="mailto:andretodd@verizon.net" target="_blank">andretodd@verizon.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif">I had debug turned off but I'll enable it to see if anything interesting happens.<br>I didn't see anything that said pause.<br><br>I'll also turn off odbc for the core db and use the default one to see if that helps. Un testing it didn't help but I'll try it in again. </div>
</div>
<div dir="ltr">
<hr>
<span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif;FONT-WEIGHT:bold">From: </span><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif"><a href="mailto:steveayre@gmail.com" target="_blank">Steven Ayre</a></span><br>
<span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif;FONT-WEIGHT:bold">Sent: </span><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif">11/20/2013 4:36 PM</span><br><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif;FONT-WEIGHT:bold">To: </span><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif"><a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">FreeSWITCH Users Help</a></span><div class="im">
<br><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif;FONT-WEIGHT:bold">Subject: </span><span style="FONT-SIZE:11pt;FONT-FAMILY:Calibri,sans-serif">Re: [Freeswitch-users] Freeswitch Performance</span><br><br></div>
</div>><div class="im"><br>> show nothing other than FreeSWITCH was receiving the new messages but no<br>> bridge or cancel actions were taking place<br><br><br>Do you have your logs at debug level and are there any log messages that<br>
indicate a call reaching a point then pausing?<br><br>I saw something recently similar under load, and it turned out to be the<br>limit subsystem was waiting on DB queries that were severely backing up<br>because there were too many commits happening per second and InnoDB was<br>
configured to flush to disk on every transaction commit which can only<br>happen ~200 times per second. (Switching to the memory engine fixed the<br>error although isn't transactional).<br><br>-Steve<br><br>-Steve<br>
<br><br>On 20 November 2013 20:38, Andre <<a href="mailto:andretodd@verizon.net" target="_blank">andretodd@verizon.net</a>> wrote:<br><br>> I tested again and when I use 25 cps for 2 hours everything worked fine.<br>
> When I sent 50 cps I noticed FreeSWITCH worked for about 5 minutes then I<br>> saw the memory jump from 122 mb to to 1.6 gb in seconds. I think FreeSWITCH<br>> crashed but the logs show nothing other than FreeSWITCH was receiving the<br>
> new messages but no bridge or cancel actions were taking place.<br>><br>><br>><br>> How do I tell what happened that made freeswitch just stop working? The<br>> GUI didn’t show anything like a throw exception and when I type version or<br>
> sofia status in the FreeSWITCH gui it doesn’t respond.<br>><br>><br>><br>> Andre<br>><br>><br>><br></div>> *From:* <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<br>
> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>] *On Behalf Of *Anthony<br>> Minessale<br>> *Sent:* Wednesday, November 20, 2013 12:33 AM<br>
> *To:* Freeswitch-users<br>><br>> *Subject:* Re: [Freeswitch-users] Freeswitch Performance<div><div class="h5"><br>><br>><br>><br>> Get HEAD of stable branch or HEAD of master for latest build files.<br>
><br>> There is no promise windows can do 500cps the industry standard is 50 and<br>> you'll have probably more luck on linux but there is only on way to fimd<br>> out.<br>><br>> On Nov 19, 2013 11:27 PM, "I put the Who? in Mishehu" <<br>
> <a href="mailto:mishehu@freeswitch.org" target="_blank">mishehu@freeswitch.org</a>> wrote:<br>><br>> The amusing thing about the recurring load test threads that pop up every<br>> now and then is that the most important tips are already documented on the<br>
> wiki in one single page. Beyond that, you probably spend more time load<br>> testing than you do just tossing up another server and using something like<br>> kamailio or opensips to load balance...<br>><br>
><br>> --<br>><br>> Yossi Neiman<br>><br>><br>><br>><br>><br>> On 11/19/2013 06:48 PM, Antonio Teixeira wrote:<br>><br>> Hello Andre.<br>><br>> Solving performance problems is always hard , but a few points to keep in<br>
> mind :<br>><br>> To my knowledge FS is pretty much a linux software , it works on windows<br>> by that i doesn't means it performs on windows ( please correct me if i'm<br>> wrong)<br>> The native DB for FS is Postgresql ( this should make you oracle admin<br>
> more happy :) ).<br>><br>> Try this :<br>> Linux Server Realtime Kernel + Last FS Stable + Postgresql ( tune it well).<br>> Take care with ulimit on linux , its needed for the extra punch of<br>> performance<br>
> Check if your "performance test" is real if you are pointing to 1500CPS<br>> for a two minute call you will have 180k calls before the first call gets<br>> disconnected, if you are also testing RTP that means 180k * 2 ports needed<br>
> , The Kernel will start to complaining and you will run out of ports you<br>> will also run out of FD probably before you run out of ports on a not tuned<br>> linux install.<br>> If the DB is on the same server it will also stall since the FD will not<br>
> be present for new connections to the back end.<br>><br>> In terms of BW you will exceed 230000 kb/s add a few more kb/s for the DB<br>> ( Not a problem here i think)<br>><br>> Another thing , no sip server ( please include all the nice names here<br>
> like B2BUA , etc ) should ever queue anything.<br>> I have never read the code that handles this on FS but the way i would do<br>> it should be something like : can we sustain the load ? No in that case<br>> send 503 else proceed.<br>
> This is done in freeswitch by this var<br>> <a href="https://wiki.freeswitch.org/wiki/XML_Switch_Configuration#max-sessions" target="_blank">https://wiki.freeswitch.org/wiki/XML_Switch_Configuration#max-sessions</a><br>
><br>> A more general approach:<br>> In SIp Servers you will generally ( if not running cpu intensive tasks,<br>> trancoding , tone detection , etc ), consume all the ram , Network IO ,than<br>> the cpu.<br>
> For the guys that plan something big you should take a look at this :<br>> <a href="http://www.solarflare.com/High-Frequency-Trading" target="_blank">http://www.solarflare.com/High-Frequency-Trading</a><br>> Not by The NIC itself but because of this :<br>
> <a href="http://www.openonload.org" target="_blank">http://www.openonload.org</a><br>><br>> Without more data i don't think we can help , above all performance<br>> testing is a dark art you just need to keep on trying different things and<br>
> above all have some type of monitor tool to understand what exactly is<br>> holding you up , maybe its no FS...<br>><br>> Also 1500CPS is something seen in carrier operations generally with a<br>> named SBC in front , keep in mind that FS is a great software but it<br>
> doesn't hold against SBC's that use special processors like ASIC to perform<br>> transcoding , sip parsing , tone detection ,etc etc<br>><br>><br>> Disclaimer : post written deep into the night...<br>
><br>> On 20/11/13 00:09, Andre wrote:<br>><br>> Hi, I am testing out my FreeSWITCH 1.2.14 64 bit windows 8.1 on my<br>> Alienware Laptop with a very fast SSD and I7 processor with 8 GB DDR3 Ram.<br>><br>
><br>><br>> I have been using a pjsip test tool to test FreeSWITCH to see how it<br>> performs. Tell me if I’m wrong here, I believe if I sent “too many” calls<br>> to FreeSWITCH that it queues up the calls and sooner or later uses all the<br>
> memory on my PC then just stops working. I do have the CPS sent to 1500<br>> for testing but I never hit that high, I hit about 200 CPS.<br>><br>><br>><br>> The CDRs that save to my database show it takes about 8 seconds from start<br>
> time to end time. It does start out only taking milliseconds then it<br>> progresses to the 8 second mark.<br>><br>><br>><br>> The calls are just going to FreeSWITCH then I’m sending back a 503.<br>><br>
><br>><br>> I have every mod turned off, logging set to 0, Core db going to a SQL<br>> Server and I even set FreeSWITCH to real time.<br>><br>><br>><br>> Can anyone give me advise on what I can do to get better performance?<br>
> I’ve also tested it out on my real server but in about 3 minutes of 50 cps<br>> FreeSWITCH uses up 3gb of memory and stops responding.<br>><br>><br>><br>> I’m at a loss here on what else to do.<br>><br>
> Andre<br>><br>><br>><br>> _________________________________________________________________________<br>><br>> Professional FreeSWITCH Consulting Services:<br>><br>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
><br>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>><br>><br>><br>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>><br>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>><br>><br>> Official FreeSWITCH Sites<br>><br>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>><br>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
><br>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>><br>><br>><br>> FreeSWITCH-users mailing list<br>><br>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
><br>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>><br>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
><br>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>><br>><br>><br>> _________________________________________________________________________<br>><br>> Professional FreeSWITCH Consulting Services:<br>
><br>> <a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>><br>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>><br>><br>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>><br>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>><br>><br>><br>> Official FreeSWITCH Sites<br>
><br>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>><br>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>><br>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>><br>><br>> FreeSWITCH-users mailing list<br>><br>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>><br>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
<br>
</div></div><div>[The entire original message is not included.]</div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div>