[Freeswitch-users] XML_CURL max connection limit

Wed Jun 12 21:25:25 MSD 2013

Apologies - I was referencing some other discussions of heated debate of
windows, but I ended up confusing the conversion! Sorry about that guys!

Cal

On Wed, Jun 12, 2013 at 5:54 PM, Jeff Leung <jleung at v10networks.ca> wrote:

> I don’t think Dave even mentioned anything about Windows his rant.****
>
> ** **
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Cal Leeming
> [Simplicity Media Ltd]
> *Sent:* Wednesday, June 12, 2013 9:39 AM
> *To:* FreeSWITCH Users Help
>
> *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit****
>
> ** **
>
> Dave, with all due respect, every time the discussion of mod_xml_curl or
> anything non-windows comes up you immediately throw in a negative note, and
> it's starting to get a bit old now. Until one of the core devs steps up and
> says "you shouldn't be using mod_xml_curl" then I don't think it's right to
> be telling users that this is the wrong approach to be using - especially
> when we don't even know if it's mod_xml_curl that is causing the problem in
> this particular case. As mentioned in the last email, the next steps for OP
> is to identify the root cause of these delays, either via profiling or
> print statements.****
>
> ** **
>
> OP - I think at this point, you really need to take this over to a JIRA
> ticket to avoid spamming the mailing list, as the thread is starting to go
> a bit stale.****
>
> ** **
>
> Once you've created a ticket, let us know in this thread and we'll see
> about getting a core dev to give it some eyes.****
>
> ** **
>
> If this is an urgent matter, you could also pay for a few hours of FS
> support which will get the immediate attention of a core dev :)****
>
> ** **
>
> Cal****
>
> ** **
>
> On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk at drkngs.net> wrote:***
> *
>
> Ok, this "square peg in rouond hole" thread is not getting anything
> solved. You all need to take a step back, and look at the overall diesign
> of the applcation you are trying to build from a top down approch.****
>
>  ****
>
> First define what the application has to do, with all of it's
> requirements, including load, resonse times, what information you need to
> look up per call, and so on...****
>
>  ****
>
> This seems to me like a backwards approch was taken, as the start of the
> project, mod_xml_curl was the starting point, and then the application was
> designed after the method was picked. Isn't this "bass awards"? I know if I
> wasted my clients time and money by picking tech. before defining the
> problem, I would have no clients!****
>
>  ****
>
> One thing I can tell you for sure, is that using a design that involves
> both client and server side synchronous technologies as the starting point,
> isn’t going to scale out to carrier rate call handling. You can try
> throwing all sorts of stuff like different web servers, caching proxies in
> the middle, and a handful of other kludges at it, and all you’re going to
> get is a big “Cluster F…”, that will be a nightmare to maintain and scale.
> You may also become a slave to maintaining it, and monitoring your
> production environment.****
>
> One other thing to remember about FreeSWITCH, is that mod_xml_curl is not
> the only way of doing XML config binding to code. There are many modules
> that use the callback to implement feeding FS with real time XML config on
> demand. ****
>
> It may be time to re-design your application, rather then trying to build
> a “better mousetrap” on the external side. We could discuss this on the
> conference call today, and get input from all the folks on a design, given
> your requirements****
>
> --Dave****
>
>  ****
> ------------------------------
>
> *From:* Muhammad Shahzad [mailto:shaheryarkh at gmail.com]
> *To:* FreeSWITCH Users Help [mailto:freeswitch-users at lists.freeswitch.org]
> *Sent:* Wed, 12 Jun 2013 07:57:18 -0700
> *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit****
>
> Well, per our observation, we don't see any load or delay from xml curl or
> web server side. Also the delay is occurring exactly at one location that
> is after sending 183 Early Media and sending first RTP packet of IVR. When
> CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it
> goes beyond 3.10 seconds. ****
>
> ** **
>
> Regarding New Relic, we don't have WAN access on server, since its a
> closed environment at telco data center. Also we see the behavior remains
> the same if we add multiple FS serve by same web server. So, i think
> problem is on FS side rather then web server side.****
>
> ** **
>
> Thank you.****
>
> ** **
>
> ** **
>
> ** **
>
> On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:****
>
> Curious, I'm wondering if maybe excessive context switching is causing
> some issues.. I'm wondering if the latency is actually caused by the
> overheads on mod_xml_curl. ****
>
> ** **
>
> Could you try putting New Relic on your application, and see what sort of
> response times you get? That will identify which queries are taking a while
> to process.****
>
>
> Cal ****
>
> ** **
>
> On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh at gmail.com>
> wrote:****
>
> We are already monitoring using top and htop. Even per core view of top
> shows no core every goes beyond 40%, average varies between 21 - 33%.****
>
> ** **
>
> Thank you.****
>
> ** **
>
> On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:****
>
> You could either throw Cacti on the server, or use server density/new
> relic trial for quick and easy graphing.. or you can just watch `top` and
> push number 1 whilst running to see the CPU breakdown. ****
>
>
> Cal ****
>
> ** **
>
> On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh at gmail.com>
> wrote:****
>
> Steven, i am using soft timer. Should i try another? Please suggest. ****
>
> ** **
>
> Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24
> CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i
> enable it.****
>
> ** **
>
> Thank you.****
>
> ** **
>
> ** **
>
> On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:****
>
> This sounds quite interesting.. From my own benchmarks, I've seen that
> IVRs applications can be quite stressful on the server. ****
>
> ** **
>
> Have you checked to see if there is a CPU util spike or context switching
> spike during these tests? Perhaps you are hitting the floor limit of what
> your CPU is capable of?****
>
> ** **
>
> Do you have CPU graphs? If not, can you enable them and try again?****
>
> ** **
>
> Cal ****
>
> ** **
>
> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh at gmail.com>
> wrote:****
>
> You are right. After many improvements and modifications to dail plan,
> removing unnecessary modules and using Ngnix web server, things are much
> better now. Except that directory information is fetched 3 times per call
> and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial
> plan). ****
>
> ** **
>
> Since Monday, we have started analyzing the results of load test,
> specially looking into packet traces, voicemail recordings and so on. We
> observed a strange thing, although we were able to test 1536 concurrent
> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls
> per second rate. But there is strange and unexplained delay in starting RTP
> from FS side.****
>
> ** **
>
> FS plays an IVR in early media state, then answers the call to directly
> start recording voicemail. The problem is if call rate is under 25 CPS,
> everything works fine. At 25 CPS there is minor delay in starting RTP in
> early media state. This delay is negligible after processing few hundred
> thousand calls but after about 1.2 million calls the delay has creeps up to
> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2
> second after around 400 calls had done. At 50 CPS its ~3 second and after
> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in
> starting playing IVR in early media state, once the IVR starts playing
> everything works perfectly fine. No call fails, but this delay would be
> annoying for anyone using the servers with over 30 CPS.****
>
> ** **
>
> Just to let you know, we checked everything, system resources, bandwidth
> usage, IVR file formats etc. all seems to be fine. We even tested with
> various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems
> to have any impact on the results.****
>
> ** **
>
> I have run of out of possible explanations, so calling you guys for help.*
> ***
>
> ** **
>
> Thank you.****
>
> ** **
>
> ** **
>
> ** **
>
> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike at jerris.com> wrote:***
> *
>
> I would need to see the specifics of your dialplan, but I suspect you have
> some transfers or execute extension that are causing it to re-enter
> dialplan.  Are you sure on all the others that they are making the exact
> same requests?  Details here will be necessary to understand exactly what
> is going on.. ****
>
> ** **
>
> Mike****
>
> ** **
>
> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh at gmail.com>
> wrote:****
>
>
>
> ****
>
> Guy, after detailed analysis i can confirm it was an Apache issue, and
> changing Apache configuration changes the concurrent call limit. However,
> one thing i observed about xml curl is that it makes several request with
> same parameters per call. Even if we have no calls running on server, as
> well as as Apache is completely free. I can see FS makes at least ****
>
> ** **
>
> 1). 3 hits for dialplan.****
>
> 2). 2 hits for directory.****
>
> 3). 4 hits for voicemail.****
>
> 4). 3 hits for ivr.****
>
> 5). 2 hits for sofia.****
>
> and so on..****
>
> ** **
>
> While ideally only one hit should be sufficient to get all these
> configurations. What is the point of making all these redundant hits? We
> have roughly 15 - 20 hits made per call, which is huge and causes
> significant load on Apache.****
>
> ** **
>
> Both Apache and FS are running on dedicated machines and connected over
> Gigabit Ethernet with zero packet loss, so i doubt these hits are
> "re-transmission".****
>
> ** **
>
> Thank you.****
>
> ** **
>
> ** **
>
> ** **
>
> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh at gmail.com>
> wrote:****
>
> Yes, its a feature, i know. I was just looking at how this feature was
> tested, as mentioned in its logs, ****
>
> ** **
>
> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing
> 21653XXXXX->121637XXXXX in context termination
> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0
> trying to fetch http://192.168.0.2/gateway.php
> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in
> /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml****
>
> ** **
>
> Thank you.****
>
> ** **
>
> ** **
>
> ** **
>
> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike at jerris.com> wrote:**
> **
>
> This is adding a feature to allow timeout to be set in ms instead of
> seconds.  This is not a problem, its a feature, that was added a while ago.
>  Until you can determine if the request is being sent for the one thats
> failing, your not going to solve this issue. ****
>
> ** **
>
> Mike****
>
> ** **
>
> ** **
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
>
>
> ****
>
> ** **
>
> -- ****
>
> Mit freundlichen Grüßen****
>
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com ****
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> ** **
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
>
>
> ****
>
> ** **
>
> -- ****
>
> Mit freundlichen Grüßen****
>
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com ****
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> ** **
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
>
>
> ****
>
> ** **
>
> -- ****
>
> Mit freundlichen Grüßen****
>
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com ****
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> ** **
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
>
>
> ****
>
> ** **
>
> -- ****
>
> Mit freundlichen Grüßen****
>
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com ****
>
>  ****
>
>  ****
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org****
>
> ** **
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130612/c41fbdc6/attachment-0001.html 