[Freeswitch-users] XML_CURL max connection limit

Cal Leeming [Simplicity Media Ltd] cal.leeming at simplicitymedialtd.co.uk
Wed Jun 12 19:17:54 MSD 2013 

Okay, so the next step is to identify what is happening in the code between
the 183 and the first RTP packet.. you could do this by either putting in
some manual print statements into the code and go for a trial and error, or
you could use a profiler or some sort..

I'd advice creating a JIRA ticket on this so that a core dev can assist
you, as this is now going outside of my area of expertise.

Cal

On Wed, Jun 12, 2013 at 3:57 PM, Muhammad Shahzad <shaheryarkh at gmail.com>wrote:

> Well, per our observation, we don't see any load or delay from xml curl or
> web server side. Also the delay is occurring exactly at one location that
> is after sending 183 Early Media and sending first RTP packet of IVR. When
> CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it
> goes beyond 3.10 seconds.
>
> Regarding New Relic, we don't have WAN access on server, since its a
> closed environment at telco data center. Also we see the behavior remains
> the same if we add multiple FS serve by same web server. So, i think
> problem is on FS side rather then web server side.
>
> Thank you.
>
>
>
>
> On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:
>
>> Curious, I'm wondering if maybe excessive context switching is causing
>> some issues.. I'm wondering if the latency is actually caused by the
>> overheads on mod_xml_curl.
>>
>> Could you try putting New Relic on your application, and see what sort of
>> response times you get? That will identify which queries are taking a while
>> to process.
>>
>> Cal
>>
>>
>> On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh at gmail.com>wrote:
>>
>>> We are already monitoring using top and htop. Even per core view of top
>>> shows no core every goes beyond 40%, average varies between 21 - 33%.
>>>
>>> Thank you.
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <
>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>
>>>> You could either throw Cacti on the server, or use server density/new
>>>> relic trial for quick and easy graphing.. or you can just watch `top` and
>>>> push number 1 whilst running to see the CPU breakdown.
>>>>
>>>> Cal
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <
>>>> shaheryarkh at gmail.com> wrote:
>>>>
>>>>> Steven, i am using soft timer. Should i try another? Please suggest.
>>>>>
>>>>> Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have
>>>>> 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can
>>>>> i enable it.
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <
>>>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>>>
>>>>>> This sounds quite interesting.. From my own benchmarks, I've seen
>>>>>> that IVRs applications can be quite stressful on the server.
>>>>>>
>>>>>> Have you checked to see if there is a CPU util spike or context
>>>>>> switching spike during these tests? Perhaps you are hitting the floor limit
>>>>>> of what your CPU is capable of?
>>>>>>
>>>>>> Do you have CPU graphs? If not, can you enable them and try again?
>>>>>>
>>>>>> Cal
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <
>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>
>>>>>>> You are right. After many improvements and modifications to dail
>>>>>>> plan, removing unnecessary modules and using Ngnix web server, things are
>>>>>>> much better now. Except that directory information is fetched 3 times per
>>>>>>> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for
>>>>>>> dial plan).
>>>>>>>
>>>>>>> Since Monday, we have started analyzing the results of load test,
>>>>>>> specially looking into packet traces, voicemail recordings and so on. We
>>>>>>> observed a strange thing, although we were able to test 1536 concurrent
>>>>>>> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls
>>>>>>> per second rate. But there is strange and unexplained delay in starting RTP
>>>>>>> from FS side.
>>>>>>>
>>>>>>> FS plays an IVR in early media state, then answers the call to
>>>>>>> directly start recording voicemail. The problem is if call rate is under 25
>>>>>>> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP
>>>>>>> in early media state. This delay is negligible after processing few hundred
>>>>>>> thousand calls but after about 1.2 million calls the delay has creeps up to
>>>>>>> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2
>>>>>>> second after around 400 calls had done. At 50 CPS its ~3 second and after
>>>>>>> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in
>>>>>>> starting playing IVR in early media state, once the IVR starts playing
>>>>>>> everything works perfectly fine. No call fails, but this delay would be
>>>>>>> annoying for anyone using the servers with over 30 CPS.
>>>>>>>
>>>>>>> Just to let you know, we checked everything, system resources,
>>>>>>> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested
>>>>>>> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing
>>>>>>> seems to have any impact on the results.
>>>>>>>
>>>>>>> I have run of out of possible explanations, so calling you guys for
>>>>>>> help.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike at jerris.com>wrote:
>>>>>>>
>>>>>>>> I would need to see the specifics of your dialplan, but I suspect
>>>>>>>> you have some transfers or execute extension that are causing it to
>>>>>>>> re-enter dialplan.  Are you sure on all the others that they are making the
>>>>>>>> exact same requests?  Details here will be necessary to understand exactly
>>>>>>>> what is going on..
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Guy, after detailed analysis i can confirm it was an Apache issue,
>>>>>>>> and changing Apache configuration changes the concurrent call limit.
>>>>>>>> However, one thing i observed about xml curl is that it makes several
>>>>>>>> request with same parameters per call. Even if we have no calls running on
>>>>>>>> server, as well as as Apache is completely free. I can see FS makes at least
>>>>>>>>
>>>>>>>> 1). 3 hits for dialplan.
>>>>>>>> 2). 2 hits for directory.
>>>>>>>> 3). 4 hits for voicemail.
>>>>>>>> 4). 3 hits for ivr.
>>>>>>>> 5). 2 hits for sofia.
>>>>>>>> and so on..
>>>>>>>>
>>>>>>>> While ideally only one hit should be sufficient to get all these
>>>>>>>> configurations. What is the point of making all these redundant hits? We
>>>>>>>> have roughly 15 - 20 hits made per call, which is huge and causes
>>>>>>>> significant load on Apache.
>>>>>>>>
>>>>>>>> Both Apache and FS are running on dedicated machines and connected
>>>>>>>> over Gigabit Ethernet with zero packet loss, so i doubt these hits are
>>>>>>>> "re-transmission".
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <
>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Yes, its a feature, i know. I was just looking at how this feature
>>>>>>>>> was tested, as mentioned in its logs,
>>>>>>>>>
>>>>>>>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408
>>>>>>>>> Processing 21653XXXXX->121637XXXXX in context termination
>>>>>>>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP
>>>>>>>>> error 0 trying to fetch http://192.168.0.2/gateway.php
>>>>>>>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML
>>>>>>>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike at jerris.com>wrote:
>>>>>>>>>
>>>>>>>>>> This is adding a feature to allow timeout to be set in ms instead
>>>>>>>>>> of seconds.  This is not a problem, its a feature, that was added a while
>>>>>>>>>> ago.  Until you can determine if the request is being sent for the one
>>>>>>>>>> thats failing, your not going to solve this issue.
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mit freundlichen Grüßen
>>>>>>> Muhammad Shahzad
>>>>>>> -----------------------------------
>>>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>>>> CISCO Certified Network Associate (CCNA)
>>>>>>> Cell: +49 176 99 83 10 85
>>>>>>> MSN: shari_786pk at hotmail.com
>>>>>>> Email: shaheryarkh at googlemail.com
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mit freundlichen Grüßen
>>>>> Muhammad Shahzad
>>>>> -----------------------------------
>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>> CISCO Certified Network Associate (CCNA)
>>>>> Cell: +49 176 99 83 10 85
>>>>> MSN: shari_786pk at hotmail.com
>>>>> Email: shaheryarkh at googlemail.com
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Mit freundlichen Grüßen
>>> Muhammad Shahzad
>>> -----------------------------------
>>> CISCO Rich Media Communication Specialist (CRMCS)
>>> CISCO Certified Network Associate (CCNA)
>>> Cell: +49 176 99 83 10 85
>>> MSN: shari_786pk at hotmail.com
>>> Email: shaheryarkh at googlemail.com
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Mit freundlichen Grüßen
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130612/b1d69fb0/attachment-0001.html

Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list