[Freeswitch-users] XML_CURL max connection limit

Wed Jun 12 19:34:47 MSD 2013

OK. I am already doing trial and error testing on using rtp related
variables. I noticed one strange warning upon pre_answer,

2013-06-12 20:30:01.143911 [WARNING] switch_core_media.c:1282 Asynchronous
PTIME not supported, changing our end from 0 to 30

Any ideas about it? Could this be cause of all trouble? Just wondering..

Thank you.

On Wed, Jun 12, 2013 at 5:17 PM, Cal Leeming [Simplicity Media Ltd] <
cal.leeming at simplicitymedialtd.co.uk> wrote:

> Okay, so the next step is to identify what is happening in the code
> between the 183 and the first RTP packet.. you could do this by either
> putting in some manual print statements into the code and go for a trial
> and error, or you could use a profiler or some sort..
>
> I'd advice creating a JIRA ticket on this so that a core dev can assist
> you, as this is now going outside of my area of expertise.
>
> Cal
>
>
> On Wed, Jun 12, 2013 at 3:57 PM, Muhammad Shahzad <shaheryarkh at gmail.com>wrote:
>
>> Well, per our observation, we don't see any load or delay from xml curl
>> or web server side. Also the delay is occurring exactly at one location
>> that is after sending 183 Early Media and sending first RTP packet of IVR.
>> When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS
>> it goes beyond 3.10 seconds.
>>
>> Regarding New Relic, we don't have WAN access on server, since its a
>> closed environment at telco data center. Also we see the behavior remains
>> the same if we add multiple FS serve by same web server. So, i think
>> problem is on FS side rather then web server side.
>>
>> Thank you.
>>
>>
>>
>>
>> On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <
>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>
>>> Curious, I'm wondering if maybe excessive context switching is causing
>>> some issues.. I'm wondering if the latency is actually caused by the
>>> overheads on mod_xml_curl.
>>>
>>> Could you try putting New Relic on your application, and see what sort
>>> of response times you get? That will identify which queries are taking a
>>> while to process.
>>>
>>> Cal
>>>
>>>
>>> On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh at gmail.com
>>> > wrote:
>>>
>>>> We are already monitoring using top and htop. Even per core view of top
>>>> shows no core every goes beyond 40%, average varies between 21 - 33%.
>>>>
>>>> Thank you.
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <
>>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>>
>>>>> You could either throw Cacti on the server, or use server density/new
>>>>> relic trial for quick and easy graphing.. or you can just watch `top` and
>>>>> push number 1 whilst running to see the CPU breakdown.
>>>>>
>>>>> Cal
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <
>>>>> shaheryarkh at gmail.com> wrote:
>>>>>
>>>>>> Steven, i am using soft timer. Should i try another? Please suggest.
>>>>>>
>>>>>> Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have
>>>>>> 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can
>>>>>> i enable it.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <
>>>>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>>>>
>>>>>>> This sounds quite interesting.. From my own benchmarks, I've seen
>>>>>>> that IVRs applications can be quite stressful on the server.
>>>>>>>
>>>>>>> Have you checked to see if there is a CPU util spike or context
>>>>>>> switching spike during these tests? Perhaps you are hitting the floor limit
>>>>>>> of what your CPU is capable of?
>>>>>>>
>>>>>>> Do you have CPU graphs? If not, can you enable them and try again?
>>>>>>>
>>>>>>> Cal
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <
>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>
>>>>>>>> You are right. After many improvements and modifications to dail
>>>>>>>> plan, removing unnecessary modules and using Ngnix web server, things are
>>>>>>>> much better now. Except that directory information is fetched 3 times per
>>>>>>>> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for
>>>>>>>> dial plan).
>>>>>>>>
>>>>>>>> Since Monday, we have started analyzing the results of load test,
>>>>>>>> specially looking into packet traces, voicemail recordings and so on. We
>>>>>>>> observed a strange thing, although we were able to test 1536 concurrent
>>>>>>>> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls
>>>>>>>> per second rate. But there is strange and unexplained delay in starting RTP
>>>>>>>> from FS side.
>>>>>>>>
>>>>>>>> FS plays an IVR in early media state, then answers the call to
>>>>>>>> directly start recording voicemail. The problem is if call rate is under 25
>>>>>>>> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP
>>>>>>>> in early media state. This delay is negligible after processing few hundred
>>>>>>>> thousand calls but after about 1.2 million calls the delay has creeps up to
>>>>>>>> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2
>>>>>>>> second after around 400 calls had done. At 50 CPS its ~3 second and after
>>>>>>>> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in
>>>>>>>> starting playing IVR in early media state, once the IVR starts playing
>>>>>>>> everything works perfectly fine. No call fails, but this delay would be
>>>>>>>> annoying for anyone using the servers with over 30 CPS.
>>>>>>>>
>>>>>>>> Just to let you know, we checked everything, system resources,
>>>>>>>> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested
>>>>>>>> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing
>>>>>>>> seems to have any impact on the results.
>>>>>>>>
>>>>>>>> I have run of out of possible explanations, so calling you guys for
>>>>>>>> help.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike at jerris.com>wrote:
>>>>>>>>
>>>>>>>>> I would need to see the specifics of your dialplan, but I suspect
>>>>>>>>> you have some transfers or execute extension that are causing it to
>>>>>>>>> re-enter dialplan.  Are you sure on all the others that they are making the
>>>>>>>>> exact same requests?  Details here will be necessary to understand exactly
>>>>>>>>> what is going on..
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <
>>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Guy, after detailed analysis i can confirm it was an Apache issue,
>>>>>>>>> and changing Apache configuration changes the concurrent call limit.
>>>>>>>>> However, one thing i observed about xml curl is that it makes several
>>>>>>>>> request with same parameters per call. Even if we have no calls running on
>>>>>>>>> server, as well as as Apache is completely free. I can see FS makes at least
>>>>>>>>>
>>>>>>>>> 1). 3 hits for dialplan.
>>>>>>>>> 2). 2 hits for directory.
>>>>>>>>> 3). 4 hits for voicemail.
>>>>>>>>> 4). 3 hits for ivr.
>>>>>>>>> 5). 2 hits for sofia.
>>>>>>>>> and so on..
>>>>>>>>>
>>>>>>>>> While ideally only one hit should be sufficient to get all these
>>>>>>>>> configurations. What is the point of making all these redundant hits? We
>>>>>>>>> have roughly 15 - 20 hits made per call, which is huge and causes
>>>>>>>>> significant load on Apache.
>>>>>>>>>
>>>>>>>>> Both Apache and FS are running on dedicated machines and connected
>>>>>>>>> over Gigabit Ethernet with zero packet loss, so i doubt these hits are
>>>>>>>>> "re-transmission".
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <
>>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, its a feature, i know. I was just looking at how this
>>>>>>>>>> feature was tested, as mentioned in its logs,
>>>>>>>>>>
>>>>>>>>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408
>>>>>>>>>> Processing 21653XXXXX->121637XXXXX in context termination
>>>>>>>>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP
>>>>>>>>>> error 0 trying to fetch http://192.168.0.2/gateway.php
>>>>>>>>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML
>>>>>>>>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike at jerris.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> This is adding a feature to allow timeout to be set in ms
>>>>>>>>>>> instead of seconds.  This is not a problem, its a feature, that was added a
>>>>>>>>>>> while ago.  Until you can determine if the request is being sent for the
>>>>>>>>>>> one thats failing, your not going to solve this issue.
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mit freundlichen Grüßen
>>>>>>>> Muhammad Shahzad
>>>>>>>> -----------------------------------
>>>>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>>>>> CISCO Certified Network Associate (CCNA)
>>>>>>>> Cell: +49 176 99 83 10 85
>>>>>>>> MSN: shari_786pk at hotmail.com
>>>>>>>> Email: shaheryarkh at googlemail.com
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mit freundlichen Grüßen
>>>>>> Muhammad Shahzad
>>>>>> -----------------------------------
>>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>>> CISCO Certified Network Associate (CCNA)
>>>>>> Cell: +49 176 99 83 10 85
>>>>>> MSN: shari_786pk at hotmail.com
>>>>>> Email: shaheryarkh at googlemail.com
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mit freundlichen Grüßen
>>>> Muhammad Shahzad
>>>> -----------------------------------
>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>> CISCO Certified Network Associate (CCNA)
>>>> Cell: +49 176 99 83 10 85
>>>> MSN: shari_786pk at hotmail.com
>>>> Email: shaheryarkh at googlemail.com
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Mit freundlichen Grüßen
>> Muhammad Shahzad
>> -----------------------------------
>> CISCO Rich Media Communication Specialist (CRMCS)
>> CISCO Certified Network Associate (CCNA)
>> Cell: +49 176 99 83 10 85
>> MSN: shari_786pk at hotmail.com
>> Email: shaheryarkh at googlemail.com
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>

-- 
Mit freundlichen Grüßen
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: +49 176 99 83 10 85
MSN: shari_786pk at hotmail.com
Email: shaheryarkh at googlemail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130612/0c90599d/attachment-0001.html 