<div dir="ltr">You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).<div>
<br></div><div style>Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.</div>
<div style><br></div><div style>FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.</div>
<div style><br></div><div style>Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.</div>
<div style><br></div><div style>I have run of out of possible explanations, so calling you guys for help.</div><div style><br></div><div style>Thank you.</div><div style><br></div><div style><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <span dir="ltr"><<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..<div>
<br></div><div>Mike</div><div class="im"><div><br><div><div>On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>> wrote:</div><br><blockquote type="cite">
<div dir="ltr">Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least<div>
<br></div><div>1). 3 hits for dialplan.</div><div>2). 2 hits for directory.</div><div>3). 4 hits for voicemail.</div><div>4). 3 hits for ivr.</div><div>5). 2 hits for sofia.</div><div>
and so on..</div>
<div><br></div><div>While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.</div>
<div><br></div><div>Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".</div><div><br></div>
<div>Thank you.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <span dir="ltr"><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,<div>
<br></div><div><span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination</span><br style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">
<font color="#ff0000"><span style="font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:13px;line-height:17px">2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch </span><a href="http://192.168.0.2/gateway.php" style="text-decoration:none;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:13px;line-height:17px" target="_blank">http://192.168.0.2/gateway.php</a></font><br style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">
<span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml</span><br style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">
</div><div><span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif"><br></span></div><div><span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif">Thank you.</span></div>
<div><span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif"><br></span></div><div><span style="line-height:17px;font-size:13px;font-family:Arial,FreeSans,Helvetica,sans-serif"><br>
</span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <span dir="ltr"><<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div style="word-wrap:break-word">This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.<div>
<br></div><div>Mike</div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div><br></div></div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Mit freundlichen Grüßen</span></div>
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Muhammad Shahzad</span><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">------------------------------</span><span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">-----</span><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">CISCO Certified Network Associate (CCNA)</span><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Cell: +49 176 99 83 10 85</span><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">MSN: </span><a href="mailto:shari_786pk@hotmail.com" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)" target="_blank">shari_786pk@hotmail.com</a><br style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Email: </span><a href="mailto:shaheryarkh@googlemail.com" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)" target="_blank">shaheryarkh@googlemail.com</a>
</div>