Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements.<div>
<br></div><div>OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale.</div><div><br></div><div>Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes.</div>
<div><br></div><div>If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :)</div><div><br></div><div>Cal</div><div><br></div><div><div><div><div>
<div><div><div><div class="gmail_quote">On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <span dir="ltr"><<a href="mailto:drk@drkngs.net" target="_blank">drk@drkngs.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
<div>Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch.</div>
<div> </div>
<div>First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on...</div>
<div> </div>
<div>This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients!</div>
<div> </div>
<p style="MARGIN:0in 0in 10pt" class="MsoNormal"><font size="3"><font face="Calibri">One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment.<u></u><u></u><u></u></font></font></p>
<p style="MARGIN:0in 0in 10pt" class="MsoNormal"><font size="3"><font face="Calibri">One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. <u></u><u></u></font></font></p>
<p style="MARGIN:0in 0in 10pt" class="MsoNormal"><font size="3"><font face="Calibri">It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements<u></u><u></u></font></font></p>
<div>--Dave</div>
<div> </div>
<div>
<hr>
</div>
<div><b>From:</b> Muhammad Shahzad [mailto:<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>]<br><b>To:</b> FreeSWITCH Users Help [mailto:<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>]<br>
<b>Sent:</b> Wed, 12 Jun 2013 07:57:18 -0700<br><b>Subject:</b> Re: [Freeswitch-users] XML_CURL max connection limit<br><br></div><div><div class="h5">
<blockquote style="BORDER-LEFT:#0000ff 2px solid;PADDING-LEFT:5px;MARGIN-LEFT:5px;MARGIN-RIGHT:0px">
<div>Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
<div><br></div>
<div>Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.</div>
<div><br></div>
<div>Thank you.</div>
<div><br></div>
<div><br></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <span><<a href="mailto:cal.leeming@simplicitymedialtd.co.uk" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
<div><br></div>
<div>Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.</div><span><font color="#888888"></font></span>
<div><span><font color="#888888"><br>Cal</font></span>
<div>
<div><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <span><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div>We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.<br></div>
<div class="gmail_extra"><br></div>
<div class="gmail_extra">Thank you.</div>
<div>
<div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <span><<a href="mailto:cal.leeming@simplicitymedialtd.co.uk" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
<div><span><font color="#888888"><br>Cal</font></span>
<div>
<div><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <span><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>Steven, i am using soft timer. Should i try another? Please suggest.
<div><br></div>
<div>Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.</div>
<div><br></div>
<div>Thank you.</div>
<div><br></div></div>
<div>
<div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <span><<a href="mailto:cal.leeming@simplicitymedialtd.co.uk" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
<div><br></div>
<div>Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?</div>
<div><br></div>
<div>Do you have CPU graphs? If not, can you enable them and try again?</div><span><font color="#888888">
<div><br></div></font></span>
<div><span><font color="#888888">Cal</font></span>
<div>
<div><br><br>
<div class="gmail_quote">On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <span><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
<div><br></div>
<div>Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.</div>
<div><br></div>
<div>FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.</div>
<div><br></div>
<div>Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.</div>
<div><br></div>
<div>I have run of out of possible explanations, so calling you guys for help.</div>
<div><br></div>
<div>Thank you.</div>
<div><br></div>
<div><br></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">
<div>
<div>On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <span><<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>></span> wrote:<br></div></div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div>
<div style="WORD-WRAP:break-word">I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
<div><br></div>
<div>Mike</div>
<div>
<div><br>
<div>
<div>On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>> wrote:</div><br>
<blockquote type="cite">
<div>Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
<div><br></div>
<div>1). 3 hits for dialplan.</div>
<div>2). 2 hits for directory.</div>
<div>3). 4 hits for voicemail.</div>
<div>4). 3 hits for ivr.</div>
<div>5). 2 hits for sofia.</div>
<div>and so on..</div>
<div><br></div>
<div>While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.</div>
<div><br></div>
<div>Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".</div>
<div><br></div>
<div>Thank you.</div>
<div><br></div>
<div><br></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <span><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
<div><br></div>
<div><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination</span><br style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">
<font color="#ff0000"><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch </span><a style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px;TEXT-DECORATION:none" href="http://192.168.0.2/gateway.php" target="_blank">http://192.168.0.2/gateway.php</a></font><br style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">
<span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml</span><br style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">
</div>
<div><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px"><br></span></div>
<div><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px">Thank you.</span></div>
<div><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px"><br></span></div>
<div><span style="LINE-HEIGHT:17px;FONT-FAMILY:Arial,FreeSans,Helvetica,sans-serif;FONT-SIZE:13px"><br></span></div></div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">
<div>
<div>On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <span><<a href="mailto:mike@jerris.com" target="_blank">mike@jerris.com</a>></span> wrote:<br></div></div>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div>
<div style="WORD-WRAP:break-word">This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
<div><br></div>
<div>Mike</div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div><br></div></div></div><br></div></div>
<div>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></div></blockquote></div>
<div><br><br clear="all">
<div><br></div>-- <br>
<div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Mit freundlichen Grüßen</span></div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Muhammad Shahzad</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">------------------------------</span><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">-----</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px"><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Certified Network Associate (CCNA)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" target="_blank">+49 176 99 83 10 85</a></span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">MSN: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shari_786pk@hotmail.com" target="_blank">shari_786pk@hotmail.com</a><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Email: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shaheryarkh@googlemail.com" target="_blank">shaheryarkh@googlemail.com</a> </div>
</div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all">
<div><br></div>-- <br>
<div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Mit freundlichen Grüßen</span></div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Muhammad Shahzad</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">------------------------------</span><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">-----</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px"><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Certified Network Associate (CCNA)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" target="_blank">+49 176 99 83 10 85</a></span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">MSN: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shari_786pk@hotmail.com" target="_blank">shari_786pk@hotmail.com</a><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Email: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shaheryarkh@googlemail.com" target="_blank">shaheryarkh@googlemail.com</a> </div>
</div></div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all">
<div><br></div>-- <br>
<div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Mit freundlichen Grüßen</span></div><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Muhammad Shahzad</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">------------------------------</span><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">-----</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px"><span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">CISCO Certified Network Associate (CCNA)</span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" target="_blank">+49 176 99 83 10 85</a></span><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">MSN: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shari_786pk@hotmail.com" target="_blank">shari_786pk@hotmail.com</a><br style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">
<span style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(136,136,136);FONT-SIZE:13px">Email: </span><a style="FONT-FAMILY:arial,sans-serif;COLOR:rgb(17,85,204);FONT-SIZE:13px" href="mailto:shaheryarkh@googlemail.com" target="_blank">shaheryarkh@googlemail.com</a> </div>
</div></div></div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div><br>_________________________________________________________________________<br>Professional FreeSWITCH Consulting Services:<br><a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com/" target="_blank">http://www.freeswitchsolutions.com</a><br><br>FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br><a href="http://www.cudatel.com/" target="_blank">http://www.cudatel.com</a><br>
<br>Official FreeSWITCH Sites<br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br><a href="http://wiki.freeswitch.org/" target="_blank">http://wiki.freeswitch.org</a><br><a href="http://www.cluecon.com/" target="_blank">http://www.cluecon.com</a><br>
<br>FreeSWITCH-users mailing list<br><a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br><a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br><a href="http://www.freeswitch.org/" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all">
<div><br></div>-- <br>
<div><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Mit freundlichen Grüßen</span></div><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Muhammad Shahzad</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">------------------------------</span><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">-----</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif"><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Certified Network Associate (CCNA)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" value="+4917699831085" target="_blank">+49 176 99 83 10 85</a></span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">MSN: </span><a style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" href="mailto:shari_786pk@hotmail.com" target="_blank">shari_786pk@hotmail.com</a><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Email: </span><a style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" href="mailto:shaheryarkh@googlemail.com" target="_blank">shaheryarkh@googlemail.com</a> </div>
</blockquote>
<div> </div>
<div> </div></div></div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div></div></div></div></div>