[Freeswitch-users] XML_CURL max connection limit

Muhammad Shahzad shaheryarkh at gmail.com
Thu Jun 13 05:09:10 MSD 2013 

Here is JIRA issue i have just opened, for future reference.

http://jira.freeswitch.org/browse/FS-5508

Thank you.



On Wed, Jun 12, 2013 at 7:40 PM, Muhammad Shahzad <shaheryarkh at gmail.com>wrote:

> Well, i am using xml curl, since its the only module i know that offers
> centralized FS configs management. I know there may be better, faster
> modules but they don't have benefit that xml curl gives us.
>
> As for the system design, client's time and money blah blah, i did
> mentioned in in my first email that we have achieved our design goals and
> target capacity requirements etc. So far client is very happy with load
> test results. But on the technical side, i see over 70% of system resources
> are free and we can go beyond our achieved capacity. This has benefit for
> all of us (me, client and FS users), since, client will save cost for
> expansion (he has 50% growth projection in next 6-9 months), i will have
> better sizing calculation for future projects, thus lowering cost for
> future clients and normal FS users will have better understanding of how to
> scale FS (both horizontally and vertically), i will post results of my test
> once done with deployment.
>
> Neither FS nor web server fail even for 1,536 concurrent calls at rate of
> 60 CPS, so its great success for both FS and xml curl in my view. Its only
> the initial delay in playing IVR which is kind of inconvenient from user
> experience point of view that i am trying to get rid of.
>
> Anyways, i am opening up a JIRA ticket now. So core developers may look
> into it. Thanks to all of you who replied and help me in optimization the
> setup.
>
> Thank you.
>
>
>
>
> On Wed, Jun 12, 2013 at 6:38 PM, Cal Leeming [Simplicity Media Ltd] <
> cal.leeming at simplicitymedialtd.co.uk> wrote:
>
>> Dave, with all due respect, every time the discussion of mod_xml_curl or
>> anything non-windows comes up you immediately throw in a negative note, and
>> it's starting to get a bit old now. Until one of the core devs steps up and
>> says "you shouldn't be using mod_xml_curl" then I don't think it's right to
>> be telling users that this is the wrong approach to be using - especially
>> when we don't even know if it's mod_xml_curl that is causing the problem in
>> this particular case. As mentioned in the last email, the next steps for OP
>> is to identify the root cause of these delays, either via profiling or
>> print statements.
>>
>> OP - I think at this point, you really need to take this over to a JIRA
>> ticket to avoid spamming the mailing list, as the thread is starting to go
>> a bit stale.
>>
>> Once you've created a ticket, let us know in this thread and we'll see
>> about getting a core dev to give it some eyes.
>>
>> If this is an urgent matter, you could also pay for a few hours of FS
>> support which will get the immediate attention of a core dev :)
>>
>> Cal
>>
>> On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk at drkngs.net> wrote:
>>
>>> **
>>> Ok, this "square peg in rouond hole" thread is not getting anything
>>> solved. You all need to take a step back, and look at the overall diesign
>>> of the applcation you are trying to build from a top down approch.
>>>
>>> First define what the application has to do, with all of it's
>>> requirements, including load, resonse times, what information you need to
>>> look up per call, and so on...
>>>
>>> This seems to me like a backwards approch was taken, as the start of the
>>> project, mod_xml_curl was the starting point, and then the application was
>>> designed after the method was picked. Isn't this "bass awards"? I know if I
>>> wasted my clients time and money by picking tech. before defining the
>>> problem, I would have no clients!
>>>
>>>
>>> One thing I can tell you for sure, is that using a design that involves
>>> both client and server side synchronous technologies as the starting point,
>>> isn’t going to scale out to carrier rate call handling. You can try
>>> throwing all sorts of stuff like different web servers, caching proxies in
>>> the middle, and a handful of other kludges at it, and all you’re going to
>>> get is a big “Cluster F…”, that will be a nightmare to maintain and scale.
>>> You may also become a slave to maintaining it, and monitoring your
>>> production environment.******
>>>
>>> One other thing to remember about FreeSWITCH, is that mod_xml_curl is
>>> not the only way of doing XML config binding to code. There are many
>>> modules that use the callback to implement feeding FS with real time XML
>>> config on demand. ****
>>>
>>> It may be time to re-design your application, rather then trying to
>>> build a “better mousetrap” on the external side. We could discuss this on
>>> the conference call today, and get input from all the folks on a design,
>>> given your requirements****
>>> --Dave
>>>
>>>  ------------------------------
>>>  *From:* Muhammad Shahzad [mailto:shaheryarkh at gmail.com]
>>> *To:* FreeSWITCH Users Help [mailto:
>>> freeswitch-users at lists.freeswitch.org]
>>> *Sent:* Wed, 12 Jun 2013 07:57:18 -0700
>>> *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit
>>>
>>>  Well, per our observation, we don't see any load or delay from xml
>>> curl or web server side. Also the delay is occurring exactly at one
>>> location that is after sending 183 Early Media and sending first RTP packet
>>> of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after
>>> 30 CPS it goes beyond 3.10 seconds.
>>>
>>> Regarding New Relic, we don't have WAN access on server, since its a
>>> closed environment at telco data center. Also we see the behavior remains
>>> the same if we add multiple FS serve by same web server. So, i think
>>> problem is on FS side rather then web server side.
>>>
>>> Thank you.
>>>
>>>
>>>
>>>
>>> On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <
>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>
>>>> Curious, I'm wondering if maybe excessive context switching is causing
>>>> some issues.. I'm wondering if the latency is actually caused by the
>>>> overheads on mod_xml_curl.
>>>>
>>>> Could you try putting New Relic on your application, and see what sort
>>>> of response times you get? That will identify which queries are taking a
>>>> while to process.
>>>>
>>>> Cal
>>>>
>>>>
>>>> On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <
>>>> shaheryarkh at gmail.com> wrote:
>>>>
>>>>>  We are already monitoring using top and htop. Even per core view of
>>>>> top shows no core every goes beyond 40%, average varies between 21 - 33%.
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>> On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <
>>>>> cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>>>
>>>>>> You could either throw Cacti on the server, or use server density/new
>>>>>> relic trial for quick and easy graphing.. or you can just watch `top` and
>>>>>> push number 1 whilst running to see the CPU breakdown.
>>>>>>
>>>>>> Cal
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <
>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>
>>>>>>> Steven, i am using soft timer. Should i try another? Please suggest.
>>>>>>>
>>>>>>> Cal, no there is nothing unusually changes in RAM or CPU. BTW, i
>>>>>>> have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph,
>>>>>>> how can i enable it.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd]
>>>>>>> <cal.leeming at simplicitymedialtd.co.uk> wrote:
>>>>>>>
>>>>>>>> This sounds quite interesting.. From my own benchmarks, I've seen
>>>>>>>> that IVRs applications can be quite stressful on the server.
>>>>>>>>
>>>>>>>> Have you checked to see if there is a CPU util spike or context
>>>>>>>> switching spike during these tests? Perhaps you are hitting the floor limit
>>>>>>>> of what your CPU is capable of?
>>>>>>>>
>>>>>>>> Do you have CPU graphs? If not, can you enable them and try again?
>>>>>>>>
>>>>>>>> Cal
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <
>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You are right. After many improvements and modifications to dail
>>>>>>>>> plan, removing unnecessary modules and using Ngnix web server, things are
>>>>>>>>> much better now. Except that directory information is fetched 3 times per
>>>>>>>>> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for
>>>>>>>>> dial plan).
>>>>>>>>>
>>>>>>>>> Since Monday, we have started analyzing the results of load test,
>>>>>>>>> specially looking into packet traces, voicemail recordings and so on. We
>>>>>>>>> observed a strange thing, although we were able to test 1536 concurrent
>>>>>>>>> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls
>>>>>>>>> per second rate. But there is strange and unexplained delay in starting RTP
>>>>>>>>> from FS side.
>>>>>>>>>
>>>>>>>>> FS plays an IVR in early media state, then answers the call to
>>>>>>>>> directly start recording voicemail. The problem is if call rate is under 25
>>>>>>>>> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP
>>>>>>>>> in early media state. This delay is negligible after processing few hundred
>>>>>>>>> thousand calls but after about 1.2 million calls the delay has creeps up to
>>>>>>>>> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2
>>>>>>>>> second after around 400 calls had done. At 50 CPS its ~3 second and after
>>>>>>>>> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in
>>>>>>>>> starting playing IVR in early media state, once the IVR starts playing
>>>>>>>>> everything works perfectly fine. No call fails, but this delay would be
>>>>>>>>> annoying for anyone using the servers with over 30 CPS.
>>>>>>>>>
>>>>>>>>> Just to let you know, we checked everything, system resources,
>>>>>>>>> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested
>>>>>>>>> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing
>>>>>>>>> seems to have any impact on the results.
>>>>>>>>>
>>>>>>>>> I have run of out of possible explanations, so calling you guys
>>>>>>>>> for help.
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike at jerris.com>wrote:
>>>>>>>>>
>>>>>>>>>>  I would need to see the specifics of your dialplan, but I
>>>>>>>>>> suspect you have some transfers or execute extension that are causing it to
>>>>>>>>>> re-enter dialplan.  Are you sure on all the others that they are making the
>>>>>>>>>> exact same requests?  Details here will be necessary to understand exactly
>>>>>>>>>> what is going on..
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>>  On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <
>>>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>  Guy, after detailed analysis i can confirm it was an Apache
>>>>>>>>>> issue, and changing Apache configuration changes the concurrent call limit.
>>>>>>>>>> However, one thing i observed about xml curl is that it makes several
>>>>>>>>>> request with same parameters per call. Even if we have no calls running on
>>>>>>>>>> server, as well as as Apache is completely free. I can see FS makes at
>>>>>>>>>> least
>>>>>>>>>>
>>>>>>>>>> 1). 3 hits for dialplan.
>>>>>>>>>> 2). 2 hits for directory.
>>>>>>>>>> 3). 4 hits for voicemail.
>>>>>>>>>> 4). 3 hits for ivr.
>>>>>>>>>> 5). 2 hits for sofia.
>>>>>>>>>> and so on..
>>>>>>>>>>
>>>>>>>>>> While ideally only one hit should be sufficient to get all these
>>>>>>>>>> configurations. What is the point of making all these redundant hits? We
>>>>>>>>>> have roughly 15 - 20 hits made per call, which is huge and causes
>>>>>>>>>> significant load on Apache.
>>>>>>>>>>
>>>>>>>>>> Both Apache and FS are running on dedicated machines and
>>>>>>>>>> connected over Gigabit Ethernet with zero packet loss, so i doubt these
>>>>>>>>>> hits are "re-transmission".
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <
>>>>>>>>>> shaheryarkh at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, its a feature, i know. I was just looking at how this
>>>>>>>>>>> feature was tested, as mentioned in its logs,
>>>>>>>>>>>
>>>>>>>>>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408
>>>>>>>>>>> Processing 21653XXXXX->121637XXXXX in context termination
>>>>>>>>>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received
>>>>>>>>>>> HTTP error 0 trying to fetch http://192.168.0.2/gateway.php
>>>>>>>>>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML
>>>>>>>>>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
>>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <
>>>>>>>>>>> mike at jerris.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>  This is adding a feature to allow timeout to be set in ms
>>>>>>>>>>>> instead of seconds.  This is not a problem, its a feature, that was added a
>>>>>>>>>>>> while ago.  Until you can determine if the request is being sent for the
>>>>>>>>>>>> one thats failing, your not going to solve this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Mit freundlichen Grüßen
>>>>>>>>> Muhammad Shahzad
>>>>>>>>> -----------------------------------
>>>>>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>>>>>> CISCO Certified Network Associate (CCNA)
>>>>>>>>> Cell: +49 176 99 83 10 85
>>>>>>>>> MSN: shari_786pk at hotmail.com
>>>>>>>>> Email: shaheryarkh at googlemail.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mit freundlichen Grüßen
>>>>>>> Muhammad Shahzad
>>>>>>> -----------------------------------
>>>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>>>> CISCO Certified Network Associate (CCNA)
>>>>>>> Cell: +49 176 99 83 10 85
>>>>>>> MSN: shari_786pk at hotmail.com
>>>>>>> Email: shaheryarkh at googlemail.com
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mit freundlichen Grüßen
>>>>> Muhammad Shahzad
>>>>> -----------------------------------
>>>>> CISCO Rich Media Communication Specialist (CRMCS)
>>>>> CISCO Certified Network Associate (CCNA)
>>>>> Cell: +49 176 99 83 10 85
>>>>> MSN: shari_786pk at hotmail.com
>>>>> Email: shaheryarkh at googlemail.com
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Mit freundlichen Grüßen
>>> Muhammad Shahzad
>>> -----------------------------------
>>> CISCO Rich Media Communication Specialist (CRMCS)
>>> CISCO Certified Network Associate (CCNA)
>>> Cell: +49 176 99 83 10 85
>>> MSN: shari_786pk at hotmail.com
>>> Email: shaheryarkh at googlemail.com
>>>
>>>
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Mit freundlichen Grüßen
> Muhammad Shahzad
> -----------------------------------
> CISCO Rich Media Communication Specialist (CRMCS)
> CISCO Certified Network Associate (CCNA)
> Cell: +49 176 99 83 10 85
> MSN: shari_786pk at hotmail.com
> Email: shaheryarkh at googlemail.com
>



-- 
Mit freundlichen Grüßen
Muhammad Shahzad
-----------------------------------
CISCO Rich Media Communication Specialist (CRMCS)
CISCO Certified Network Associate (CCNA)
Cell: +49 176 99 83 10 85
MSN: shari_786pk at hotmail.com
Email: shaheryarkh at googlemail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130613/dd442f2b/attachment-0001.html

Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list