[Freeswitch-users] FreeSwitch - Performance issues

Steven Ayre steveayre at gmail.com
Wed Sep 2 23:48:17 MSD 2015


That's assuming 100% ASR. It might be acceptable to return 503 when too
busy but still need to process those INVITEs to reject the call at that
rate.

On 2 September 2015 at 15:01, Brian West <brian at freeswitch.org> wrote:

> There is absolutely no way you can have 800 concurrent calls at 400cps,
> and have them last 10 seconds, Your math doesn't add up.  400cps with 10
> seconds acd would be around 4000 concurrent sessions.
>
> This is the exact reason we don't endorse any types of load testing.
>
> While we can do crazy things like that in specific test cases in a lab,
> real world is never that structured.  Sorry but the glove doesn't fit!
>
> /b
>
>
> On Wed, Sep 2, 2015 at 3:53 AM, Shaun Stokes <
> shaun.stokes at itec-support.co.uk> wrote:
>
>> Hi Brian,
>>
>>
>>
>> I’m a huge fan of virtualization, but in our environment it simply
>> doesn’t provide the performance we’re looking for. The tests carried out
>> were 10 second calls which were routed to a Call Centre Queue with one
>> registered SIP extension as a member.
>>
>>
>>
>> There was a significant improvement when we tested on an Amazon VM in
>> comparison to Hyper-V, but we’d like a system which can handle large
>> volumes of call traffic, based on our tests this will need to be physical
>> hardware to achieve the best performance.
>>
>>
>>
>> Further to this, we have managed to resolve our initial problem with CPU
>> spikes on Hyper-V following these steps:
>>
>> ·         Disable VMQ on all physical and virtual adapters on the
>> Hyper-V host which are linked to the FreeSwitch VM
>>
>> ·         Reduce the virtual cores on all VMs to at least half of the
>> available physical cores
>>
>> ·         Increase the Processor Relative weight on the FreeSwitch VMs
>> to 200 (default 100)
>>
>>
>>
>> Thanks,
>>
>> Shaun
>>
>>
>>
>> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Brian West
>> *Sent:* 01 September 2015 21:21
>>
>> *To:* FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance issues
>>
>>
>>
>> Shaun,
>>
>>
>>
>> 800 concurrent with 400cps, expectations are a little out of touch with
>> reality and your test is no where near real world.  Your call duration is
>> ~2000ms, little to no time to actually do anything useful.  You're burning
>> the bulk of your CPU time on signaling and doing no real work with the
>> session.
>>
>>
>>
>> So yes, you'll always get the best results out of bare metal, but you can
>> also get good performance on a VM (*)
>>
>>
>>
>> (*) This is in no way an endorsement of running FreeSWITCH in a VM, Your
>> mileage may vary depending on your specific application and load
>> requirements.
>>
>>
>>
>> Thanks,
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Sep 1, 2015 at 3:08 PM, Shaun Stokes <
>> shaun.stokes at itec-support.co.uk> wrote:
>>
>> Update
>>
>> We've performed various tests and have come to the conclusion as advised
>> by others, that visualization is causing bottlenecks.
>>
>> Using Amazon EC2 on an 8 core high performance virtual server this maxed
>> out at around 800 concurrent sessions, 400 sessions per second.
>>
>> Using a Dell 1950 8 core server this was able to handle 1000 concurrent
>> sessions, 400 sessions per second and still have plenty of resource
>> available. Average CPU was fluctuating between 30-40% under peak load.
>>
>> Conclusion, if you want to run a high performance FreeSwitch server in a
>> production you should stick with physical hardware and avoid visualization
>> where possible.
>> ________________________________________
>> From: freeswitch-users-bounces at lists.freeswitch.org [
>> freeswitch-users-bounces at lists.freeswitch.org] on behalf of Shaun Stokes
>> [shaun.stokes at itec-support.co.uk]
>> Sent: 28 August 2015 15:37
>> To: 'FreeSWITCH Users Help'
>>
>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues
>>
>> Hi Nathan,
>>
>> Thanks for the suggestion, we still see the CPU spikes when is under load
>> after reducing to 8 cores but I think there may actually be an improvement
>> in the audio quality.
>>
>> It's still not what we're expecting, so we will have to test our build on
>> physical hardware too.
>>
>> Thanks,
>> Shaun
>>
>> -----Original Message-----
>> From: freeswitch-users-bounces at lists.freeswitch.org [mailto:
>> freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Nathan
>> Neulinger
>> Sent: 28 August 2015 15:04
>> To: FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
>> Subject: Re: [Freeswitch-users] FreeSwitch - Performance issues
>>
>> Yes, I know you said Hyper-V, I don't know if it has the same sort of
>> scheduler behavior or not, just mentioned the vmware for context.
>>
>> On 08/28/2015 08:46 AM, Nathan Neulinger wrote:
>> > One thing that I see jump out - you've probably got too many virtual
>> > cores defined on that VM, particularly since you've assigned # vcpu the
>> same as the number of real cores on the host system.
>> >
>> > Try your test again dropping that to 8 virtual CPUs and I think you
>> > might see better results. (Or at least drop to less than 100% of the
>> > total cores on the hardware.)
>> >
>> > Particularly if this host system has other vms running on it, you're
>> > hurting the vm scheduler's ability to hand out resources. With vmware,
>> > I believe a simple rule of thumb is that scheduler requires N-1 CPUs
>> > to be locked to schedule execution of an N cpu virtual machine - so
>> even if it doesn't need all of the CPUs, it has to reserve all of them any
>> time you have more than a 2 vCPU vm.
>> >
>> > -- Nathan
>> >
>> > On 08/28/2015 08:18 AM, Michael Jerris wrote:
>> >> The numbers your posting are fairly low, I'm guessing this is a
>> >> grossly underpowered vm that is causing your issues
>> >>
>> >> On Friday, August 28, 2015, Shaun Stokes <
>> shaun.stokes at itec-support.co.uk <mailto:shaun.stokes at itec-support.co.uk>>
>> wrote:
>> >>
>> >>      Hi Peter,____
>> >>
>> >>      __ __
>> >>
>> >>      Thanks for the advice, this is something we're already looking
>> into but we don't have the new hardware available
>> >>      yet.____
>> >>
>> >>      __ __
>> >>
>> >>      Debian 8 has great integration with Hyper-V on 2012R2 and
>> operates as a Generation 2 VM, obviously it's never going
>> >>      to be as good as running directly on the hardware but we were
>> >> hoping for better performance.____
>> >>
>> >>      __ __
>> >>
>> >>      Thanks,____
>> >>
>> >>      Shaun____
>> >>
>> >>      __ __
>> >>
>> >>      *From:*freeswitch-users-bounces at lists.freeswitch.org
>> >>      <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users-bounces at lists.freeswitch.org');>
>> >>      [mailto:freeswitch-users-bounces at lists.freeswitch.org
>> >>      <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of *Peter
>> Olsson
>> >>      *Sent:* 28 August 2015 10:43
>> >>      *To:* FreeSWITCH Users Help <
>> freeswitch-users at lists.freeswitch.org
>> >>      <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users at lists.freeswitch.org');>>
>> >>      *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance
>> >> issues____
>> >>
>> >>      __ __
>> >>
>> >>      If you want high performance, I recommend using real hardware,
>> not a virtual machine setup - it will cause you
>> >>      issues.____
>> >>
>> >>      __ __
>> >>
>> >>      /Peter____
>> >>
>> >>      __ __
>> >>
>> >>      2015-08-28 9:44 GMT+02:00 Shaun Stokes <
>> shaun.stokes at itec-support.co.uk
>> >>
>> >> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>:__
>> >> __
>> >>
>> >>          Hi Michael,____
>> >>
>> >>          ____
>> >>
>> >>          Thanks for the response and recommendation.____
>> >>
>> >>          ____
>> >>
>> >>          Our new build is now on Debian 8 Jessie, performance on the
>> system is noticeably better and any audio problems
>> >>          while the system is under load are significantly reduced but
>> we do still experience spikes along with reduced
>> >>          audio quality when the system is processing 20 calls per
>> >> second while already maintaining 80 existing calls.____
>> >>
>> >>          ____
>> >>
>> >>          We're still hitting some kind of bottleneck, what should we
>> expect to be able to support on a single piece of
>> >>          hardware? We've kept our dialplans as small as possible and
>> we're using Memcache, but something tells me it's
>> >>          the call setup process which is triggering the spikes, is
>> this possibly a limitation on the SIP profiles which
>> >>          are single threaded?____
>> >>
>> >>          ____
>> >>
>> >>          I also see FreeSWITCH supports clustering but it's in
>> relation to high availability, is it possible to cluster a
>> >>          single FreeSWITCH instance across multiple hardware or should
>> we be looking elsewhere to solve these
>> >>          bottlenecks?____
>> >>
>> >>          ____
>> >>
>> >>          We switched to TCP Vegas on our new build as recommended on
>> the FreeSWITCH Wiki, this has provided a noticeable
>> >>          reduction in audio latency but doesn't solve the issue with
>> >> CPU spikes.____
>> >>
>> >>          ____
>> >>
>> >>          Many Thanks,____
>> >>
>> >>          Shaun____
>> >>
>> >>          ____
>> >>
>> >>          *From:*freeswitch-users-bounces at lists.freeswitch.org
>> >>          <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users-bounces at lists.freeswitch.org');>
>> >>          [mailto:freeswitch-users-bounces at lists.freeswitch.org
>> >>          <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users-bounces at lists.freeswitch.org');>] *On Behalf Of
>> *Michael Jerris
>> >>          *Sent:* 15 August 2015 23:00
>> >>          *To:* FreeSWITCH Users Help <
>> freeswitch-users at lists.freeswitch.org
>> >>          <javascript:_e(%7B%7D,'cvml','
>> freeswitch-users at lists.freeswitch.org');>>
>> >>          *Subject:* Re: [Freeswitch-users] FreeSwitch - Performance
>> >> issues____
>> >>
>> >>          ____
>> >>
>> >>          There are multiple known issues with Ubuntu 12.04.  We
>> reccomend Debian 8 Jessie.
>> >>
>> >>          On Friday, August 14, 2015, Shaun Stokes <
>> shaun.stokes at itec-support.co.uk
>> >>
>> >> <javascript:_e(%7B%7D,'cvml','shaun.stokes at itec-support.co.uk');>>
>> >> wrote:____
>> >>
>> >>              Hi,____
>> >>
>> >>              ____
>> >>
>> >>              We're experiencing performance issues with FreeSwitch,
>> our target is 500 concurrent sessions, but at the
>> >>              moment this starts to bottleneck around 30.____
>> >>
>> >>              ____
>> >>
>> >>              Host system:____
>> >>
>> >>              Windows Server 2012 Hyper-V Host____
>> >>
>> >>              AMD Opteron 4386 (2 processors) - 16 cores total____
>> >>
>> >>              128GB DDR3____
>> >>
>> >>              2TB RAID 5 (700MB/s tested read and write
>> >> throughput)____
>> >>
>> >>              ____
>> >>
>> >>              FreeSwitch Virtual Machine:____
>> >>
>> >>              FreeSwitch 1.4.15____
>> >>
>> >>              Ubuntu 12.04 LTS____
>> >>
>> >>              16 Virtual cores (high priority)____
>> >>
>> >>              2GB RAM (would assign more but FreeSwitch never seems to
>> >> use much)____
>> >>
>> >>              500GB HD (on VHDX)____
>> >>
>> >>              ____
>> >>
>> >>              After around 30 concurrent sessions we begin to see CPU
>> spikes almost every time a new call comes in, as the
>> >>              sessions increase the size and frequency of the CPU
>> spikes also increase. The system seems to be able to sit
>> >>              comfortably with over 100 concurrent sessions and 80%
>> idle CPU, providing we don't have any new calls
>> >>              hitting the platform. The spikes are causing audio (RTP)
>> to stutter or in some cases drop completely for a
>> >>              few seconds.____
>> >>
>> >>              ____
>> >>
>> >>              The FreeSwitch spikes are occurring on all 16 cores, we
>> have been monitoring the system using htop and
>> >>              mpstat.____
>> >>
>> >>              ____
>> >>
>> >>              This is an example of when we receive an inbound call
>> >> while we have 34 concurrent sessions:____
>> >>
>> >>              14:26:04     CPU    %usr   %nice    %sys %iowait    %irq
>>  %soft  %steal  %guest   %idle____
>> >>
>> >>              14:26:05     all    1.67    0.00   27.34    0.00    0.25
>>   0.00    0.00    0.00   70.74____
>> >>
>> >>              14:26:05       0    1.00    0.00   28.00    0.00    0.00
>>   0.00    0.00    0.00   71.00____
>> >>
>> >>              14:26:05       1    0.99    0.00   26.73    0.00    0.00
>>   0.00    0.00    0.00   72.28____
>> >>
>> >>              14:26:05       2    0.99    0.00   26.73    0.00    0.00
>>   0.00    0.00    0.00   72.28____
>> >>
>> >>              14:26:05       3    3.92    0.00   26.47    0.00    0.00
>>   0.00    0.00    0.00   69.61____
>> >>
>> >>              14:26:05       4    0.99    0.00   25.74    0.00    1.98
>>   0.00    0.00    0.00   71.29____
>> >>
>> >>              14:26:05       5    0.99    0.00   26.73    0.00    0.99
>>   0.00    0.00    0.00   71.29____
>> >>
>> >>              14:26:05       6    0.00    0.00   26.47    0.00    0.98
>>   0.00    0.00    0.00   72.55____
>> >>
>> >>              14:26:05       7    0.00    0.00   26.73    0.00    0.00
>>   0.00    0.00    0.00   73.27____
>> >>
>> >>              14:26:05       8    3.96    0.00   27.72    0.00    0.00
>>   0.00    0.00    0.00   68.32____
>> >>
>> >>              14:26:05       9    2.00    0.00   27.00    0.00    0.00
>>   0.00    0.00    0.00   71.00____
>> >>
>> >>              14:26:05      10    6.00    0.00   32.00    0.00    0.00
>>   0.00    0.00    0.00   62.00____
>> >>
>> >>              14:26:05      11    1.96    0.00   27.45    0.00    0.00
>>   0.00    0.00    0.00   70.59____
>> >>
>> >>              14:26:05      12    0.99    0.00   26.73    0.00    0.00
>>   0.00    0.00    0.00   72.28____
>> >>
>> >>              14:26:05      13    1.00    0.00   28.00    0.00    0.00
>>   0.00    0.00    0.00   71.00____
>> >>
>> >>              14:26:05      14    0.00    0.00   27.00    0.00    0.00
>>   0.00    0.00    0.00   73.00____
>> >>
>> >>              ____
>> >>
>> >>              This is when the system is not receiving an inbound call
>> but is sitting comfortably at 34 concurrent
>> >>              sessions:____
>> >>
>> >>              14:25:57     CPU    %usr   %nice    %sys %iowait    %irq
>>  %soft  %steal  %guest   %idle____
>> >>
>> >>              14:25:58     all    0.87    0.00    0.62    0.00    0.12
>>   0.00    0.00    0.00   98.39____
>> >>
>> >>              14:25:58       0    0.99    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   98.02____
>> >>
>> >>              14:25:58       1    0.99    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   98.02____
>> >>
>> >>              14:25:58       2    0.98    0.00    0.98    0.00    0.00
>>   0.00    0.00    0.00   98.04____
>> >>
>> >>              14:25:58       3    0.00    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   99.01____
>> >>
>> >>              14:25:58       4    0.00    0.00    0.00    0.00    1.98
>>   0.00    0.00    0.00   98.02____
>> >>
>> >>              14:25:58       5    0.00    0.00    0.00    0.00    0.99
>>   0.00    0.00    0.00   99.01____
>> >>
>> >>              14:25:58       6    0.00    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   99.01____
>> >>
>> >>              14:25:58       7    0.00    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   99.01____
>> >>
>> >>              14:25:58       8    5.00    0.00    1.00    0.00    0.00
>>   0.00    0.00    0.00   94.00____
>> >>
>> >>              14:25:58       9    1.00    0.00    0.00    0.00    0.00
>>   0.00    0.00    0.00   99.00____
>> >>
>> >>              14:25:58      10    1.00    0.00    0.00    0.00    0.00
>>   0.00    0.00    0.00   99.00____
>> >>
>> >>              14:25:58      11    0.99    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   98.02____
>> >>
>> >>              14:25:58      12    0.00    0.00    0.00    0.00    0.00
>>   0.00    0.00    0.00  100.00____
>> >>
>> >>              14:25:58      13    1.98    0.00    0.99    0.00    0.00
>>   0.00    0.00    0.00   97.03____
>> >>
>> >>              14:25:58      14    0.00    0.00    0.00    0.00    0.00
>>   0.00    0.00    0.00  100.00____
>> >>
>> >>              14:25:58      15    0.00    0.00    0.00    0.00    0.00
>>   0.00    0.00    0.00  100.00____
>> >>
>> >>              ____
>> >>
>> >>              We're using FreeSwitch in multi-tenant mode, we have
>> tried dedicating a single VM for one tenancy but still
>> >>              experience the issue. My assumption has been that this
>> will be due to the dialplan, I've optimized this
>> >>              slightly by writing LUA scripts to handle some of the
>> inbound calls this appears to take away some of the
>> >>              load but we're still using the internal dialplans for
>> each tenancy (tenancy dialplans have an average of 300
>> >>              entries).____
>> >>
>> >>              ____
>> >>
>> >>              We are using the following arguments when running
>> >> FreeSwitch:____
>> >>
>> >>              -rp -nc -nonat____
>> >>
>> >>              ____
>> >>
>> >>              I've seen another post which says we should operate
>> FreeSwitch in High Performance mode using -hp but can't
>> >>              find anything information about this, is this still a
>> >> valid argument to use with FreeSwitch?____
>> >>
>> >>              ____
>> >>
>> >>              Has anyone experienced similar performance issues before
>> >> or have any advice?____
>> >>
>> >>              ____
>> >>
>> >>              Hope someone may be able help.____
>> >>
>> >>              ____
>> >>
>> >>              Thanks,____
>> >>
>> >>              Shaun____
>> >>
>> >>
>> >>
>> ______________________________________________________________________
>> >>              This message has been checked for all known viruses by
>> MessageLabs Virus Scanning Service.
>> >>
>> >> _____________________________________________________________________
>> >> _____
>> >>
>> >>
>> >>
>> ______________________________________________________________________
>> >>          This message has been checked for all known viruses by
>> MessageLabs Virus Scanning Service.
>> >>
>> >> _____________________________________________________________________
>> >> _____
>> >>
>> >>
>> >>
>> ______________________________________________________________________
>> >>          This message has been checked for all known viruses by
>> MessageLabs Virus Scanning Service.
>> >>
>> >> _____________________________________________________________________
>> >> _____
>> >>
>> >>
>> >>
>> _________________________________________________________________________
>> >>          Professional FreeSWITCH Consulting Services:
>> >>          consulting at freeswitch.org <javascript:_e(%7B%7D,'cvml','
>> consulting at freeswitch.org');>
>> >>          http://www.freeswitchsolutions.com
>> >>
>> >>          Official FreeSWITCH Sites
>> >>          http://www.freeswitch.org
>> >>          http://confluence.freeswitch.org
>> >>          http://www.cluecon.com
>> >>
>> >>          FreeSWITCH-users mailing list
>> >>          FreeSWITCH-users at lists.freeswitch.org
>> <javascript:_e(%7B%7D,'cvml','FreeSWITCH-users at lists.freeswitch.org');>
>> >>          http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >>          UNSUBSCRIBE:
>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>> >>          http://www.freeswitch.org____
>> >>
>> >>      __ __
>> >>
>> >>
>> >>
>> ______________________________________________________________________
>> >>      This message has been checked for all known viruses by
>> MessageLabs Virus Scanning Service.
>> >>
>> >> _____________________________________________________________________
>> >> _____
>> >>
>> >>
>> >>
>> ______________________________________________________________________
>> >>      This message has been checked for all known viruses by
>> MessageLabs Virus Scanning Service.
>> >>
>> >> _____________________________________________________________________
>> >> _
>> >>
>> >>
>> >>
>> >> _____________________________________________________________________
>> >> ____ Professional FreeSWITCH Consulting Services:
>> >> consulting at freeswitch.org
>> >> http://www.freeswitchsolutions.com
>> >>
>> >> Official FreeSWITCH Sites
>> >> http://www.freeswitch.org
>> >> http://confluence.freeswitch.org
>> >> http://www.cluecon.com
>> >>
>> >> FreeSWITCH-users mailing list
>> >> FreeSWITCH-users at lists.freeswitch.org
>> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-us
>> >> ers
>> >> http://www.freeswitch.org
>> >>
>> >
>>
>> --
>> ------------------------------------------------------------
>> Nathan Neulinger                       nneul at mst.edu
>> Missouri S&T Information Technology    (573) 612-1412
>> System Administrator - Architect
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by  MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by  MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>
>>
>>
>> --
>>
>> *Brian West*
>> brian at freeswitch.org
>>
>> *Twitter: @FreeSWITCH , @briankwest*
>> http://www.freeswitchbook.com
>> http://www.freeswitchcookbook.com
>>
>> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit:
>> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>>
>> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378)
>> *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
>>
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> ______________________________________________________________________
>> This message has been checked for all known viruses by MessageLabs Virus
>> Scanning Service.
>> ______________________________________________________________________
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
>
> *Brian West*
> brian at freeswitch.org
>
>
> *Twitter: @FreeSWITCH , @briankwest*
> http://www.freeswitchbook.com
> http://www.freeswitchcookbook.com
>
> Got Bugs? Report them here <https://freeswitch.org/jira>! | Reddit:
> /r/freeswitch <https://www.reddit.com/r/freeswitch>
>
> *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378)
> *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20150902/a9345aa1/attachment-0001.html 


Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list