[Freeswitch-users] Problems with RTP delay at around 300 concurrent calls
Guillermo Ruiz Camauer
grcamauer at gmail.com
Tue Jul 1 01:25:21 MSD 2014
Anthony,
Are the rest of the recommendations on this page (
https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations)
still valid for a modern OS with 3.6 kernel? Specially the ulimit settings.
Thanks,
Guillermo
On Fri, Jun 27, 2014 at 8:59 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:
> Yes it should. This was from several years ago.
> On Jun 27, 2014 6:13 PM, "Guillermo Ruiz Camauer" <grcamauer at gmail.com>
> wrote:
>
>> Is this new? I always installed on Centos because of this entry on the
>> wiki:
>>
>> Recommended hardware/OS
>>
>> A 64-bit CPU running a 64-bit OS and a 64-bit version of FreeSWITCH is
>> recommended.
>>
>> CentOS Linux is the recommended OS, since that's the OS used by the core
>> developers and therefore the best tested. It will work on other systems
>> though.
>>
>>
>> Should this be updated?
>>
>>
>> Guillermo
>>
>>
>> On Fri, Jun 27, 2014 at 7:50 PM, Anthony Minessale <
>> anthony.minessale at gmail.com> wrote:
>>
>>> Our commercial platform will revolve around debian and that is what we
>>> develop on primarily. We are testing with many system libs from it for
>>> current and development versions.
>>>
>>> I would recommend debian 7 if you do not have a strong reason to
>>> avoid it.
>>> On Jun 27, 2014 5:44 PM, "Guillermo Ruiz Camauer" <grcamauer at gmail.com>
>>> wrote:
>>>
>>>> Anthony,
>>>>
>>>> Would CentOS 6.5 with the 2.6 kernel not be considered an optimal
>>>> distribution then? What do you recommend for a high perfomance 500-1000
>>>> call system?
>>>>
>>>> Guillermo
>>>>
>>>>
>>>> On Fri, Jun 27, 2014 at 7:06 PM, Anthony Minessale <
>>>> anthony.minessale at gmail.com> wrote:
>>>>
>>>>> rtp_timer_name none makes the rtp blocking and can result in side
>>>>> effects if async rtp is desired.
>>>>>
>>>>> More importantly, make sure it's a 3.x kernel and modern os. Also
>>>>> make sure your test is accurate. We get countless dups of this thread with
>>>>> load testing questions that always turn out to be improper environments or
>>>>> test setup. Consider commercial assistance:
>>>>>
>>>>> consulting at freeswitch.org
>>>>> On Jun 27, 2014 3:02 PM, "Oleg Stolyar" <olegstolyar at gmail.com> wrote:
>>>>>
>>>>>> Hi Ali,
>>>>>>
>>>>>> when you say setting the rtp-timer-name to none and rtp-timeout-sec
>>>>>> to zero made a big difference in performance, do you mean CPU usage
>>>>>> per call? What difference did you observe?
>>>>>>
>>>>>>
>>>>>> On Fri, Jun 27, 2014 at 12:09 PM, Ali Pey <alipey at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Guillermo,
>>>>>>>
>>>>>>> Short answer: I'm not sure.
>>>>>>>
>>>>>>> Long answer: I think this timer is to detect if there is no rtp.
>>>>>>> Sometimes you would want to know if there is no RTP and then hang up the
>>>>>>> call to avoid possible ghost calls in your system. For instance if there is
>>>>>>> a call between two sip phones and they both get disconnected without
>>>>>>> sending a bye, the call could stay active in freeswitch even though both
>>>>>>> sides are dead.
>>>>>>>
>>>>>>> Another alternative to that, if you think your environment is prone
>>>>>>> to that sort of scenario, is to user session timers to ping the devices in
>>>>>>> a call.
>>>>>>>
>>>>>>> From performance point of view, it makes a big difference.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ali Pey
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 27, 2014 at 11:37 AM, Guillermo Ruiz Camauer <
>>>>>>> grcamauer at gmail.com> wrote:
>>>>>>>
>>>>>>>> Ali,
>>>>>>>>
>>>>>>>>
>>>>>>>> What benefits and or consecuences does setting rtp-timer-name to
>>>>>>>> none and rtp-timeout-sec to zero bring about?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guillermo
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jun 27, 2014 at 9:09 AM, Ali Pey <alipey at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi David,
>>>>>>>>>
>>>>>>>>> Yes, we have done more than 1000 concurrent calls. Here are a few
>>>>>>>>> things we needed to do:
>>>>>>>>>
>>>>>>>>> 1) Add the following to /etc/security/limits.conf
>>>>>>>>> * hard nofile 65536
>>>>>>>>> * soft nofile 65536
>>>>>>>>> root hard nofile 65536
>>>>>>>>> root soft nofile 65536
>>>>>>>>>
>>>>>>>>> freeswitch hard stack 10240
>>>>>>>>> freeswitch soft stack 244
>>>>>>>>>
>>>>>>>>> freeswitch hard nproc unlimited
>>>>>>>>> freeswitch soft nproc unlimited
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2) Add the followings to /etc/init.d/freeswitch
>>>>>>>>> ulimit -c unlimited
>>>>>>>>> ulimit -d unlimited
>>>>>>>>> ulimit -f unlimited
>>>>>>>>> ulimit -i unlimited
>>>>>>>>> ulimit -n 999999
>>>>>>>>> ulimit -q unlimited
>>>>>>>>> ulimit -u unlimited
>>>>>>>>> ulimit -v unlimited
>>>>>>>>> ulimit -x unlimited
>>>>>>>>> ulimit -s 240
>>>>>>>>> ulimit -l unlimited
>>>>>>>>>
>>>>>>>>> 3) Set rtp-timer-name to none and rtp-timeout-sec to zero in your
>>>>>>>>> sip profile.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Ali Pey
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 24, 2014 at 3:42 PM, Guillermo Ruiz Camauer <
>>>>>>>>> grcamauer at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Have you tried spreading the calls over more than 1 SIP profile?
>>>>>>>>>> Each SIP profile gets its own thread.
>>>>>>>>>>
>>>>>>>>>> Also, is this all within your LAN? No firewalls/filters? Are
>>>>>>>>>> you sure you are linking at 1Gbps, full duplex?
>>>>>>>>>>
>>>>>>>>>> Guillermo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Jun 7, 2014 at 2:22 PM, D D <tru083 at yahoo.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I would like to put 500 - 1000 calls into a FS system.
>>>>>>>>>>>
>>>>>>>>>>> I have a Dell R900 server with these properties:
>>>>>>>>>>> 8 processors at 2.40GHz
>>>>>>>>>>> 8GB RAM
>>>>>>>>>>> Gigabit ethernet
>>>>>>>>>>>
>>>>>>>>>>> I use sipp to send RTP streams.
>>>>>>>>>>>
>>>>>>>>>>> I use the FS settings recommended here:
>>>>>>>>>>>
>>>>>>>>>>> https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=7e360a1f-cf3a-4b18-86e3-239b2604c34b&cm_destination=https://wiki.freeswitch.org/wiki/Performance_testing_and_configurations>
>>>>>>>>>>> including the in-memory core.db.
>>>>>>>>>>>
>>>>>>>>>>> I also have used the "-rp" and "-nosql" settings on the FS
>>>>>>>>>>> command-line, and
>>>>>>>>>>> also used the "chrt" command to increase the priority to
>>>>>>>>>>> real-time.
>>>>>>>>>>>
>>>>>>>>>>> In all cases, at around 300 concurrent calls, the RTP
>>>>>>>>>>> inter-packet delay becomes
>>>>>>>>>>> unacceptably high. The voice quality is severely impacted.
>>>>>>>>>>>
>>>>>>>>>>> During the test, the CPU is not overloaded. It is using only
>>>>>>>>>>> about 1.5 of the 8 processors.
>>>>>>>>>>>
>>>>>>>>>>> Are these results expected?
>>>>>>>>>>>
>>>>>>>>>>> Has anyone gone higher than 300 concurrent calls?
>>>>>>>>>>>
>>>>>>>>>>> Is there anything else that I can do with configuration to
>>>>>>>>>>> improve throughput?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=2948a5bf-4e7a-494f-b0b4-63d67ead4ef3&cm_destination=http://www.freeswitchsolutions.com>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=1cff4656-a22a-4280-9595-28d217c01fc7&cm_destination=>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=3403d37b-7a13-4890-94c6-b82682cdc4da&cm_destination=>
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=fc6ba37b-6f57-4942-931e-ea5ebe703cdf&cm_destination=>
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=14e1a431-97a8-40f8-8da9-24343b8d67ae&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=5987c3ed-e2f0-4a49-9f65-5119a1e789cc&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=09500db2-115c-4206-8172-290b5fb8c939&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=f3d270a1-56f0-4bf7-bb18-592d4e25ff24&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=15351782-2a38-489c-ae2e-8d7b24fb220c&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=af63310e-2036-45ff-b22b-243e5a97793d&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=f8766d0f-35c3-4b50-9bfb-72967d772906&cm_destination=http://www.cluecon.com>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=5258f0a2-7e2c-41e6-8e68-8f0915c8af79&cm_destination=http://www.cluecon.com>
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=c2c98dc7-a7fc-4c3c-b5f9-38303b20e724&cm_destination=http://www.cluecon.com>
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=b26d2ad4-f432-488e-9b36-74b4df7d02ff&cm_destination=http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=d1d4884d-f6fe-447f-8948-4ee14a2c08af&cm_destination=http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>>>>>>>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=d56663a2-0974-404f-8e4a-f7dbadcb6b00&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=7683450b-96d4-415b-98ca-902b24c7ecf4&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=f708ad0c-6c6c-4493-913d-e8a695afc0a4&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Guillermo Ruiz Camauer
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=6d9408cf-5e17-425f-91af-681541157fed&cm_destination=http://www.freeswitchsolutions.com>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=910a829b-8c90-45ca-815c-a43a5dcef37f&cm_destination=>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=9c4e1c68-ca53-4d9f-bace-47639a4b089c&cm_destination=>
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=d24569d5-2d53-4685-b82b-73cd0568e865&cm_destination=>
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=a7b59abd-d326-4b72-a317-a0b74899f408&cm_destination=http://www.freeswitch.org>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=ebfb92df-32ea-46ce-910a-17e617c75c76&cm_destination=http://www.freeswitch.org>
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=b06d51a0-d420-473a-9ea6-52549aafa5ea&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=7caae55d-419d-4840-bfaa-e37ec0cff403&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=49697cec-f092-4b8d-a2bd-63fc18c1895a&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=e6e5cc06-94ac-44b0-be3e-e220cd926517&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=78861f6e-1302-4e0b-b3f1-c8ebacef5b99&cm_destination=http://www.cluecon.com>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=e00a1bcb-2ec8-4700-8272-d5fb794c9493&cm_destination=http://www.cluecon.com>
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=275084eb-5627-47f8-a113-fa085c11a59d&cm_destination=http://www.cluecon.com>
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=3c40999c-53be-474a-9fad-99af38eabfed&cm_destination=http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=40f275fa-9031-4526-a37e-8ebf9f4d8693&cm_destination=http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>>>>>>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=0c9205c6-f5fa-4964-81ad-13628f8168ce&cm_destination=http://www.freeswitch.org>
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=c6e7bb42-d00d-46cc-bfdd-e3072b818af8&cm_type=link&cm_link=a5db476c-fa6c-403b-919e-81a6d1ca712f&cm_destination=http://www.freeswitch.org>
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=85154abf-df17-4100-a504-dd1f8cf0169a&cm_destination=http://www.freeswitch.org>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=dc0b0d5f-f5b9-4ada-bc29-3418fe8a6745&cm_destination=http://www.freeswitchsolutions.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=aab60e20-d740-480f-ae80-93ee0abad324&cm_destination=>
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=86c96ed5-9255-437b-90f5-f0214f40847c&cm_destination=>
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=4d20d8f2-85db-497f-9db1-905ff4d5353f&cm_destination=http://www.freeswitch.org>
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=91fae31f-b9d9-4c3d-a698-54863aea7079&cm_destination=http://www.freeswitch.org>
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=9c9ef6ed-d997-47cc-aa5f-c08dc1f42d23&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>> http://wiki.freeswitch.org
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=c4a6915e-c6f9-4b88-b919-296821bbb3c1&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=eb81d653-ee77-49e3-8133-b7572b17fee0&cm_destination=http://www.cluecon.com>
>>>>>>>>> http://www.cluecon.com
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=3241b56c-21f1-4b5a-9d46-6818711ce636&cm_destination=http://www.cluecon.com>
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=d3f79e95-d1e8-4246-ad42-ab7ae863afd3&cm_destination=http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=ff115ca0-36ca-4097-a6c1-af51b18ee434&cm_destination=http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>>>>>
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=06fdb952-dbe4-48e4-b809-8e29f892c424&cm_destination=http://www.freeswitch.org>
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=9ee84770-c29c-42f9-ae12-723654efa6f2&cm_destination=http://www.freeswitch.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillermo Ruiz Camauer
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=d5f3d732-f262-43ea-b034-f84d2005f4ea&cm_destination=http://www.freeswitchsolutions.com>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=64f8d725-d5ba-4a43-9153-c407ceaf7485&cm_destination=>
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=5201f654-4d04-46a6-b69a-934639f3018e&cm_destination=>
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=9e09fb9d-93f7-4b58-9189-6947ff0d4a24&cm_destination=http://www.freeswitch.org>
>>>>>>>> http://www.freeswitch.org
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=f6ce4116-7a32-4717-abf4-20f38a3eb407&cm_destination=http://www.freeswitch.org>
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=3b20336f-1240-4b4c-94e3-0987baedb35f&cm_destination=http://wiki.freeswitch.org>
>>>>>>>> http://wiki.freeswitch.org
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=13b80518-e686-4845-a449-d3472b96676e&cm_destination=http://wiki.freeswitch.org>
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=7be0e4e2-5473-4fb0-9b8f-7cee69e7a044&cm_destination=http://www.cluecon.com>
>>>>>>>> http://www.cluecon.com
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=21ba895e-55bd-4f2a-998d-47ecac90ddb3&cm_destination=http://www.cluecon.com>
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=39882397-3237-4527-accc-a1457260bd67&cm_destination=http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=5d951800-48e9-4554-9acf-fc5f5f2763a0&cm_destination=http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>>>>
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=06ef5b8e-56d7-496f-8fb9-e435e4538bf1&cm_destination=http://www.freeswitch.org>
>>>>>>>> http://www.freeswitch.org
>>>>>>>> <https://contactmonkey.com/api/tracker?cm_session=80e5e316-65ef-4f96-a5fb-b29e485f59b2&cm_type=link&cm_link=14e61f72-8fe8-4e48-93bf-c487c8c4f185&cm_destination=http://www.freeswitch.org>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://wiki.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Guillermo Ruiz Camauer
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Guillermo Ruiz Camauer
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
--
Guillermo Ruiz Camauer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140630/207100a8/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list