[Freeswitch-users] Leg stuck in a conference
Oleg Stolyar
olegstolyar at gmail.com
Thu Sep 25 01:35:47 MSD 2014
It's an AWS instance. No firewall rules that would prevent this that I
know of.
Since I cannot repro this at will, the only way to reproduce it on Debian
is to put it into production. Unfortunately I cannot do that - there are
too many dependencies.
On Wed, Sep 24, 2014 at 2:24 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:
> The KILL sig is more severe than break, it calls shutdown on the socket.
> I think your box has some kernel params causing blocking calls to
> recvfrom, calling the break should be sending a udp packet to itself to
> wake it up even when blocking.
> Do you have any firewall rules that might block it from sending itself RTP
> packets?
>
> It'd be nice if you could reproduce this on debian stable!
>
> On Wed, Sep 24, 2014 at 4:12 PM, Oleg Stolyar <olegstolyar at gmail.com>
> wrote:
>
>> Sure, here it is. I still have two more stuck conferences if you need me
>> to try something else. BTW, the last one is from a different client - not
>> WebRTC, so it's likely not client specific.
>>
>> freeswitch at internal> fsctl loglevel debug
>> +OK log level: DEBUG [7]
>>
>> freeswitch at internal> console loglevel debug
>> +OK log level debug [7]
>> +OK console log level set to DEBUG
>>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick all
>> OK kicked 21698
>>
>> 2014-09-24 21:07:33.206950 [DEBUG] mod_conference.c:5619 Send signal
>> sofia/agent/anonymous at anonymous.invalid [BREAK]
>> freeswitch at internal>
>> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch at internal
>> >
>> freeswitch at internal>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick all
>> OK kicked 21698
>>
>> 2014-09-24 21:07:36.886946 [DEBUG] mod_conference.c:5619 Send signal
>> sofia/agent/anonymous at anonymous.invalid [BREAK]
>> freeswitch at internal>
>> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch at internal
>> >
>> freeswitch at internal>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick 21698
>> Non-Existant ID 21698
>>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick 21698
>>
>>
>> freeswitch at internal> uuid_kill 42e7c025-b2ad-4b15-ad58-bbf7609d73d7
>> +OK
>>
>> 2014-09-24 21:08:33.526948 [NOTICE] switch_ivr.c:3704 Hangup
>> sofia/agent/anonymous at anonymous.invalid [CS_EXECUTE] [NORMAL_CLEARING]
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_channel.c:3216 Send signal
>> sofia/agent/anonymous at anonymous.invalid [KILL]
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:1387 Send signal
>> sofia/agent/anonymous at anonymous.invalid [BREAK]
>> 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:4491 Channel leaving
>> conference, cause: NORMAL_CLEARING
>> 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:8702
>> sofia/agent/anonymous at anonymous.invalid skip receive message [UNBRIDGE]
>> (channel is hungup already)
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_codec.c:246
>> sofia/agent/anonymous at anonymous.invalid Restore previous codec PCMU:0.
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:2875
>> sofia/agent/anonymous at anonymous.invalid skip receive message
>> [APPLICATION_EXEC_COMPLETE] (channel is hungup already)
>> 2014-09-24 21:08:33.546949 [DEBUG] sofia.c:1298 Channel is already hungup.
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:530
>> (sofia/agent/anonymous at anonymous.invalid) State EXECUTE going to sleep
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:467
>> (sofia/agent/anonymous at anonymous.invalid) Running State Change CS_HANGUP
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:730
>> (sofia/agent/anonymous at anonymous.invalid) Callstate Change ACTIVE ->
>> HANGUP
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732
>> (sofia/agent/anonymous at anonymous.invalid) State HANGUP
>> 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:413 Channel
>> sofia/agent/anonymous at anonymous.invalid hanging up, cause:
>> NORMAL_CLEARING
>> 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:465 Sending BYE to
>> sofia/agent/anonymous at anonymous.invalid
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:58
>> sofia/agent/anonymous at anonymous.invalid Standard HANGUP, cause:
>> NORMAL_CLEARING
>> 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732
>> (sofia/agent/anonymous at anonymous.invalid) State HANGUP going to sleep
>> 2014-09-24 21:08:33.546949 [DEBUG] mod_curl.c:166 method: get, url:
>> http://<mywebapp>answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7,
>> content-type: (null)
>> 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3074 Write Lock ON
>> 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3077 Write Lock OFF
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:663 Hangup
>> Command with no Session curl(http://
>> <mywebapp>/answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7):
>> conf-5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:499
>> (sofia/agent/anonymous at anonymous.invalid) State Change CS_HANGUP ->
>> CS_REPORTING
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal
>> sofia/agent/anonymous at anonymous.invalid [BREAK]
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:467
>> (sofia/agent/anonymous at anonymous.invalid) Running State Change
>> CS_REPORTING
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818
>> (sofia/agent/anonymous at anonymous.invalid) State REPORTING
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:102
>> sofia/agent/anonymous at anonymous.invalid Standard REPORTING, cause:
>> NORMAL_CLEARING
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818
>> (sofia/agent/anonymous at anonymous.invalid) State REPORTING going to sleep
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:493
>> (sofia/agent/anonymous at anonymous.invalid) State Change CS_REPORTING ->
>> CS_DESTROY
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal
>> sofia/agent/anonymous at anonymous.invalid [BREAK]
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1614 Session
>> 28810 (sofia/agent/anonymous at anonymous.invalid) Locked, Waiting on
>> external entities
>> 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1632 Session
>> 28810 (sofia/agent/anonymous at anonymous.invalid) Ended
>> 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1636 Close
>> Channel sofia/agent/anonymous at anonymous.invalid [CS_DESTROY]
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:618
>> (sofia/agent/anonymous at anonymous.invalid) Callstate Change HANGUP -> DOWN
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:621
>> (sofia/agent/anonymous at anonymous.invalid) Running State Change CS_DESTROY
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631
>> (sofia/agent/anonymous at anonymous.invalid) State DESTROY
>> 2014-09-24 21:08:33.566948 [DEBUG] mod_sofia.c:323
>> sofia/agent/anonymous at anonymous.invalid SOFIA DESTROY
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:109
>> sofia/agent/anonymous at anonymous.invalid Standard DESTROY
>> 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631
>> (sofia/agent/anonymous at anonymous.invalid) State DESTROY going to sleep
>>
>>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick 21698
>> Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
>>
>> freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c
>> kick all
>> Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
>>
>>
>> On Wed, Sep 24, 2014 at 1:33 PM, Anthony Minessale <
>> anthony.minessale at gmail.com> wrote:
>>
>>> Do you know what is on the other side of these clients?
>>> I get a bad vibe when i see cent6 in reports of odd behaviors as you
>>> know ;)
>>>
>>> So if you uuid_kill it, then it exits? The only logical explanation
>>> would be that its stuck doing recvfrom. But in that case the uuid_kill
>>> shouldn't work either.
>>>
>>> do these 2 commands first then a few kick commands then the uuid_kill
>>> to compare
>>>
>>> fsctl loglevel debug
>>> console loglevel debug
>>>
>>>
>>>
>>>
>>> On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>> wrote:
>>>
>>>> Hi Anthony,
>>>>
>>>> One more piece of information.
>>>>
>>>> If I do "kick all" it returns "OK kicked 21698"
>>>>
>>>> But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
>>>>
>>>>
>>>> On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>>> wrote:
>>>>
>>>>> Yes, exactly
>>>>>
>>>>> On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale <
>>>>> anthony.minessale at gmail.com> wrote:
>>>>>
>>>>>> Doing more kick commands results in more break messages but it never
>>>>>> works even with subsequent calls to kick?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Version is: FreeSWITCH Version
>>>>>>> 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29
>>>>>>> 19:46:31Z 64bit)
>>>>>>>
>>>>>>> Since I cannot reproduce this at will, I would need to upgrade my
>>>>>>> production to the latest master to try to reproduce it there.
>>>>>>>
>>>>>>> The client is WebRTC with JsSip on Chrome 34.
>>>>>>>
>>>>>>> I do not enable proxy_media and in fact there is no reference to
>>>>>>> proxy_media anywhere in my config.
>>>>>>> Similarly, I did not change anything related to timers but if you;d
>>>>>>> like me to check a specific setting, I'll be happy to.
>>>>>>>
>>>>>>> I still have that FS instance running, so if there is anything else
>>>>>>> I can do to help track this down, please let me know.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale <
>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>
>>>>>>>> It looks like its in recvfrom. What version of FS is this and what
>>>>>>>> type of call client etc is this?
>>>>>>>> Do you have proxy_media on or timers disabled?
>>>>>>>>
>>>>>>>> It doesn't seem like this should be a likely scenario based on the
>>>>>>>> description and the code I'm looking at. Have you tried to reproduce it
>>>>>>>> with master?
>>>>>>>> The break signal should cause the read to come up to the conference
>>>>>>>> app where it will detect that its done.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <
>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I'll do you one better :-)
>>>>>>>>>
>>>>>>>>> Here is the complete output file of "thread apply all bt"
>>>>>>>>>
>>>>>>>>> https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale <
>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Are there conference_loop_input threads to go with those?
>>>>>>>>>> search for other threads with one of the sessions in it e.g.
>>>>>>>>>> 0x7f3f503a8b48
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <
>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Anthony!
>>>>>>>>>>> I got two such stuck legs right now. Here are their threads.
>>>>>>>>>>>
>>>>>>>>>>> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)):
>>>>>>>>>>> #0 0x0000003d8d207c65 in pthread_join () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #1 0x00007f3f712b9db9 in apr_thread_join () from
>>>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>>>> #2 0x00007f3f71186d2c in switch_thread_join
>>>>>>>>>>> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171
>>>>>>>>>>> #3 0x00007f3f667b233c in conference_loop_output
>>>>>>>>>>> (member=0x7f3f574fa540) at mod_conference.c:4481
>>>>>>>>>>> #4 0x00007f3f667c1a4c in conference_function
>>>>>>>>>>> (session=0x7f3f503a8b48, data=0x7f3f681cbd10
>>>>>>>>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default+flags{join-only}")
>>>>>>>>>>> at mod_conference.c:8695
>>>>>>>>>>> #5 0x00007f3f711c23e7 in switch_core_session_exec
>>>>>>>>>>> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920,
>>>>>>>>>>> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default
>>>>>>>>>>> +flags{join-only}")
>>>>>>>>>>> at src/switch_core_session.c:2862
>>>>>>>>>>> #6 0x00007f3f711c1c85 in
>>>>>>>>>>> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48,
>>>>>>>>>>> app=0x7f3f681cbd00 "conference",
>>>>>>>>>>> arg=0x7f3f681cbd10
>>>>>>>>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default+flags{join-only}",
>>>>>>>>>>> flags=0x0) at src/switch_core_session.c:2737
>>>>>>>>>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute
>>>>>>>>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281
>>>>>>>>>>> #8 0x00007f3f711c4de0 in switch_core_session_run
>>>>>>>>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530
>>>>>>>>>>> #9 0x00007f3f711be45c in switch_core_session_thread
>>>>>>>>>>> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at
>>>>>>>>>>> src/switch_core_session.c:1597
>>>>>>>>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker
>>>>>>>>>>> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at
>>>>>>>>>>> src/switch_core_session.c:1689
>>>>>>>>>>> #11 0x00007f3f712b9b7d in dummy_worker () from
>>>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>>>> #12 0x0000003d8d20683d in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)):
>>>>>>>>>>> #0 0x0000003d8d207c65 in pthread_join () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #1 0x00007f3f712b9db9 in apr_thread_join () from
>>>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>>>> #2 0x00007f3f71186d2c in switch_thread_join
>>>>>>>>>>> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171
>>>>>>>>>>> #3 0x00007f3f667b233c in conference_loop_output
>>>>>>>>>>> (member=0x7f3f575ae540) at mod_conference.c:4481
>>>>>>>>>>> #4 0x00007f3f667c1a4c in conference_function
>>>>>>>>>>> (session=0x7f3f50087998, data=0x2152a38
>>>>>>>>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default+flags{join-only}")
>>>>>>>>>>> at mod_conference.c:8695
>>>>>>>>>>> #5 0x00007f3f711c23e7 in switch_core_session_exec
>>>>>>>>>>> (session=0x7f3f50087998, application_interface=0x7f3f600e1920,
>>>>>>>>>>> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default
>>>>>>>>>>> +flags{join-only}")
>>>>>>>>>>> at src/switch_core_session.c:2862
>>>>>>>>>>> #6 0x00007f3f711c1c85 in
>>>>>>>>>>> switch_core_session_execute_application_get_flags (session=0x7f3f50087998,
>>>>>>>>>>> app=0x2152a28 "conference", arg=0x2152a38
>>>>>>>>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default+flags{join-only}",
>>>>>>>>>>> flags=0x0) at src/switch_core_session.c:2737
>>>>>>>>>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute
>>>>>>>>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281
>>>>>>>>>>> #8 0x00007f3f711c4de0 in switch_core_session_run
>>>>>>>>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530
>>>>>>>>>>> #9 0x00007f3f711be45c in switch_core_session_thread
>>>>>>>>>>> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597
>>>>>>>>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker
>>>>>>>>>>> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689
>>>>>>>>>>> #11 0x00007f3f712b9b7d in dummy_worker () from
>>>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>>>> #12 0x0000003d8d20683d in start_thread () from
>>>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale <
>>>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> It may be trapped inside some blocking call from within the
>>>>>>>>>>>> conference.
>>>>>>>>>>>> Your best bet would be to catch it in that state, do the kick,
>>>>>>>>>>>> when it doesn't work gcore the process and open it in gdb and get "thread
>>>>>>>>>>>> apply all bt" then you can see the stack trace for that session and see
>>>>>>>>>>>> which code its blocking on.
>>>>>>>>>>>>
>>>>>>>>>>>> The BREAK signal you show there is designed to pull it out of
>>>>>>>>>>>> most blocking situations so it must be something particular.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <
>>>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I wonder if anyone else ran into this situation: Very rarely
>>>>>>>>>>>>> I have a conference participant stuck in a conference. When I try to run
>>>>>>>>>>>>> kick command on that conference, it returns "OK kicked <member ID>" but
>>>>>>>>>>>>> nothing happens. In the logs there is only this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous at anonymous.invalid [BREAK]
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any suggestions?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>>>>>>>>>>
>>>>>>>>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>>>>>>>>>>> http://twitter.com/FreeSWITCH
>>>>>>>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>>>>>>>> <http://freeswitch.org/g+>*
>>>>>>>>>>>>
>>>>>>>>>>>> ClueCon Weekly Development Call
>>>>>>>>>>>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>>
>>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>>>>>>>>
>>>>>>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>>>>>>>>> http://twitter.com/FreeSWITCH
>>>>>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>>>>>> <http://freeswitch.org/g+>*
>>>>>>>>>>
>>>>>>>>>> ClueCon Weekly Development Call
>>>>>>>>>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>>>>>>
>>>>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>>>>>>> http://twitter.com/FreeSWITCH
>>>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>>>> <http://freeswitch.org/g+>*
>>>>>>>>
>>>>>>>> ClueCon Weekly Development Call
>>>>>>>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>>>>
>>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>>>>> http://twitter.com/FreeSWITCH
>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>> <http://freeswitch.org/g+>*
>>>>>>
>>>>>> ClueCon Weekly Development Call
>>>>>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>>
>>>>
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>>>
>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
>>> http://twitter.com/FreeSWITCH
>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>> <http://freeswitch.org/g+>*
>>>
>>> ClueCon Weekly Development Call
>>> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://confluence.freeswitch.org
>>> http://www.cluecon.com
>>>
>>>
>>>
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>>
>>
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
>
> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org ☎ +19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
>
>
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140924/9ec8b776/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list