[Freeswitch-users] Leg stuck in a conference

Oleg Stolyar olegstolyar at gmail.com
Thu Sep 25 01:12:11 MSD 2014 

Sure, here it is.  I still have two more stuck conferences if you need me
to try something else.  BTW, the last one is from a different client - not
WebRTC, so it's likely not client specific.

freeswitch at internal> fsctl loglevel debug
+OK log level: DEBUG [7]

freeswitch at internal> console loglevel debug
+OK log level debug [7]
+OK console log level set to DEBUG

freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
all
OK kicked 21698

2014-09-24 21:07:33.206950 [DEBUG] mod_conference.c:5619 Send signal
sofia/agent/anonymous at anonymous.invalid [BREAK]
freeswitch at internal>
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch at internal
>
freeswitch at internal>
freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
all
OK kicked 21698

2014-09-24 21:07:36.886946 [DEBUG] mod_conference.c:5619 Send signal
sofia/agent/anonymous at anonymous.invalid [BREAK]
freeswitch at internal>
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch at internal
>
freeswitch at internal>
freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
21698
Non-Existant ID 21698

freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
21698


freeswitch at internal> uuid_kill 42e7c025-b2ad-4b15-ad58-bbf7609d73d7
+OK

2014-09-24 21:08:33.526948 [NOTICE] switch_ivr.c:3704 Hangup
sofia/agent/anonymous at anonymous.invalid [CS_EXECUTE] [NORMAL_CLEARING]
2014-09-24 21:08:33.546949 [DEBUG] switch_channel.c:3216 Send signal
sofia/agent/anonymous at anonymous.invalid [KILL]
2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:1387 Send signal
sofia/agent/anonymous at anonymous.invalid [BREAK]
2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:4491 Channel leaving
conference, cause: NORMAL_CLEARING
2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:8702
sofia/agent/anonymous at anonymous.invalid skip receive message [UNBRIDGE]
(channel is hungup already)
2014-09-24 21:08:33.546949 [DEBUG] switch_core_codec.c:246
sofia/agent/anonymous at anonymous.invalid Restore previous codec PCMU:0.
2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:2875
sofia/agent/anonymous at anonymous.invalid skip receive message
[APPLICATION_EXEC_COMPLETE] (channel is hungup already)
2014-09-24 21:08:33.546949 [DEBUG] sofia.c:1298 Channel is already hungup.
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:530
(sofia/agent/anonymous at anonymous.invalid) State EXECUTE going to sleep
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:467
(sofia/agent/anonymous at anonymous.invalid) Running State Change CS_HANGUP
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:730
(sofia/agent/anonymous at anonymous.invalid) Callstate Change ACTIVE -> HANGUP
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732
(sofia/agent/anonymous at anonymous.invalid) State HANGUP
2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:413 Channel
sofia/agent/anonymous at anonymous.invalid hanging up, cause: NORMAL_CLEARING
2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:465 Sending BYE to
sofia/agent/anonymous at anonymous.invalid
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:58
sofia/agent/anonymous at anonymous.invalid Standard HANGUP, cause:
NORMAL_CLEARING
2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732
(sofia/agent/anonymous at anonymous.invalid) State HANGUP going to sleep
2014-09-24 21:08:33.546949 [DEBUG] mod_curl.c:166 method: get, url:
http://<mywebapp>answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7,
content-type: (null)
2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3074 Write Lock ON
2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3077 Write Lock OFF
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:663 Hangup
Command with no Session curl(http://
<mywebapp>/answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7):
conf-5d7b86e0-3ea5-11e4-b722-1231393fb01c
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:499
(sofia/agent/anonymous at anonymous.invalid) State Change CS_HANGUP ->
CS_REPORTING
2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal
sofia/agent/anonymous at anonymous.invalid [BREAK]
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:467
(sofia/agent/anonymous at anonymous.invalid) Running State Change CS_REPORTING
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818
(sofia/agent/anonymous at anonymous.invalid) State REPORTING
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:102
sofia/agent/anonymous at anonymous.invalid Standard REPORTING, cause:
NORMAL_CLEARING
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818
(sofia/agent/anonymous at anonymous.invalid) State REPORTING going to sleep
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:493
(sofia/agent/anonymous at anonymous.invalid) State Change CS_REPORTING ->
CS_DESTROY
2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal
sofia/agent/anonymous at anonymous.invalid [BREAK]
2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1614 Session 28810
(sofia/agent/anonymous at anonymous.invalid) Locked, Waiting on external
entities
2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1632 Session
28810 (sofia/agent/anonymous at anonymous.invalid) Ended
2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1636 Close
Channel sofia/agent/anonymous at anonymous.invalid [CS_DESTROY]
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:618
(sofia/agent/anonymous at anonymous.invalid) Callstate Change HANGUP -> DOWN
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:621
(sofia/agent/anonymous at anonymous.invalid) Running State Change CS_DESTROY
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631
(sofia/agent/anonymous at anonymous.invalid) State DESTROY
2014-09-24 21:08:33.566948 [DEBUG] mod_sofia.c:323
sofia/agent/anonymous at anonymous.invalid SOFIA DESTROY
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:109
sofia/agent/anonymous at anonymous.invalid Standard DESTROY
2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631
(sofia/agent/anonymous at anonymous.invalid) State DESTROY going to sleep


freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
21698
Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found

freeswitch at internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick
all
Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found


On Wed, Sep 24, 2014 at 1:33 PM, Anthony Minessale <
anthony.minessale at gmail.com> wrote:

> Do you know what is on the other side of these clients?
> I get a bad vibe when i see cent6 in reports of odd behaviors as you know
> ;)
>
> So if you uuid_kill it, then it exits?  The only logical explanation would
> be that its stuck doing recvfrom.  But in that case the uuid_kill shouldn't
> work either.
>
> do these 2 commands first then a few kick commands then the uuid_kill  to
> compare
>
> fsctl loglevel debug
> console loglevel debug
>
>
>
>
> On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar at gmail.com>
> wrote:
>
>> Hi Anthony,
>>
>> One more piece of information.
>>
>> If I do "kick all" it returns "OK kicked 21698"
>>
>> But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
>>
>>
>> On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar at gmail.com>
>> wrote:
>>
>>> Yes, exactly
>>>
>>> On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale <
>>> anthony.minessale at gmail.com> wrote:
>>>
>>>> Doing more kick commands results in more break messages but it never
>>>> works even with subsequent calls to kick?
>>>>
>>>>
>>>>
>>>> On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar at gmail.com>
>>>> wrote:
>>>>
>>>>> Version is: FreeSWITCH Version
>>>>> 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29
>>>>> 19:46:31Z 64bit)
>>>>>
>>>>> Since I cannot reproduce this at will, I would need to upgrade my
>>>>> production to the latest master to try to reproduce it there.
>>>>>
>>>>> The client is WebRTC with JsSip on Chrome 34.
>>>>>
>>>>> I do not enable proxy_media and in fact there is no reference to
>>>>> proxy_media anywhere in my config.
>>>>> Similarly, I did not change anything related to timers but if you;d
>>>>> like me to check a specific setting, I'll be happy to.
>>>>>
>>>>> I still have that FS instance running, so if there is anything else I
>>>>> can do to help track this down, please let me know.
>>>>>
>>>>>
>>>>> On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale <
>>>>> anthony.minessale at gmail.com> wrote:
>>>>>
>>>>>> It looks like its in recvfrom.  What version of FS is this and what
>>>>>> type of call client etc is this?
>>>>>> Do you have proxy_media on or timers disabled?
>>>>>>
>>>>>> It doesn't seem like this should be a likely scenario based on the
>>>>>> description and the code I'm looking at.  Have you tried to reproduce it
>>>>>> with master?
>>>>>> The break signal should cause the read to come up to the conference
>>>>>> app where it will detect that its done.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar at gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I'll do you one better :-)
>>>>>>>
>>>>>>> Here is the complete output file of "thread apply all bt"
>>>>>>>
>>>>>>> https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale <
>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>
>>>>>>>> Are there conference_loop_input threads to go with those?
>>>>>>>> search for other threads with one of the sessions in it e.g.
>>>>>>>> 0x7f3f503a8b48
>>>>>>>>
>>>>>>>> On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <
>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Anthony!
>>>>>>>>> I got two such stuck legs right now.  Here are their threads.
>>>>>>>>>
>>>>>>>>> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)):
>>>>>>>>> #0  0x0000003d8d207c65 in pthread_join () from
>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>> #1  0x00007f3f712b9db9 in apr_thread_join () from
>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>> #2  0x00007f3f71186d2c in switch_thread_join
>>>>>>>>> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171
>>>>>>>>> #3  0x00007f3f667b233c in conference_loop_output
>>>>>>>>> (member=0x7f3f574fa540) at mod_conference.c:4481
>>>>>>>>> #4  0x00007f3f667c1a4c in conference_function
>>>>>>>>> (session=0x7f3f503a8b48, data=0x7f3f681cbd10
>>>>>>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default+flags{join-only}")
>>>>>>>>> at mod_conference.c:8695
>>>>>>>>> #5  0x00007f3f711c23e7 in switch_core_session_exec
>>>>>>>>> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920,
>>>>>>>>> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default
>>>>>>>>> +flags{join-only}")
>>>>>>>>>     at src/switch_core_session.c:2862
>>>>>>>>> #6  0x00007f3f711c1c85 in
>>>>>>>>> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48,
>>>>>>>>> app=0x7f3f681cbd00 "conference",
>>>>>>>>>     arg=0x7f3f681cbd10
>>>>>>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c at default+flags{join-only}",
>>>>>>>>> flags=0x0) at src/switch_core_session.c:2737
>>>>>>>>> #7  0x00007f3f711c3c3c in switch_core_standard_on_execute
>>>>>>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281
>>>>>>>>> #8  0x00007f3f711c4de0 in switch_core_session_run
>>>>>>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530
>>>>>>>>> #9  0x00007f3f711be45c in switch_core_session_thread
>>>>>>>>> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at
>>>>>>>>> src/switch_core_session.c:1597
>>>>>>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker
>>>>>>>>> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at
>>>>>>>>> src/switch_core_session.c:1689
>>>>>>>>> #11 0x00007f3f712b9b7d in dummy_worker () from
>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>> #12 0x0000003d8d20683d in start_thread () from
>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)):
>>>>>>>>> #0  0x0000003d8d207c65 in pthread_join () from
>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>> #1  0x00007f3f712b9db9 in apr_thread_join () from
>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>> #2  0x00007f3f71186d2c in switch_thread_join
>>>>>>>>> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171
>>>>>>>>> #3  0x00007f3f667b233c in conference_loop_output
>>>>>>>>> (member=0x7f3f575ae540) at mod_conference.c:4481
>>>>>>>>> #4  0x00007f3f667c1a4c in conference_function
>>>>>>>>> (session=0x7f3f50087998, data=0x2152a38
>>>>>>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default+flags{join-only}")
>>>>>>>>> at mod_conference.c:8695
>>>>>>>>> #5  0x00007f3f711c23e7 in switch_core_session_exec
>>>>>>>>> (session=0x7f3f50087998, application_interface=0x7f3f600e1920,
>>>>>>>>> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default
>>>>>>>>> +flags{join-only}")
>>>>>>>>>     at src/switch_core_session.c:2862
>>>>>>>>> #6  0x00007f3f711c1c85 in
>>>>>>>>> switch_core_session_execute_application_get_flags (session=0x7f3f50087998,
>>>>>>>>> app=0x2152a28 "conference", arg=0x2152a38
>>>>>>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e at default+flags{join-only}",
>>>>>>>>>     flags=0x0) at src/switch_core_session.c:2737
>>>>>>>>> #7  0x00007f3f711c3c3c in switch_core_standard_on_execute
>>>>>>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281
>>>>>>>>> #8  0x00007f3f711c4de0 in switch_core_session_run
>>>>>>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530
>>>>>>>>> #9  0x00007f3f711be45c in switch_core_session_thread
>>>>>>>>> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597
>>>>>>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker
>>>>>>>>> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689
>>>>>>>>> #11 0x00007f3f712b9b7d in dummy_worker () from
>>>>>>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1
>>>>>>>>> #12 0x0000003d8d20683d in start_thread () from
>>>>>>>>> /lib64/libpthread.so.0
>>>>>>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
>>>>>>>>>
>>>>>>>>> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale <
>>>>>>>>> anthony.minessale at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> It may be trapped inside some blocking call from within the
>>>>>>>>>> conference.
>>>>>>>>>> Your best bet would be to catch it in that state, do the kick,
>>>>>>>>>> when it doesn't work gcore the process and open it in gdb and get "thread
>>>>>>>>>> apply all bt" then you can see the stack trace for that session and see
>>>>>>>>>> which code its blocking on.
>>>>>>>>>>
>>>>>>>>>> The BREAK signal you show there is designed to pull it out of
>>>>>>>>>> most blocking situations so it must be something particular.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <
>>>>>>>>>> olegstolyar at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Guys,
>>>>>>>>>>>
>>>>>>>>>>> I wonder if anyone else ran into this situation:  Very rarely I
>>>>>>>>>>> have a conference participant stuck in a conference.  When I try to run
>>>>>>>>>>> kick command on that conference, it returns "OK kicked <member ID>" but
>>>>>>>>>>> nothing happens.  In the logs there is only this:
>>>>>>>>>>>
>>>>>>>>>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous at anonymous.invalid [BREAK]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
>>>>>>>>>>>
>>>>>>>>>>> Any suggestions?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>>
>>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>>
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>
>>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>>>>>>>
>>>>>>>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>>>>>>>> http://twitter.com/FreeSWITCH
>>>>>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>>>>>> <http://freeswitch.org/g+>*
>>>>>>>>>>
>>>>>>>>>> ClueCon Weekly Development Call
>>>>>>>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _________________________________________________________________________
>>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>>> consulting at freeswitch.org
>>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>>
>>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>>> http://www.cluecon.com
>>>>>>>>>>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>
>>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _________________________________________________________________________
>>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>>> consulting at freeswitch.org
>>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>>
>>>>>>>>> Official FreeSWITCH Sites
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>> http://confluence.freeswitch.org
>>>>>>>>> http://www.cluecon.com
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>
>>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>>> UNSUBSCRIBE:
>>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>>> http://www.freeswitch.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>>>>>
>>>>>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>>>>>> http://twitter.com/FreeSWITCH
>>>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>>>> <http://freeswitch.org/g+>*
>>>>>>>>
>>>>>>>> ClueCon Weekly Development Call
>>>>>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________________________________
>>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>>> consulting at freeswitch.org
>>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>>
>>>>>>>> Official FreeSWITCH Sites
>>>>>>>> http://www.freeswitch.org
>>>>>>>> http://confluence.freeswitch.org
>>>>>>>> http://www.cluecon.com
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> FreeSWITCH-users mailing list
>>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>>> UNSUBSCRIBE:
>>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>>> http://www.freeswitch.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________________________________
>>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>>> consulting at freeswitch.org
>>>>>>> http://www.freeswitchsolutions.com
>>>>>>>
>>>>>>> Official FreeSWITCH Sites
>>>>>>> http://www.freeswitch.org
>>>>>>> http://confluence.freeswitch.org
>>>>>>> http://www.cluecon.com
>>>>>>>
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> FreeSWITCH-users mailing list
>>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>> UNSUBSCRIBE:
>>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>> http://www.freeswitch.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>>>
>>>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>>>> http://twitter.com/FreeSWITCH
>>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>>>> <http://freeswitch.org/g+>*
>>>>>>
>>>>>> ClueCon Weekly Development Call
>>>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>>>
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://confluence.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://confluence.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> 
>>>>> 
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>>>>
>>>> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
>>>> http://twitter.com/FreeSWITCH
>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
>>>> <http://freeswitch.org/g+>*
>>>>
>>>> ClueCon Weekly Development Call
>>>> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>>>>
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://confluence.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> 
>>>> 
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://confluence.freeswitch.org
>> http://www.cluecon.com
>>
>> 
>> 
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>
>
>
> --
> Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬
>
> ☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞
> http://twitter.com/FreeSWITCH
> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
> <http://freeswitch.org/g+>*
>
> ClueCon Weekly Development Call
> ☎ sip:888 at conference.freeswitch.org  ☎ +19193869900
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> 
> 
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140924/1dc24ea1/attachment-0001.html

Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users mailing list