Leg stuck in a conference
Guys, I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this: db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. Any suggestions?
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on. The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular. On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Thanks Anthony! I got two such stuck legs right now. Here are their threads. Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48 On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
I'll do you one better :-) Here is the complete output file of "thread apply all bt" https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled? It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done. On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit) Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there. The client is WebRTC with JsSip on Chrome 34. I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to. I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know. On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Guys,
I wonder if anyone else ran into this situation: Very rarely I have a conference participant stuck in a conference. When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens. In the logs there is only this:
db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]
As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.
Any suggestions?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick? On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It may be trapped inside some blocking call from within the conference. Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.
The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.
On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com > wrote:
> Guys, > > I wonder if anyone else ran into this situation: Very rarely I have > a conference participant stuck in a conference. When I try to run kick > command on that conference, it returns "OK kicked <member ID>" but nothing > happens. In the logs there is only this: > > db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] > > > As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. > > Any suggestions? > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Yes, exactly On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Thanks Anthony! I got two such stuck legs right now. Here are their threads.
Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference", arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 #3 0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481 #4 0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695 #5 0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default +flags{join-only}") at src/switch_core_session.c:2862 #6 0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737 #7 0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 #8 0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 #9 0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 #11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1 #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6
On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
> It may be trapped inside some blocking call from within the > conference. > Your best bet would be to catch it in that state, do the kick, when > it doesn't work gcore the process and open it in gdb and get "thread apply > all bt" then you can see the stack trace for that session and see which > code its blocking on. > > The BREAK signal you show there is designed to pull it out of most > blocking situations so it must be something particular. > > > > > On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < > olegstolyar@gmail.com> wrote: > >> Guys, >> >> I wonder if anyone else ran into this situation: Very rarely I >> have a conference participant stuck in a conference. When I try to run >> kick command on that conference, it returns "OK kicked <member ID>" but >> nothing happens. In the logs there is only this: >> >> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >> >> >> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >> >> Any suggestions? >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ > <http://freeswitch.org/g+>* > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org ☎ +19193869900 > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Anthony, One more piece of information. If I do "kick all" it returns "OK kicked 21698" But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log. On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, exactly
On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Are there conference_loop_input threads to go with those? search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48
On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com > wrote:
> Thanks Anthony! > I got two such stuck legs right now. Here are their threads. > > Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): > #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00007f3f712b9db9 in apr_thread_join () from > /usr/local/freeswitch/lib/libfreeswitch.so.1 > #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, > thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 > #3 0x00007f3f667b233c in conference_loop_output > (member=0x7f3f574fa540) at mod_conference.c:4481 > #4 0x00007f3f667c1a4c in conference_function > (session=0x7f3f503a8b48, data=0x7f3f681cbd10 > "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at > mod_conference.c:8695 > #5 0x00007f3f711c23e7 in switch_core_session_exec > (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, > arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default > +flags{join-only}") > at src/switch_core_session.c:2862 > #6 0x00007f3f711c1c85 in > switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, > app=0x7f3f681cbd00 "conference", > arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", > flags=0x0) at src/switch_core_session.c:2737 > #7 0x00007f3f711c3c3c in switch_core_standard_on_execute > (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 > #8 0x00007f3f711c4de0 in switch_core_session_run > (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 > #9 0x00007f3f711be45c in switch_core_session_thread > (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at > src/switch_core_session.c:1597 > #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker > (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at > src/switch_core_session.c:1689 > #11 0x00007f3f712b9b7d in dummy_worker () from > /usr/local/freeswitch/lib/libfreeswitch.so.1 > #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 > #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 > > > Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): > #0 0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00007f3f712b9db9 in apr_thread_join () from > /usr/local/freeswitch/lib/libfreeswitch.so.1 > #2 0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, > thd=0x215a358) at src/switch_apr.c:1171 > #3 0x00007f3f667b233c in conference_loop_output > (member=0x7f3f575ae540) at mod_conference.c:4481 > #4 0x00007f3f667c1a4c in conference_function > (session=0x7f3f50087998, data=0x2152a38 > "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at > mod_conference.c:8695 > #5 0x00007f3f711c23e7 in switch_core_session_exec > (session=0x7f3f50087998, application_interface=0x7f3f600e1920, > arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default > +flags{join-only}") > at src/switch_core_session.c:2862 > #6 0x00007f3f711c1c85 in > switch_core_session_execute_application_get_flags (session=0x7f3f50087998, > app=0x2152a28 "conference", arg=0x2152a38 > "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", > flags=0x0) at src/switch_core_session.c:2737 > #7 0x00007f3f711c3c3c in switch_core_standard_on_execute > (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 > #8 0x00007f3f711c4de0 in switch_core_session_run > (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 > #9 0x00007f3f711be45c in switch_core_session_thread > (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 > #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker > (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 > #11 0x00007f3f712b9b7d in dummy_worker () from > /usr/local/freeswitch/lib/libfreeswitch.so.1 > #12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0 > #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 > > On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < > anthony.minessale@gmail.com> wrote: > >> It may be trapped inside some blocking call from within the >> conference. >> Your best bet would be to catch it in that state, do the kick, when >> it doesn't work gcore the process and open it in gdb and get "thread apply >> all bt" then you can see the stack trace for that session and see which >> code its blocking on. >> >> The BREAK signal you show there is designed to pull it out of most >> blocking situations so it must be something particular. >> >> >> >> >> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> Guys, >>> >>> I wonder if anyone else ran into this situation: Very rarely I >>> have a conference participant stuck in a conference. When I try to run >>> kick command on that conference, it returns "OK kicked <member ID>" but >>> nothing happens. In the logs there is only this: >>> >>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >>> >>> >>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >>> >>> Any suggestions? >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> -- >> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >> >> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >> http://twitter.com/FreeSWITCH >> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >> <http://freeswitch.org/g+>* >> >> ClueCon Weekly Development Call >> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Do you know what is on the other side of these clients? I get a bad vibe when i see cent6 in reports of odd behaviors as you know ;) So if you uuid_kill it, then it exits? The only logical explanation would be that its stuck doing recvfrom. But in that case the uuid_kill shouldn't work either. do these 2 commands first then a few kick commands then the uuid_kill to compare fsctl loglevel debug console loglevel debug On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony,
One more piece of information.
If I do "kick all" it returns "OK kicked 21698"
But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, exactly
On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
I'll do you one better :-)
Here is the complete output file of "thread apply all bt"
https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt
On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
> Are there conference_loop_input threads to go with those? > search for other threads with one of the sessions in it e.g. > 0x7f3f503a8b48 > > On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar < > olegstolyar@gmail.com> wrote: > >> Thanks Anthony! >> I got two such stuck legs right now. Here are their threads. >> >> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): >> #0 0x0000003d8d207c65 in pthread_join () from >> /lib64/libpthread.so.0 >> #1 0x00007f3f712b9db9 in apr_thread_join () from >> /usr/local/freeswitch/lib/libfreeswitch.so.1 >> #2 0x00007f3f71186d2c in switch_thread_join >> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 >> #3 0x00007f3f667b233c in conference_loop_output >> (member=0x7f3f574fa540) at mod_conference.c:4481 >> #4 0x00007f3f667c1a4c in conference_function >> (session=0x7f3f503a8b48, data=0x7f3f681cbd10 >> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") >> at mod_conference.c:8695 >> #5 0x00007f3f711c23e7 in switch_core_session_exec >> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, >> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default >> +flags{join-only}") >> at src/switch_core_session.c:2862 >> #6 0x00007f3f711c1c85 in >> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, >> app=0x7f3f681cbd00 "conference", >> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", >> flags=0x0) at src/switch_core_session.c:2737 >> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 >> #8 0x00007f3f711c4de0 in switch_core_session_run >> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 >> #9 0x00007f3f711be45c in switch_core_session_thread >> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at >> src/switch_core_session.c:1597 >> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at >> src/switch_core_session.c:1689 >> #11 0x00007f3f712b9b7d in dummy_worker () from >> /usr/local/freeswitch/lib/libfreeswitch.so.1 >> #12 0x0000003d8d20683d in start_thread () from >> /lib64/libpthread.so.0 >> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >> >> >> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): >> #0 0x0000003d8d207c65 in pthread_join () from >> /lib64/libpthread.so.0 >> #1 0x00007f3f712b9db9 in apr_thread_join () from >> /usr/local/freeswitch/lib/libfreeswitch.so.1 >> #2 0x00007f3f71186d2c in switch_thread_join >> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 >> #3 0x00007f3f667b233c in conference_loop_output >> (member=0x7f3f575ae540) at mod_conference.c:4481 >> #4 0x00007f3f667c1a4c in conference_function >> (session=0x7f3f50087998, data=0x2152a38 >> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") >> at mod_conference.c:8695 >> #5 0x00007f3f711c23e7 in switch_core_session_exec >> (session=0x7f3f50087998, application_interface=0x7f3f600e1920, >> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default >> +flags{join-only}") >> at src/switch_core_session.c:2862 >> #6 0x00007f3f711c1c85 in >> switch_core_session_execute_application_get_flags (session=0x7f3f50087998, >> app=0x2152a28 "conference", arg=0x2152a38 >> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", >> flags=0x0) at src/switch_core_session.c:2737 >> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 >> #8 0x00007f3f711c4de0 in switch_core_session_run >> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 >> #9 0x00007f3f711be45c in switch_core_session_thread >> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 >> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 >> #11 0x00007f3f712b9b7d in dummy_worker () from >> /usr/local/freeswitch/lib/libfreeswitch.so.1 >> #12 0x0000003d8d20683d in start_thread () from >> /lib64/libpthread.so.0 >> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >> >> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < >> anthony.minessale@gmail.com> wrote: >> >>> It may be trapped inside some blocking call from within the >>> conference. >>> Your best bet would be to catch it in that state, do the kick, >>> when it doesn't work gcore the process and open it in gdb and get "thread >>> apply all bt" then you can see the stack trace for that session and see >>> which code its blocking on. >>> >>> The BREAK signal you show there is designed to pull it out of most >>> blocking situations so it must be something particular. >>> >>> >>> >>> >>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < >>> olegstolyar@gmail.com> wrote: >>> >>>> Guys, >>>> >>>> I wonder if anyone else ran into this situation: Very rarely I >>>> have a conference participant stuck in a conference. When I try to run >>>> kick command on that conference, it returns "OK kicked <member ID>" but >>>> nothing happens. In the logs there is only this: >>>> >>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >>>> >>>> >>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >>>> >>>> Any suggestions? >>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://confluence.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> >>> -- >>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>> >>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>> http://twitter.com/FreeSWITCH >>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>> <http://freeswitch.org/g+>* >>> >>> ClueCon Weekly Development Call >>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ > <http://freeswitch.org/g+>* > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org ☎ +19193869900 > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
Sure, here it is. I still have two more stuck conferences if you need me to try something else. BTW, the last one is from a different client - not WebRTC, so it's likely not client specific. freeswitch@internal> fsctl loglevel debug +OK log level: DEBUG [7] freeswitch@internal> console loglevel debug +OK log level debug [7] +OK console log level set to DEBUG freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698 2014-09-24 21:07:33.206950 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698 2014-09-24 21:07:36.886946 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Non-Existant ID 21698 freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 freeswitch@internal> uuid_kill 42e7c025-b2ad-4b15-ad58-bbf7609d73d7 +OK 2014-09-24 21:08:33.526948 [NOTICE] switch_ivr.c:3704 Hangup sofia/agent/anonymous@anonymous.invalid [CS_EXECUTE] [NORMAL_CLEARING] 2014-09-24 21:08:33.546949 [DEBUG] switch_channel.c:3216 Send signal sofia/agent/anonymous@anonymous.invalid [KILL] 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:4491 Channel leaving conference, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:8702 sofia/agent/anonymous@anonymous.invalid skip receive message [UNBRIDGE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] switch_core_codec.c:246 sofia/agent/anonymous@anonymous.invalid Restore previous codec PCMU:0. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:2875 sofia/agent/anonymous@anonymous.invalid skip receive message [APPLICATION_EXEC_COMPLETE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] sofia.c:1298 Channel is already hungup. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:530 (sofia/agent/anonymous@anonymous.invalid) State EXECUTE going to sleep 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:730 (sofia/agent/anonymous@anonymous.invalid) Callstate Change ACTIVE -> HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:413 Channel sofia/agent/anonymous@anonymous.invalid hanging up, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:465 Sending BYE to sofia/agent/anonymous@anonymous.invalid 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:58 sofia/agent/anonymous@anonymous.invalid Standard HANGUP, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP going to sleep 2014-09-24 21:08:33.546949 [DEBUG] mod_curl.c:166 method: get, url: http://<mywebapp>answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7, content-type: (null) 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3074 Write Lock ON 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3077 Write Lock OFF 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:663 Hangup Command with no Session curl(http:// <mywebapp>/answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7): conf-5d7b86e0-3ea5-11e4-b722-1231393fb01c 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:499 (sofia/agent/anonymous@anonymous.invalid) State Change CS_HANGUP -> CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:102 sofia/agent/anonymous@anonymous.invalid Standard REPORTING, cause: NORMAL_CLEARING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING going to sleep 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:493 (sofia/agent/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1614 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Locked, Waiting on external entities 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1632 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Ended 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1636 Close Channel sofia/agent/anonymous@anonymous.invalid [CS_DESTROY] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:618 (sofia/agent/anonymous@anonymous.invalid) Callstate Change HANGUP -> DOWN 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:621 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY 2014-09-24 21:08:33.566948 [DEBUG] mod_sofia.c:323 sofia/agent/anonymous@anonymous.invalid SOFIA DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:109 sofia/agent/anonymous@anonymous.invalid Standard DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY going to sleep freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found On Wed, Sep 24, 2014 at 1:33 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Do you know what is on the other side of these clients? I get a bad vibe when i see cent6 in reports of odd behaviors as you know ;)
So if you uuid_kill it, then it exits? The only logical explanation would be that its stuck doing recvfrom. But in that case the uuid_kill shouldn't work either.
do these 2 commands first then a few kick commands then the uuid_kill to compare
fsctl loglevel debug console loglevel debug
On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony,
One more piece of information.
If I do "kick all" it returns "OK kicked 21698"
But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, exactly
On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
It looks like its in recvfrom. What version of FS is this and what type of call client etc is this? Do you have proxy_media on or timers disabled?
It doesn't seem like this should be a likely scenario based on the description and the code I'm looking at. Have you tried to reproduce it with master? The break signal should cause the read to come up to the conference app where it will detect that its done.
On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar <olegstolyar@gmail.com > wrote:
> I'll do you one better :-) > > Here is the complete output file of "thread apply all bt" > > https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt > > > On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < > anthony.minessale@gmail.com> wrote: > >> Are there conference_loop_input threads to go with those? >> search for other threads with one of the sessions in it e.g. >> 0x7f3f503a8b48 >> >> On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> Thanks Anthony! >>> I got two such stuck legs right now. Here are their threads. >>> >>> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): >>> #0 0x0000003d8d207c65 in pthread_join () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>> #2 0x00007f3f71186d2c in switch_thread_join >>> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 >>> #3 0x00007f3f667b233c in conference_loop_output >>> (member=0x7f3f574fa540) at mod_conference.c:4481 >>> #4 0x00007f3f667c1a4c in conference_function >>> (session=0x7f3f503a8b48, data=0x7f3f681cbd10 >>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") >>> at mod_conference.c:8695 >>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, >>> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default >>> +flags{join-only}") >>> at src/switch_core_session.c:2862 >>> #6 0x00007f3f711c1c85 in >>> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, >>> app=0x7f3f681cbd00 "conference", >>> arg=0x7f3f681cbd10 >>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", >>> flags=0x0) at src/switch_core_session.c:2737 >>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 >>> #8 0x00007f3f711c4de0 in switch_core_session_run >>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 >>> #9 0x00007f3f711be45c in switch_core_session_thread >>> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at >>> src/switch_core_session.c:1597 >>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at >>> src/switch_core_session.c:1689 >>> #11 0x00007f3f712b9b7d in dummy_worker () from >>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>> #12 0x0000003d8d20683d in start_thread () from >>> /lib64/libpthread.so.0 >>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>> >>> >>> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): >>> #0 0x0000003d8d207c65 in pthread_join () from >>> /lib64/libpthread.so.0 >>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>> #2 0x00007f3f71186d2c in switch_thread_join >>> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 >>> #3 0x00007f3f667b233c in conference_loop_output >>> (member=0x7f3f575ae540) at mod_conference.c:4481 >>> #4 0x00007f3f667c1a4c in conference_function >>> (session=0x7f3f50087998, data=0x2152a38 >>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") >>> at mod_conference.c:8695 >>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>> (session=0x7f3f50087998, application_interface=0x7f3f600e1920, >>> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default >>> +flags{join-only}") >>> at src/switch_core_session.c:2862 >>> #6 0x00007f3f711c1c85 in >>> switch_core_session_execute_application_get_flags (session=0x7f3f50087998, >>> app=0x2152a28 "conference", arg=0x2152a38 >>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", >>> flags=0x0) at src/switch_core_session.c:2737 >>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 >>> #8 0x00007f3f711c4de0 in switch_core_session_run >>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 >>> #9 0x00007f3f711be45c in switch_core_session_thread >>> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 >>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 >>> #11 0x00007f3f712b9b7d in dummy_worker () from >>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>> #12 0x0000003d8d20683d in start_thread () from >>> /lib64/libpthread.so.0 >>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>> >>> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < >>> anthony.minessale@gmail.com> wrote: >>> >>>> It may be trapped inside some blocking call from within the >>>> conference. >>>> Your best bet would be to catch it in that state, do the kick, >>>> when it doesn't work gcore the process and open it in gdb and get "thread >>>> apply all bt" then you can see the stack trace for that session and see >>>> which code its blocking on. >>>> >>>> The BREAK signal you show there is designed to pull it out of >>>> most blocking situations so it must be something particular. >>>> >>>> >>>> >>>> >>>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < >>>> olegstolyar@gmail.com> wrote: >>>> >>>>> Guys, >>>>> >>>>> I wonder if anyone else ran into this situation: Very rarely I >>>>> have a conference participant stuck in a conference. When I try to run >>>>> kick command on that conference, it returns "OK kicked <member ID>" but >>>>> nothing happens. In the logs there is only this: >>>>> >>>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >>>>> >>>>> >>>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >>>>> >>>>> Any suggestions? >>>>> >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://confluence.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>>> >>>> >>>> >>>> >>>> -- >>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>>> >>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>>> http://twitter.com/FreeSWITCH >>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>>> <http://freeswitch.org/g+>* >>>> >>>> ClueCon Weekly Development Call >>>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://confluence.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> -- >> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >> >> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >> http://twitter.com/FreeSWITCH >> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >> <http://freeswitch.org/g+>* >> >> ClueCon Weekly Development Call >> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The KILL sig is more severe than break, it calls shutdown on the socket. I think your box has some kernel params causing blocking calls to recvfrom, calling the break should be sending a udp packet to itself to wake it up even when blocking. Do you have any firewall rules that might block it from sending itself RTP packets? It'd be nice if you could reproduce this on debian stable! On Wed, Sep 24, 2014 at 4:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Sure, here it is. I still have two more stuck conferences if you need me to try something else. BTW, the last one is from a different client - not WebRTC, so it's likely not client specific.
freeswitch@internal> fsctl loglevel debug +OK log level: DEBUG [7]
freeswitch@internal> console loglevel debug +OK log level debug [7] +OK console log level set to DEBUG
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698
2014-09-24 21:07:33.206950 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698
2014-09-24 21:07:36.886946 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Non-Existant ID 21698
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698
freeswitch@internal> uuid_kill 42e7c025-b2ad-4b15-ad58-bbf7609d73d7 +OK
2014-09-24 21:08:33.526948 [NOTICE] switch_ivr.c:3704 Hangup sofia/agent/anonymous@anonymous.invalid [CS_EXECUTE] [NORMAL_CLEARING] 2014-09-24 21:08:33.546949 [DEBUG] switch_channel.c:3216 Send signal sofia/agent/anonymous@anonymous.invalid [KILL] 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:4491 Channel leaving conference, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:8702 sofia/agent/anonymous@anonymous.invalid skip receive message [UNBRIDGE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] switch_core_codec.c:246 sofia/agent/anonymous@anonymous.invalid Restore previous codec PCMU:0. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:2875 sofia/agent/anonymous@anonymous.invalid skip receive message [APPLICATION_EXEC_COMPLETE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] sofia.c:1298 Channel is already hungup. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:530 (sofia/agent/anonymous@anonymous.invalid) State EXECUTE going to sleep 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:730 (sofia/agent/anonymous@anonymous.invalid) Callstate Change ACTIVE -> HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:413 Channel sofia/agent/anonymous@anonymous.invalid hanging up, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:465 Sending BYE to sofia/agent/anonymous@anonymous.invalid 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:58 sofia/agent/anonymous@anonymous.invalid Standard HANGUP, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP going to sleep 2014-09-24 21:08:33.546949 [DEBUG] mod_curl.c:166 method: get, url: http://<mywebapp>answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7, content-type: (null) 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3074 Write Lock ON 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3077 Write Lock OFF 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:663 Hangup Command with no Session curl(http:// <mywebapp>/answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7): conf-5d7b86e0-3ea5-11e4-b722-1231393fb01c 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:499 (sofia/agent/anonymous@anonymous.invalid) State Change CS_HANGUP -> CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:102 sofia/agent/anonymous@anonymous.invalid Standard REPORTING, cause: NORMAL_CLEARING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING going to sleep 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:493 (sofia/agent/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1614 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Locked, Waiting on external entities 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1632 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Ended 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1636 Close Channel sofia/agent/anonymous@anonymous.invalid [CS_DESTROY] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:618 (sofia/agent/anonymous@anonymous.invalid) Callstate Change HANGUP -> DOWN 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:621 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY 2014-09-24 21:08:33.566948 [DEBUG] mod_sofia.c:323 sofia/agent/anonymous@anonymous.invalid SOFIA DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:109 sofia/agent/anonymous@anonymous.invalid Standard DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY going to sleep
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
On Wed, Sep 24, 2014 at 1:33 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Do you know what is on the other side of these clients? I get a bad vibe when i see cent6 in reports of odd behaviors as you know ;)
So if you uuid_kill it, then it exits? The only logical explanation would be that its stuck doing recvfrom. But in that case the uuid_kill shouldn't work either.
do these 2 commands first then a few kick commands then the uuid_kill to compare
fsctl loglevel debug console loglevel debug
On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony,
One more piece of information.
If I do "kick all" it returns "OK kicked 21698"
But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, exactly
On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Version is: FreeSWITCH Version 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 19:46:31Z 64bit)
Since I cannot reproduce this at will, I would need to upgrade my production to the latest master to try to reproduce it there.
The client is WebRTC with JsSip on Chrome 34.
I do not enable proxy_media and in fact there is no reference to proxy_media anywhere in my config. Similarly, I did not change anything related to timers but if you;d like me to check a specific setting, I'll be happy to.
I still have that FS instance running, so if there is anything else I can do to help track this down, please let me know.
On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
> It looks like its in recvfrom. What version of FS is this and what > type of call client etc is this? > Do you have proxy_media on or timers disabled? > > It doesn't seem like this should be a likely scenario based on the > description and the code I'm looking at. Have you tried to reproduce it > with master? > The break signal should cause the read to come up to the conference > app where it will detect that its done. > > > > On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar < > olegstolyar@gmail.com> wrote: > >> I'll do you one better :-) >> >> Here is the complete output file of "thread apply all bt" >> >> https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt >> >> >> On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < >> anthony.minessale@gmail.com> wrote: >> >>> Are there conference_loop_input threads to go with those? >>> search for other threads with one of the sessions in it e.g. >>> 0x7f3f503a8b48 >>> >>> On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar < >>> olegstolyar@gmail.com> wrote: >>> >>>> Thanks Anthony! >>>> I got two such stuck legs right now. Here are their threads. >>>> >>>> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): >>>> #0 0x0000003d8d207c65 in pthread_join () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>> #2 0x00007f3f71186d2c in switch_thread_join >>>> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 >>>> #3 0x00007f3f667b233c in conference_loop_output >>>> (member=0x7f3f574fa540) at mod_conference.c:4481 >>>> #4 0x00007f3f667c1a4c in conference_function >>>> (session=0x7f3f503a8b48, data=0x7f3f681cbd10 >>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") >>>> at mod_conference.c:8695 >>>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>>> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, >>>> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default >>>> +flags{join-only}") >>>> at src/switch_core_session.c:2862 >>>> #6 0x00007f3f711c1c85 in >>>> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, >>>> app=0x7f3f681cbd00 "conference", >>>> arg=0x7f3f681cbd10 >>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", >>>> flags=0x0) at src/switch_core_session.c:2737 >>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 >>>> #8 0x00007f3f711c4de0 in switch_core_session_run >>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 >>>> #9 0x00007f3f711be45c in switch_core_session_thread >>>> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at >>>> src/switch_core_session.c:1597 >>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>>> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at >>>> src/switch_core_session.c:1689 >>>> #11 0x00007f3f712b9b7d in dummy_worker () from >>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>> #12 0x0000003d8d20683d in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>>> >>>> >>>> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): >>>> #0 0x0000003d8d207c65 in pthread_join () from >>>> /lib64/libpthread.so.0 >>>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>> #2 0x00007f3f71186d2c in switch_thread_join >>>> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 >>>> #3 0x00007f3f667b233c in conference_loop_output >>>> (member=0x7f3f575ae540) at mod_conference.c:4481 >>>> #4 0x00007f3f667c1a4c in conference_function >>>> (session=0x7f3f50087998, data=0x2152a38 >>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") >>>> at mod_conference.c:8695 >>>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>>> (session=0x7f3f50087998, application_interface=0x7f3f600e1920, >>>> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default >>>> +flags{join-only}") >>>> at src/switch_core_session.c:2862 >>>> #6 0x00007f3f711c1c85 in >>>> switch_core_session_execute_application_get_flags (session=0x7f3f50087998, >>>> app=0x2152a28 "conference", arg=0x2152a38 >>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", >>>> flags=0x0) at src/switch_core_session.c:2737 >>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 >>>> #8 0x00007f3f711c4de0 in switch_core_session_run >>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 >>>> #9 0x00007f3f711be45c in switch_core_session_thread >>>> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 >>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>>> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 >>>> #11 0x00007f3f712b9b7d in dummy_worker () from >>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>> #12 0x0000003d8d20683d in start_thread () from >>>> /lib64/libpthread.so.0 >>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>>> >>>> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < >>>> anthony.minessale@gmail.com> wrote: >>>> >>>>> It may be trapped inside some blocking call from within the >>>>> conference. >>>>> Your best bet would be to catch it in that state, do the kick, >>>>> when it doesn't work gcore the process and open it in gdb and get "thread >>>>> apply all bt" then you can see the stack trace for that session and see >>>>> which code its blocking on. >>>>> >>>>> The BREAK signal you show there is designed to pull it out of >>>>> most blocking situations so it must be something particular. >>>>> >>>>> >>>>> >>>>> >>>>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < >>>>> olegstolyar@gmail.com> wrote: >>>>> >>>>>> Guys, >>>>>> >>>>>> I wonder if anyone else ran into this situation: Very rarely I >>>>>> have a conference participant stuck in a conference. When I try to run >>>>>> kick command on that conference, it returns "OK kicked <member ID>" but >>>>>> nothing happens. In the logs there is only this: >>>>>> >>>>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >>>>>> >>>>>> >>>>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >>>>>> >>>>>> Any suggestions? >>>>>> >>>>>> >>>>>> >>>>>> _________________________________________________________________________ >>>>>> Professional FreeSWITCH Consulting Services: >>>>>> consulting@freeswitch.org >>>>>> http://www.freeswitchsolutions.com >>>>>> >>>>>> Official FreeSWITCH Sites >>>>>> http://www.freeswitch.org >>>>>> http://confluence.freeswitch.org >>>>>> http://www.cluecon.com >>>>>> >>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>>> http://www.cudatel.com >>>>>> >>>>>> FreeSWITCH-users mailing list >>>>>> FreeSWITCH-users@lists.freeswitch.org >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>>> UNSUBSCRIBE: >>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>>> http://www.freeswitch.org >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>>>> >>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>>>> http://twitter.com/FreeSWITCH >>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>>>> <http://freeswitch.org/g+>* >>>>> >>>>> ClueCon Weekly Development Call >>>>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>>>> >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://confluence.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>>> >>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://confluence.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> >>> -- >>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>> >>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>> http://twitter.com/FreeSWITCH >>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>> <http://freeswitch.org/g+>* >>> >>> ClueCon Weekly Development Call >>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > -- > Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ > > ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ > http://twitter.com/FreeSWITCH > ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ > <http://freeswitch.org/g+>* > > ClueCon Weekly Development Call > ☎ sip:888@conference.freeswitch.org ☎ +19193869900 > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
It's an AWS instance. No firewall rules that would prevent this that I know of. Since I cannot repro this at will, the only way to reproduce it on Debian is to put it into production. Unfortunately I cannot do that - there are too many dependencies. On Wed, Sep 24, 2014 at 2:24 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
The KILL sig is more severe than break, it calls shutdown on the socket. I think your box has some kernel params causing blocking calls to recvfrom, calling the break should be sending a udp packet to itself to wake it up even when blocking. Do you have any firewall rules that might block it from sending itself RTP packets?
It'd be nice if you could reproduce this on debian stable!
On Wed, Sep 24, 2014 at 4:12 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Sure, here it is. I still have two more stuck conferences if you need me to try something else. BTW, the last one is from a different client - not WebRTC, so it's likely not client specific.
freeswitch@internal> fsctl loglevel debug +OK log level: DEBUG [7]
freeswitch@internal> console loglevel debug +OK log level debug [7] +OK console log level set to DEBUG
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698
2014-09-24 21:07:33.206950 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all OK kicked 21698
2014-09-24 21:07:36.886946 [DEBUG] mod_conference.c:5619 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] freeswitch@internal> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@freeswitch@internal
freeswitch@internal> freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Non-Existant ID 21698
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698
freeswitch@internal> uuid_kill 42e7c025-b2ad-4b15-ad58-bbf7609d73d7 +OK
2014-09-24 21:08:33.526948 [NOTICE] switch_ivr.c:3704 Hangup sofia/agent/anonymous@anonymous.invalid [CS_EXECUTE] [NORMAL_CLEARING] 2014-09-24 21:08:33.546949 [DEBUG] switch_channel.c:3216 Send signal sofia/agent/anonymous@anonymous.invalid [KILL] 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:4491 Channel leaving conference, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_conference.c:8702 sofia/agent/anonymous@anonymous.invalid skip receive message [UNBRIDGE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] switch_core_codec.c:246 sofia/agent/anonymous@anonymous.invalid Restore previous codec PCMU:0. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_session.c:2875 sofia/agent/anonymous@anonymous.invalid skip receive message [APPLICATION_EXEC_COMPLETE] (channel is hungup already) 2014-09-24 21:08:33.546949 [DEBUG] sofia.c:1298 Channel is already hungup. 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:530 (sofia/agent/anonymous@anonymous.invalid) State EXECUTE going to sleep 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:730 (sofia/agent/anonymous@anonymous.invalid) Callstate Change ACTIVE -> HANGUP 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:413 Channel sofia/agent/anonymous@anonymous.invalid hanging up, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] mod_sofia.c:465 Sending BYE to sofia/agent/anonymous@anonymous.invalid 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:58 sofia/agent/anonymous@anonymous.invalid Standard HANGUP, cause: NORMAL_CLEARING 2014-09-24 21:08:33.546949 [DEBUG] switch_core_state_machine.c:732 (sofia/agent/anonymous@anonymous.invalid) State HANGUP going to sleep 2014-09-24 21:08:33.546949 [DEBUG] mod_curl.c:166 method: get, url: http://<mywebapp>answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7, content-type: (null) 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3074 Write Lock ON 2014-09-24 21:08:33.566948 [DEBUG] mod_conference.c:3077 Write Lock OFF 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:663 Hangup Command with no Session curl(http:// <mywebapp>/answerhangup?lcid=5d7b86e0-3ea5-11e4-b722-1231393fb01c&leg_uuid=42e7c025-b2ad-4b15-ad58-bbf7609d73d7): conf-5d7b86e0-3ea5-11e4-b722-1231393fb01c 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:499 (sofia/agent/anonymous@anonymous.invalid) State Change CS_HANGUP -> CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:467 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:102 sofia/agent/anonymous@anonymous.invalid Standard REPORTING, cause: NORMAL_CLEARING 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:818 (sofia/agent/anonymous@anonymous.invalid) State REPORTING going to sleep 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:493 (sofia/agent/anonymous@anonymous.invalid) State Change CS_REPORTING -> CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1387 Send signal sofia/agent/anonymous@anonymous.invalid [BREAK] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_session.c:1614 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Locked, Waiting on external entities 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1632 Session 28810 (sofia/agent/anonymous@anonymous.invalid) Ended 2014-09-24 21:08:33.566948 [NOTICE] switch_core_session.c:1636 Close Channel sofia/agent/anonymous@anonymous.invalid [CS_DESTROY] 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:618 (sofia/agent/anonymous@anonymous.invalid) Callstate Change HANGUP -> DOWN 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:621 (sofia/agent/anonymous@anonymous.invalid) Running State Change CS_DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY 2014-09-24 21:08:33.566948 [DEBUG] mod_sofia.c:323 sofia/agent/anonymous@anonymous.invalid SOFIA DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:109 sofia/agent/anonymous@anonymous.invalid Standard DESTROY 2014-09-24 21:08:33.566948 [DEBUG] switch_core_state_machine.c:631 (sofia/agent/anonymous@anonymous.invalid) State DESTROY going to sleep
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick 21698 Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
freeswitch@internal> conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c kick all Conference 5d7b86e0-3ea5-11e4-b722-1231393fb01c not found
On Wed, Sep 24, 2014 at 1:33 PM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Do you know what is on the other side of these clients? I get a bad vibe when i see cent6 in reports of odd behaviors as you know ;)
So if you uuid_kill it, then it exits? The only logical explanation would be that its stuck doing recvfrom. But in that case the uuid_kill shouldn't work either.
do these 2 commands first then a few kick commands then the uuid_kill to compare
fsctl loglevel debug console loglevel debug
On Wed, Sep 24, 2014 at 2:08 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Hi Anthony,
One more piece of information.
If I do "kick all" it returns "OK kicked 21698"
But if I do "kick 21698" it returns "Non-Existant ID 21698" and there is no BREAK entry in the log.
On Tue, Sep 23, 2014 at 12:03 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
Yes, exactly
On Tue, Sep 23, 2014 at 11:37 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
Doing more kick commands results in more break messages but it never works even with subsequent calls to kick?
On Tue, Sep 23, 2014 at 1:23 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:
> Version is: FreeSWITCH Version > 1.5.13b+git~20140529T194631Z~0d7b1ff42a~64bit (git 0d7b1ff 2014-05-29 > 19:46:31Z 64bit) > > Since I cannot reproduce this at will, I would need to upgrade my > production to the latest master to try to reproduce it there. > > The client is WebRTC with JsSip on Chrome 34. > > I do not enable proxy_media and in fact there is no reference to > proxy_media anywhere in my config. > Similarly, I did not change anything related to timers but if you;d > like me to check a specific setting, I'll be happy to. > > I still have that FS instance running, so if there is anything else > I can do to help track this down, please let me know. > > > On Tue, Sep 23, 2014 at 11:06 AM, Anthony Minessale < > anthony.minessale@gmail.com> wrote: > >> It looks like its in recvfrom. What version of FS is this and what >> type of call client etc is this? >> Do you have proxy_media on or timers disabled? >> >> It doesn't seem like this should be a likely scenario based on the >> description and the code I'm looking at. Have you tried to reproduce it >> with master? >> The break signal should cause the read to come up to the conference >> app where it will detect that its done. >> >> >> >> On Tue, Sep 23, 2014 at 12:49 PM, Oleg Stolyar < >> olegstolyar@gmail.com> wrote: >> >>> I'll do you one better :-) >>> >>> Here is the complete output file of "thread apply all bt" >>> >>> https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt >>> >>> >>> On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale < >>> anthony.minessale@gmail.com> wrote: >>> >>>> Are there conference_loop_input threads to go with those? >>>> search for other threads with one of the sessions in it e.g. >>>> 0x7f3f503a8b48 >>>> >>>> On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar < >>>> olegstolyar@gmail.com> wrote: >>>> >>>>> Thanks Anthony! >>>>> I got two such stuck legs right now. Here are their threads. >>>>> >>>>> Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)): >>>>> #0 0x0000003d8d207c65 in pthread_join () from >>>>> /lib64/libpthread.so.0 >>>>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>>> #2 0x00007f3f71186d2c in switch_thread_join >>>>> (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171 >>>>> #3 0x00007f3f667b233c in conference_loop_output >>>>> (member=0x7f3f574fa540) at mod_conference.c:4481 >>>>> #4 0x00007f3f667c1a4c in conference_function >>>>> (session=0x7f3f503a8b48, data=0x7f3f681cbd10 >>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") >>>>> at mod_conference.c:8695 >>>>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>>>> (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, >>>>> arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default >>>>> +flags{join-only}") >>>>> at src/switch_core_session.c:2862 >>>>> #6 0x00007f3f711c1c85 in >>>>> switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, >>>>> app=0x7f3f681cbd00 "conference", >>>>> arg=0x7f3f681cbd10 >>>>> "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", >>>>> flags=0x0) at src/switch_core_session.c:2737 >>>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281 >>>>> #8 0x00007f3f711c4de0 in switch_core_session_run >>>>> (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530 >>>>> #9 0x00007f3f711be45c in switch_core_session_thread >>>>> (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at >>>>> src/switch_core_session.c:1597 >>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>>>> (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at >>>>> src/switch_core_session.c:1689 >>>>> #11 0x00007f3f712b9b7d in dummy_worker () from >>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>>> #12 0x0000003d8d20683d in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>>>> >>>>> >>>>> Thread 60 (Thread 0x7f3f575af940 (LWP 28590)): >>>>> #0 0x0000003d8d207c65 in pthread_join () from >>>>> /lib64/libpthread.so.0 >>>>> #1 0x00007f3f712b9db9 in apr_thread_join () from >>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>>> #2 0x00007f3f71186d2c in switch_thread_join >>>>> (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171 >>>>> #3 0x00007f3f667b233c in conference_loop_output >>>>> (member=0x7f3f575ae540) at mod_conference.c:4481 >>>>> #4 0x00007f3f667c1a4c in conference_function >>>>> (session=0x7f3f50087998, data=0x2152a38 >>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") >>>>> at mod_conference.c:8695 >>>>> #5 0x00007f3f711c23e7 in switch_core_session_exec >>>>> (session=0x7f3f50087998, application_interface=0x7f3f600e1920, >>>>> arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default >>>>> +flags{join-only}") >>>>> at src/switch_core_session.c:2862 >>>>> #6 0x00007f3f711c1c85 in >>>>> switch_core_session_execute_application_get_flags (session=0x7f3f50087998, >>>>> app=0x2152a28 "conference", arg=0x2152a38 >>>>> "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}", >>>>> flags=0x0) at src/switch_core_session.c:2737 >>>>> #7 0x00007f3f711c3c3c in switch_core_standard_on_execute >>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:281 >>>>> #8 0x00007f3f711c4de0 in switch_core_session_run >>>>> (session=0x7f3f50087998) at src/switch_core_state_machine.c:530 >>>>> #9 0x00007f3f711be45c in switch_core_session_thread >>>>> (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597 >>>>> #10 0x00007f3f711be913 in switch_core_session_thread_pool_worker >>>>> (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689 >>>>> #11 0x00007f3f712b9b7d in dummy_worker () from >>>>> /usr/local/freeswitch/lib/libfreeswitch.so.1 >>>>> #12 0x0000003d8d20683d in start_thread () from >>>>> /lib64/libpthread.so.0 >>>>> #13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6 >>>>> >>>>> On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale < >>>>> anthony.minessale@gmail.com> wrote: >>>>> >>>>>> It may be trapped inside some blocking call from within the >>>>>> conference. >>>>>> Your best bet would be to catch it in that state, do the kick, >>>>>> when it doesn't work gcore the process and open it in gdb and get "thread >>>>>> apply all bt" then you can see the stack trace for that session and see >>>>>> which code its blocking on. >>>>>> >>>>>> The BREAK signal you show there is designed to pull it out of >>>>>> most blocking situations so it must be something particular. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar < >>>>>> olegstolyar@gmail.com> wrote: >>>>>> >>>>>>> Guys, >>>>>>> >>>>>>> I wonder if anyone else ran into this situation: Very rarely >>>>>>> I have a conference participant stuck in a conference. When I try to run >>>>>>> kick command on that conference, it returns "OK kicked <member ID>" but >>>>>>> nothing happens. In the logs there is only this: >>>>>>> >>>>>>> db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK] >>>>>>> >>>>>>> >>>>>>> As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference. >>>>>>> >>>>>>> Any suggestions? >>>>>>> >>>>>>> >>>>>>> >>>>>>> _________________________________________________________________________ >>>>>>> Professional FreeSWITCH Consulting Services: >>>>>>> consulting@freeswitch.org >>>>>>> http://www.freeswitchsolutions.com >>>>>>> >>>>>>> Official FreeSWITCH Sites >>>>>>> http://www.freeswitch.org >>>>>>> http://confluence.freeswitch.org >>>>>>> http://www.cluecon.com >>>>>>> >>>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>>>> http://www.cudatel.com >>>>>>> >>>>>>> FreeSWITCH-users mailing list >>>>>>> FreeSWITCH-users@lists.freeswitch.org >>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>>>> UNSUBSCRIBE: >>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>>>> http://www.freeswitch.org >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>>>>> >>>>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>>>>> http://twitter.com/FreeSWITCH >>>>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>>>>> <http://freeswitch.org/g+>* >>>>>> >>>>>> ClueCon Weekly Development Call >>>>>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>>>>> >>>>>> >>>>>> >>>>>> _________________________________________________________________________ >>>>>> Professional FreeSWITCH Consulting Services: >>>>>> consulting@freeswitch.org >>>>>> http://www.freeswitchsolutions.com >>>>>> >>>>>> Official FreeSWITCH Sites >>>>>> http://www.freeswitch.org >>>>>> http://confluence.freeswitch.org >>>>>> http://www.cluecon.com >>>>>> >>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>>> http://www.cudatel.com >>>>>> >>>>>> FreeSWITCH-users mailing list >>>>>> FreeSWITCH-users@lists.freeswitch.org >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>>> UNSUBSCRIBE: >>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>>> http://www.freeswitch.org >>>>>> >>>>> >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://confluence.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE: >>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>>> >>>> >>>> >>>> >>>> -- >>>> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >>>> >>>> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >>>> http://twitter.com/FreeSWITCH >>>> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >>>> <http://freeswitch.org/g+>* >>>> >>>> ClueCon Weekly Development Call >>>> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://confluence.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://confluence.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >> >> -- >> Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ >> >> ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ >> http://twitter.com/FreeSWITCH >> ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ >> <http://freeswitch.org/g+>* >> >> ClueCon Weekly Development Call >> ☎ sip:888@conference.freeswitch.org ☎ +19193869900 >> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://confluence.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://confluence.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (2)
-
Anthony Minessale
-
Oleg Stolyar