I'll do you one better :-)

Here is the complete output file of "thread apply all bt"

https://www.dropbox.com/s/gslm6l13jrxkr6s/gdb.txt

On Tue, Sep 23, 2014 at 10:40 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:

Are there conference_loop_input threads to go with those?

search for other threads with one of the sessions in it e.g. 0x7f3f503a8b48

On Tue, Sep 23, 2014 at 12:33 PM, Oleg Stolyar <olegstolyar@gmail.com> wrote:

Thanks Anthony!

I got two such stuck legs right now.  Here are their threads.  

Thread 45 (Thread 0x7f3f574fb940 (LWP 25467)):

#0  0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0

#1  0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1

#2  0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f574fa0fc, thd=0x7f3f683f8cb8) at src/switch_apr.c:1171

#3  0x00007f3f667b233c in conference_loop_output (member=0x7f3f574fa540) at mod_conference.c:4481

#4  0x00007f3f667c1a4c in conference_function (session=0x7f3f503a8b48, data=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}") at mod_conference.c:8695

#5  0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f503a8b48, application_interface=0x7f3f600e1920, arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}")

    at src/switch_core_session.c:2862

#6  0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f503a8b48, app=0x7f3f681cbd00 "conference",

    arg=0x7f3f681cbd10 "5d7b86e0-3ea5-11e4-b722-1231393fb01c@default+flags{join-only}", flags=0x0) at src/switch_core_session.c:2737

#7  0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:281

#8  0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f503a8b48) at src/switch_core_state_machine.c:530

#9  0x00007f3f711be45c in switch_core_session_thread (thread=0x7f3f50b9f990, obj=0x7f3f503a8b48) at src/switch_core_session.c:1597

#10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x7f3f50b9f990, obj=0x7f3f50b9f820) at src/switch_core_session.c:1689

#11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1

#12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0

#13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6

Thread 60 (Thread 0x7f3f575af940 (LWP 28590)):

#0  0x0000003d8d207c65 in pthread_join () from /lib64/libpthread.so.0

#1  0x00007f3f712b9db9 in apr_thread_join () from /usr/local/freeswitch/lib/libfreeswitch.so.1

#2  0x00007f3f71186d2c in switch_thread_join (retval=0x7f3f575ae0fc, thd=0x215a358) at src/switch_apr.c:1171

#3  0x00007f3f667b233c in conference_loop_output (member=0x7f3f575ae540) at mod_conference.c:4481

#4  0x00007f3f667c1a4c in conference_function (session=0x7f3f50087998, data=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}") at mod_conference.c:8695

#5  0x00007f3f711c23e7 in switch_core_session_exec (session=0x7f3f50087998, application_interface=0x7f3f600e1920, arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}")

    at src/switch_core_session.c:2862

#6  0x00007f3f711c1c85 in switch_core_session_execute_application_get_flags (session=0x7f3f50087998, app=0x2152a28 "conference", arg=0x2152a38 "2e42aba0-430b-11e4-bc8b-22000a1cca7e@default+flags{join-only}",

    flags=0x0) at src/switch_core_session.c:2737

#7  0x00007f3f711c3c3c in switch_core_standard_on_execute (session=0x7f3f50087998) at src/switch_core_state_machine.c:281

#8  0x00007f3f711c4de0 in switch_core_session_run (session=0x7f3f50087998) at src/switch_core_state_machine.c:530

#9  0x00007f3f711be45c in switch_core_session_thread (thread=0x214d4b0, obj=0x7f3f50087998) at src/switch_core_session.c:1597

#10 0x00007f3f711be913 in switch_core_session_thread_pool_worker (thread=0x214d4b0, obj=0x214d340) at src/switch_core_session.c:1689

#11 0x00007f3f712b9b7d in dummy_worker () from /usr/local/freeswitch/lib/libfreeswitch.so.1

#12 0x0000003d8d20683d in start_thread () from /lib64/libpthread.so.0

#13 0x0000003d8cad4f8d in clone () from /lib64/libc.so.6

On Tue, Sep 23, 2014 at 9:16 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:

It may be trapped inside some blocking call from within the conference.

Your best bet would be to catch it in that state, do the kick, when it doesn't work gcore the process and open it in gdb and get "thread apply all bt" then you can see the stack trace for that session and see which code its blocking on.

The BREAK signal you show there is designed to pull it out of most blocking situations so it must be something particular.

On Tue, Sep 23, 2014 at 10:53 AM, Oleg Stolyar <olegstolyar@gmail.com> wrote:

Guys,

I wonder if anyone else ran into this situation:  Very rarely I have a conference participant stuck in a conference.  When I try to run kick command on that conference, it returns "OK kicked <member ID>" but nothing happens.  In the logs there is only this:

db1790fe-fa75-42a6-9054-75c600b9539c 2014-09-23 15:43:07.066947 [DEBUG] mod_conference.c:5619 Send signal sofia/<profile>/anonymous@anonymous.invalid [BREAK]

As an additional piece of information if I use uuid_kill on the leg, it does disconnect the leg and ends the conference.

Any suggestions?

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--

Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞ http://twitter.com/FreeSWITCH

☞ irc.freenode.net #freeswitch ☞ http://freeswitch.org/g+

ClueCon Weekly Development Call 

☎ sip:888@conference.freeswitch.org  ☎ +19193869900 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--

Anthony Minessale II       ♬ @anthmfs  ♬ @FreeSWITCH  ♬

☞ http://freeswitch.org/  ☞ http://cluecon.com/  ☞ http://twitter.com/FreeSWITCH

☞ irc.freenode.net #freeswitch ☞ http://freeswitch.org/g+

ClueCon Weekly Development Call 

☎ sip:888@conference.freeswitch.org  ☎ +19193869900 

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org