Threads remain after calling close on Java client
Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
I experienced similar problems when load testing Plivo on the latest Git, but didn't feel like I had enough data gathered for a "proper" report. After each call, the remote ESL socket would be left in a CLOSE_WAIT state. I think the problem started on December 8th, with the resolution of this Jira: http://jira.freeswitch.org/browse/FS-3750 http://fisheye.freeswitch.org/browse/freeswitch.git/src/mod/event_handle rs/mod_event_socket/mod_event_socket.c?r2=19dad4a527e4e87bdbecf7b97e3d07 fd11e2a04c&r1=6bd2798ea1df47e2a5b9de99defbd79e33f5726f When I rolled the code back to the revision just prior to this change, my problems went away. git checkout 1868e145201cc6ba5a14d6929695977780917a38 Dave. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
Dave, This patch only would come into play if you were sending the linger command over the socket once connected. Before the patch, if you executed "linger" it would never disconnect the socket until you did it yourself from the client side. After the patch it should apply a default timeout of 600 seconds or allow you to specify a smaller one such as "linger 20" In any case this would be moot if you manually call the disconnect yourself before you close your script. Are you sure this patch is causing you problems? This really should go in a JIRA if so, at least re-open the existing one that added the patch and post your findings. Also the guy with the local mods to the java stuff, please contribute them and we can add them to tree. On Sat, Dec 17, 2011 at 6:26 AM, Dave May <Dave.May@patlive.com> wrote:
I experienced similar problems when load testing Plivo on the latest Git, but didn't feel like I had enough data gathered for a "proper" report. After each call, the remote ESL socket would be left in a CLOSE_WAIT state. ****
** **
I think the problem started on December 8th, with the resolution of this Jira:****
** **
http://jira.freeswitch.org/browse/FS-3750****
** **
http://fisheye.freeswitch.org/browse/freeswitch.git/src/mod/event_handlers/m... ****
** **
When I rolled the code back to the revision just prior to this change, my problems went away.****
** **
git checkout 1868e145201cc6ba5a14d6929695977780917a38****
** **
Dave.****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Friday, December 16, 2011 7:09 PM *To:* FreeSWITCH-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Threads remain after calling close on Java client****
** **
Hi,****
****
I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. ****
****
The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully?****
****
Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging.****
****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
****
****
Thanks,****
****
Neil Davis****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I've fixed a couple of bugs for this library, but the patches haven't been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a “waiting on condition” state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Hector Geraldino *Sent:* Monday, December 19, 2011 8:29 AM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I’ve fixed a couple of bugs for this library, but the patches haven’t been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Friday, December 16, 2011 7:09 PM *To:* FreeSWITCH-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
Hi Neil, This doesn't seem to be the same concurrency issue I had, but I'm attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls. In case it doesn't work you might try to "manually" close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Neil Davis Sent: Wednesday, December 21, 2011 11:17 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a "waiting on condition" state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Hector Geraldino Sent: Monday, December 19, 2011 8:29 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I've fixed a couple of bugs for this library, but the patches haven't been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
if this patch is necessary can you post it to a jira? On Wed, Dec 21, 2011 at 1:26 PM, Hector Geraldino < Hector.Geraldino@ip-soft.net> wrote:
Hi Neil,****
** **
This doesn’t seem to be the same concurrency issue I had, but I’m attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls.****
** **
In case it doesn’t work you might try to “manually” close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens.****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Wednesday, December 21, 2011 11:17 AM
*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client****
** **
Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a “waiting on condition” state that appear to have to do with the netty package on which the Freeswitch client is dependent.****
****
****
2011-12-21 09:05:17****
Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode):****
****
"DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000]****
java.lang.Thread.State: RUNNABLE****
at java.net.SocketInputStream.socketRead0(Native Method)** **
at java.net.SocketInputStream.read(Unknown Source)****
at java.io.BufferedInputStream.fill(Unknown Source)****
at java.io.BufferedInputStream.read(Unknown Source)****
- locked <0x08f5c2d0> (a java.io.BufferedInputStream)****
at java.io.FilterInputStream.read(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)* ***
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)** **
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)****
****
"JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000]****
java.lang.Thread.State: TIMED_WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x08e9bdb0> (a [I)****
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)****
- locked <0x08e9bdb0> (a [I)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)****
at java.util.concurrent.DelayQueue.take(Unknown Source)*** *
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)****
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000]****
java.lang.Thread.State: RUNNABLE****
at java.net.PlainSocketImpl.socketAccept(Native Method)*** *
at java.net.PlainSocketImpl.accept(Unknown Source)****
- locked <0x08e9e158> (a java.net.SocksSocketImpl)****
at java.net.ServerSocket.implAccept(Unknown Source)****
at java.net.ServerSocket.accept(Unknown Source)****
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000]****
java.lang.Thread.State: TIMED_WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b1270> (a sun.misc.GC$LatencyLock)****
at sun.misc.GC$Daemon.run(Unknown Source)****
- locked <0x089b1270> (a sun.misc.GC$LatencyLock)****
****
Locked ownable synchronizers:****
- None****
****
"Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000]****
java.lang.Thread.State: WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock)****
at java.lang.ref.ReferenceQueue.remove(Unknown Source)****
- locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) ****
at java.lang.ref.ReferenceQueue.remove(Unknown Source)****
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000]****
java.lang.Thread.State: WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) ****
at java.lang.Object.wait(Object.java:485)****
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)****
- locked <0x089b2878> (a java.lang.ref.Reference$Lock)****
****
Locked ownable synchronizers:****
- None****
****
"VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable ****
****
"VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition ****
****
JNI global references: 21240****
****
****
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Hector Geraldino *Sent:* Monday, December 19, 2011 8:29 AM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client****
****
Hi Neil,****
****
Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on.****
****
IIRC I’ve fixed a couple of bugs for this library, but the patches haven’t been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you.****
****
Good luck!****
****
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Friday, December 16, 2011 7:09 PM *To:* FreeSWITCH-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Threads remain after calling close on Java client****
****
Hi,****
****
I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. ****
****
The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully?****
****
Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging.****
****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
****
****
Thanks,****
****
Neil Davis****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
hello , does anybody know how to set up the codec selection negotiation list in freeswitch? How can you know which codec has been selected for such a call and which codecs are offered from telephone A and B. I am using log 7 thank you in advance k.regards Sherif
it's found here: http://wiki.freeswitch.org/wiki/Codec_negotiation hope this answers your question. On Thu, Dec 22, 2011 at 5:25 AM, Sherif Omran <sherifomran2000@yahoo.com>wrote:
hello ,
does anybody know how to set up the codec selection negotiation list in freeswitch? How can you know which codec has been selected for such a call and which codecs are offered from telephone A and B.
I am using log 7
thank you in advance
k.regards Sherif
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi, You can set codec for a phone from dialplan using absolute_codec_string parameter as stated in codec negotiation link. Trans-coding issues in case of G729 or G723 can be handled as: http://wiki.freeswitch.org/wiki/Sofia#sip_renegotiate_codec_on_reinvite See http://wiki.freeswitch.org/wiki/Channel_Variables#Codec_Related for codec related variables. Also, there is a configuration in internal and external profiles to define a list of codec in each profile which can be set in global configuration http://wiki.freeswitch.org/wiki/Config_external.xml <param name="codec-prefs" value="$${outbound_codec_prefs}"/> you can see SIP dialogues on each phone IP to verify which codec are being offered ngrep -q -Wbyline '' port 5060 and host x.x.x.x Regards, Zohair Raza On Thu, Dec 22, 2011 at 2:54 AM, Nandy Dagondon <gcd@i.ph> wrote:
it's found here: http://wiki.freeswitch.org/wiki/Codec_negotiation
hope this answers your question.
On Thu, Dec 22, 2011 at 5:25 AM, Sherif Omran <sherifomran2000@yahoo.com>wrote:
hello ,
does anybody know how to set up the codec selection negotiation list in freeswitch? How can you know which codec has been selected for such a call and which codecs are offered from telephone A and B.
I am using log 7
thank you in advance
k.regards Sherif
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Zohair, In which file should i put this parameter <param name="inbound-late-negotiation" value="true"/> How does this command work ngrep -q -Wbyline '' port 5060 and host x.x.x.x Should i make a call and during the call type it? kind regards, Sherif --- On Thu, 12/22/11, Zohair Raza <engineerzuhairraza@gmail.com> wrote: From: Zohair Raza <engineerzuhairraza@gmail.com> Subject: Re: [Freeswitch-users] how is codec selected To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Date: Thursday, December 22, 2011, 9:00 AM Hi, You can set codec for a phone from dialplan using absolute_codec_string parameter as stated in codec negotiation link. Trans-coding issues in case of G729 or G723 can be handled as: http://wiki.freeswitch.org/wiki/Sofia#sip_renegotiate_codec_on_reinvite See http://wiki.freeswitch.org/wiki/Channel_Variables#Codec_Related for codec related variables. Also, there is a configuration in internal and external profiles to define a list of codec in each profile which can be set in global configurationhttp://wiki.freeswitch.org/wiki/Config_external.xml <param name="codec-prefs" value="$${outbound_codec_prefs}"/> you can see SIP dialogues on each phone IP to verify which codec are being offered ngrep -q -Wbyline '' port 5060 and host x.x.x.x Regards, Zohair Raza On Thu, Dec 22, 2011 at 2:54 AM, Nandy Dagondon <gcd@i.ph> wrote: it's found here: http://wiki.freeswitch.org/wiki/Codec_negotiation hope this answers your question. On Thu, Dec 22, 2011 at 5:25 AM, Sherif Omran <sherifomran2000@yahoo.com> wrote: hello , does anybody know how to set up the codec selection negotiation list in freeswitch? How can you know which codec has been selected for such a call and which codecs are offered from telephone A and B. I am using log 7 thank you in advance k.regards Sherif _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -----Inline Attachment Follows----- _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi, You can find it in sofia.conf.xml For ngrep, you need to have ngrep installed in your system try "yum install ngrep" if you are using CentOS/RedHat and "apt-get install ngrep" if Ubuntu Yes, you need to make a call, as you want to know which codecs are being carried from leg A to FS and FS to Log B Regards, Zohair Raza On Thu, Dec 22, 2011 at 3:02 PM, Sherif Omran <sherifomran2000@yahoo.com>wrote:
Hi Zohair,
In which file should i put this parameter <param name="inbound-late-negotiation" value="true"/>
How does this command work
ngrep -q -Wbyline '' port 5060 and host x.x.x.x
Should i make a call and during the call type it?
kind regards, Sherif
--- On *Thu, 12/22/11, Zohair Raza <engineerzuhairraza@gmail.com>* wrote:
From: Zohair Raza <engineerzuhairraza@gmail.com> Subject: Re: [Freeswitch-users] how is codec selected To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Date: Thursday, December 22, 2011, 9:00 AM
Hi,
You can set codec for a phone from dialplan using absolute_codec_string parameter as stated in codec negotiation link.
Trans-coding issues in case of G729 or G723 can be handled as: http://wiki.freeswitch.org/wiki/Sofia#sip_renegotiate_codec_on_reinvite
See http://wiki.freeswitch.org/wiki/Channel_Variables#Codec_Related for codec related variables.
Also, there is a configuration in internal and external profiles to define a list of codec in each profile which can be set in global configuration http://wiki.freeswitch.org/wiki/Config_external.xml
<param name="codec-prefs" value="$${outbound_codec_prefs}"/>
you can see SIP dialogues on each phone IP to verify which codec are being offered
ngrep -q -Wbyline '' port 5060 and host x.x.x.x
Regards, Zohair Raza
On Thu, Dec 22, 2011 at 2:54 AM, Nandy Dagondon <gcd@i.ph<http://mc/compose?to=gcd@i.ph>
wrote:
it's found here: http://wiki.freeswitch.org/wiki/Codec_negotiation
hope this answers your question.
On Thu, Dec 22, 2011 at 5:25 AM, Sherif Omran <sherifomran2000@yahoo.com<http://mc/compose?to=sherifomran2000@yahoo.com>
wrote:
hello ,
does anybody know how to set up the codec selection negotiation list in freeswitch? How can you know which codec has been selected for such a call and which codecs are offered from telephone A and B.
I am using log 7
thank you in advance
k.regards Sherif
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <http://mc/compose?to=consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<http://mc/compose?to=FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <http://mc/compose?to=consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<http://mc/compose?to=FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-----Inline Attachment Follows-----
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <http://mc/compose?to=consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<http://mc/compose?to=FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Sure, I can post it to Jira. Can you please help me to find an appropriate category to create the bug report? There is no project category for freeswitch-contrib projects. Also I'm not sure the patch format is compatible with git (I personally use SVN, don't know if the diff format is different between these two). From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, December 21, 2011 4:14 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client if this patch is necessary can you post it to a jira? On Wed, Dec 21, 2011 at 1:26 PM, Hector Geraldino <Hector.Geraldino@ip-soft.net<mailto:Hector.Geraldino@ip-soft.net>> wrote: Hi Neil, This doesn't seem to be the same concurrency issue I had, but I'm attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls. In case it doesn't work you might try to "manually" close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Neil Davis Sent: Wednesday, December 21, 2011 11:17 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a "waiting on condition" state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Hector Geraldino Sent: Monday, December 19, 2011 8:29 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I've fixed a couple of bugs for this library, but the patches haven't been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com<mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net<http://irc.freenode.net> #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
you can just file it under esl or general, the patch should be ok as long as its a unified diff or you can get the FS tree and patch your local copy and submit a git diff from that if all else fails. On Wed, Dec 21, 2011 at 4:33 PM, Hector Geraldino < Hector.Geraldino@ip-soft.net> wrote:
Sure, I can post it to Jira. Can you please help me to find an appropriate category to create the bug report? There is no project category for freeswitch-contrib projects. ****
** **
Also I’m not sure the patch format is compatible with git (I personally use SVN, don’t know if the diff format is different between these two).*** *
** **
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Anthony Minessale *Sent:* Wednesday, December 21, 2011 4:14 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client****
** **
if this patch is necessary can you post it to a jira?****
On Wed, Dec 21, 2011 at 1:26 PM, Hector Geraldino < Hector.Geraldino@ip-soft.net> wrote:****
Hi Neil,****
****
This doesn’t seem to be the same concurrency issue I had, but I’m attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls.****
****
In case it doesn’t work you might try to “manually” close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens.****
****
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Wednesday, December 21, 2011 11:17 AM****
*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client****
****
Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a “waiting on condition” state that appear to have to do with the netty package on which the Freeswitch client is dependent.****
****
****
2011-12-21 09:05:17****
Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode):****
****
"DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) ****
at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) ****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000]****
java.lang.Thread.State: RUNNABLE****
at java.net.SocketInputStream.socketRead0(Native Method)** **
at java.net.SocketInputStream.read(Unknown Source)****
at java.io.BufferedInputStream.fill(Unknown Source)****
at java.io.BufferedInputStream.read(Unknown Source)****
- locked <0x08f5c2d0> (a java.io.BufferedInputStream)****
at java.io.FilterInputStream.read(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)* ***
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)** **
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)****
****
"JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000]****
java.lang.Thread.State: TIMED_WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x08e9bdb0> (a [I)****
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)****
- locked <0x08e9bdb0> (a [I)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)****
at java.util.concurrent.DelayQueue.take(Unknown Source)*** *
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)****
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000]****
java.lang.Thread.State: TIMED_WAITING (parking)****
at sun.misc.Unsafe.park(Native Method)****
- parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack)****
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)****
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) ****
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)****
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000]****
java.lang.Thread.State: RUNNABLE****
at java.net.PlainSocketImpl.socketAccept(Native Method)*** *
at java.net.PlainSocketImpl.accept(Unknown Source)****
- locked <0x08e9e158> (a java.net.SocksSocketImpl)****
at java.net.ServerSocket.implAccept(Unknown Source)****
at java.net.ServerSocket.accept(Unknown Source)****
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)****
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)****
at java.lang.Thread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000]****
java.lang.Thread.State: TIMED_WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b1270> (a sun.misc.GC$LatencyLock)****
at sun.misc.GC$Daemon.run(Unknown Source)****
- locked <0x089b1270> (a sun.misc.GC$LatencyLock)****
****
Locked ownable synchronizers:****
- None****
****
"Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000]****
java.lang.Thread.State: RUNNABLE****
****
Locked ownable synchronizers:****
- None****
****
"Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000]****
java.lang.Thread.State: WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock)****
at java.lang.ref.ReferenceQueue.remove(Unknown Source)****
- locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) ****
at java.lang.ref.ReferenceQueue.remove(Unknown Source)****
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)****
****
Locked ownable synchronizers:****
- None****
****
"Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000]****
java.lang.Thread.State: WAITING (on object monitor)****
at java.lang.Object.wait(Native Method)****
- waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) ****
at java.lang.Object.wait(Object.java:485)****
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)****
- locked <0x089b2878> (a java.lang.ref.Reference$Lock)****
****
Locked ownable synchronizers:****
- None****
****
"VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable ****
****
"VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition ****
****
JNI global references: 21240****
****
****
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Hector Geraldino *Sent:* Monday, December 19, 2011 8:29 AM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client****
****
Hi Neil,****
****
Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on.****
****
IIRC I’ve fixed a couple of bugs for this library, but the patches haven’t been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you.****
****
Good luck!****
****
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Friday, December 16, 2011 7:09 PM *To:* FreeSWITCH-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Threads remain after calling close on Java client****
****
Hi,****
****
I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. ****
****
The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully?****
****
Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging.****
****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads****
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak.****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks****
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. ****
****
****
Thanks,****
****
Neil Davis****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
****
** **
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Done. http://jira.freeswitch.org/browse/ESL-61 From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, December 21, 2011 5:36 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client you can just file it under esl or general, the patch should be ok as long as its a unified diff or you can get the FS tree and patch your local copy and submit a git diff from that if all else fails. On Wed, Dec 21, 2011 at 4:33 PM, Hector Geraldino <Hector.Geraldino@ip-soft.net<mailto:Hector.Geraldino@ip-soft.net>> wrote: Sure, I can post it to Jira. Can you please help me to find an appropriate category to create the bug report? There is no project category for freeswitch-contrib projects. Also I'm not sure the patch format is compatible with git (I personally use SVN, don't know if the diff format is different between these two). From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: Wednesday, December 21, 2011 4:14 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client if this patch is necessary can you post it to a jira? On Wed, Dec 21, 2011 at 1:26 PM, Hector Geraldino <Hector.Geraldino@ip-soft.net<mailto:Hector.Geraldino@ip-soft.net>> wrote: Hi Neil, This doesn't seem to be the same concurrency issue I had, but I'm attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls. In case it doesn't work you might try to "manually" close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Neil Davis Sent: Wednesday, December 21, 2011 11:17 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a "waiting on condition" state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Hector Geraldino Sent: Monday, December 19, 2011 8:29 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I've fixed a couple of bugs for this library, but the patches haven't been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com<mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net<http://irc.freenode.net> #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900<tel:%2B19193869900> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com<mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net<http://irc.freenode.net> #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
Hector, Thanks for your suggestions. I tried both your fix and modifying the close method as suggested, but I still have threads hanging around in the TIMED-WAITING status. They eventually terminate after about a minute. For the time being, I’m just stalling the Tomcat shutdown with Thread.sleeps until all the hanging threads have terminated. Tomcat is then able to shutdown gracefully. I’ll look a little more at the esl code as I have time and update the mailing list if I find a solution. Thanks, Neil Davis *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Hector Geraldino *Sent:* Wednesday, December 21, 2011 12:26 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, This doesn’t seem to be the same concurrency issue I had, but I’m attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls. In case it doesn’t work you might try to “manually” close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens. *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Wednesday, December 21, 2011 11:17 AM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a “waiting on condition” state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Hector Geraldino *Sent:* Monday, December 19, 2011 8:29 AM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I’ve fixed a couple of bugs for this library, but the patches haven’t been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! *From:* freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Neil Davis *Sent:* Friday, December 16, 2011 7:09 PM *To:* FreeSWITCH-users@lists.freeswitch.org *Subject:* [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
Yeah, I was going to suggest you that, as there's a chance for those threads to come to an end after a few seconds. The last thing I can think about is to create your own variable instances of Executors and pass them to the ClientBootstrap constructor on the connect() method of the Client class: ClientBootstrap bootstrap = new ClientBootstrap( new NioClientSocketChannelFactory( Executors.newCachedThreadPool(), <- replace this with your own instances Executors.newCachedThreadPool() ) ); By doing so, you can later call the ExecutorService.shutdown() method to immediately terminate all running threads. I think it's worth to give it a shot. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Neil Davis Sent: Wednesday, December 21, 2011 6:19 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hector, Thanks for your suggestions. I tried both your fix and modifying the close method as suggested, but I still have threads hanging around in the TIMED-WAITING status. They eventually terminate after about a minute. For the time being, I'm just stalling the Tomcat shutdown with Thread.sleeps until all the hanging threads have terminated. Tomcat is then able to shutdown gracefully. I'll look a little more at the esl code as I have time and update the mailing list if I find a solution. Thanks, Neil Davis From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Hector Geraldino Sent: Wednesday, December 21, 2011 12:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, This doesn't seem to be the same concurrency issue I had, but I'm attaching the patch that fixes my issue anyway. Feel free to test it and send me back the restuls. In case it doesn't work you might try to "manually" close the channel by modifying the close() method on the org.freeswitch.esl.client.inbound.Client. Try to do a channel.disconnect(); and channel=null; and see what happens. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Neil Davis Sent: Wednesday, December 21, 2011 11:17 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Here is a thread dump of my Tomcat process at the point when it is hanging on shutdown. There are a number of threads in a "waiting on condition" state that appear to have to do with the netty package on which the Freeswitch client is dependent. 2011-12-21 09:05:17 Full thread dump Java HotSpot(TM) Client VM (14.3-b01 mixed mode): "DestroyJavaVM" prio=6 tid=0x546f7400 nid=0x640 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "pool-5-thread-16" prio=6 tid=0x544b5800 nid=0x1098 waiting on condition [0x55e4f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-15" prio=6 tid=0x54f7f400 nid=0x10f0 waiting on condition [0x54e5f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-14" prio=6 tid=0x544b3c00 nid=0xdb4 waiting on condition [0x5670f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-13" prio=6 tid=0x544b4000 nid=0x12c8 waiting on condition [0x566bf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-12" prio=6 tid=0x544b4800 nid=0xe18 waiting on condition [0x5666f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-11" prio=6 tid=0x544b3000 nid=0x620 waiting on condition [0x5661f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-10" prio=6 tid=0x552a1800 nid=0x131c waiting on condition [0x565cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-9" prio=6 tid=0x552a1400 nid=0x1710 waiting on condition [0x5657f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-8" prio=6 tid=0x552a0c00 nid=0x1094 waiting on condition [0x5652f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-7" prio=6 tid=0x552a0800 nid=0x1040 waiting on condition [0x564df000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-6" prio=6 tid=0x552a0000 nid=0x179c waiting on condition [0x5648f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-5" prio=6 tid=0x5529fc00 nid=0x3e4 waiting on condition [0x5643f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-4" prio=6 tid=0x5529f400 nid=0x63c waiting on condition [0x563ef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-3" prio=6 tid=0x5529f000 nid=0x17e8 waiting on condition [0x5639f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-2" prio=6 tid=0x5529e800 nid=0x574 waiting on condition [0x5634f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-4-thread-1" prio=6 tid=0x5529e400 nid=0x10cc waiting on condition [0x562ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970378> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-3-thread-1" prio=6 tid=0x546f8800 nid=0x654 waiting on condition [0x562af000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x09970580> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "pool-5-thread-1" prio=6 tid=0x546f8000 nid=0x594 waiting on condition [0x5625f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at org.jboss.netty.util.internal.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:766) at org.jboss.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:673) at org.jboss.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1164) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(2)-10.0.0.22" daemon prio=6 tid=0x551db800 nid=0xf6c runnable [0x5593f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) - locked <0x08f5c2d0> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - <0x08f62b10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "JMX server connection timeout 22" daemon prio=6 tid=0x5474fc00 nid=0x424 in Object.wait() [0x558ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x08e9bdb0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source) - locked <0x08e9bdb0> (a [I) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=6 tid=0x5511d400 nid=0x1420 waiting on condition [0x5589f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08e9bdd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Connection(idle)" daemon prio=6 tid=0x55176800 nid=0x884 waiting on condition [0x557ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x08ebd088> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source) at java.util.concurrent.SynchronousQueue.poll(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "RMI TCP Accept-0" daemon prio=6 tid=0x5518bc00 nid=0x163c runnable [0x557af000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(Unknown Source) - locked <0x08e9e158> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(Unknown Source) at java.net.ServerSocket.accept(Unknown Source) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: - None "GC Daemon" daemon prio=2 tid=0x5464b000 nid=0x1718 in Object.wait() [0x5497f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b1270> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(Unknown Source) - locked <0x089b1270> (a sun.misc.GC$LatencyLock) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=6 tid=0x01a12c00 nid=0x14fc runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=10 tid=0x01a0f800 nid=0x1ec waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "JDWP Event Helper Thread" daemon prio=6 tid=0x01a01400 nid=0x173c runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Attach Listener" daemon prio=10 tid=0x019f5000 nid=0x13a4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=10 tid=0x019ea000 nid=0x17b8 runnable [0x00000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x019bb400 nid=0x1720 in Object.wait() [0x53fcf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x089b2858> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x019ba000 nid=0x53c in Object.wait() [0x53f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x089b2878> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x089b2878> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=10 tid=0x019b7400 nid=0xab8 runnable "VM Periodic Task Thread" prio=10 tid=0x01a1bc00 nid=0xc7c waiting on condition JNI global references: 21240 From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Hector Geraldino Sent: Monday, December 19, 2011 8:29 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] Threads remain after calling close on Java client Hi Neil, Can you get a thread dump of the tomcat process to try to figure out what this problem is about? Or at least, try to connect the jconsole to the tomcat process and get the StackTrace of one of these threads to have a better idea of what is going on. IIRC I've fixed a couple of bugs for this library, but the patches haven't been tested by the main developer (dvarnes) nor integrated on the repository (freeswitch-contrib). If this problem can be fixed with my patched code, I would be happy to share it with you. Good luck! From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org]<mailto:[mailto:freeswitch-users-bounces@lists.freeswitch.org]> On Behalf Of Neil Davis Sent: Friday, December 16, 2011 7:09 PM To: FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> Subject: [Freeswitch-users] Threads remain after calling close on Java client Hi, I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down. The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully? Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. Thanks, Neil Davis
participants (7)
-
Anthony Minessale
-
Dave May
-
Hector Geraldino
-
Nandy Dagondon
-
Neil Davis
-
Sherif Omran
-
Zohair Raza