I experienced similar problems when load testing Plivo on the latest Git, but didn't feel like I had enough data gathered for a "proper" report. After each call, the remote ESL socket would be left in a CLOSE_WAIT state.
I think the problem started on December 8th, with the resolution of this Jira:
http://jira.freeswitch.org/browse/FS-3750
When I rolled the code back to the revision just prior to this change, my problems went away.
git checkout 1868e145201cc6ba5a14d6929695977780917a38
Dave.
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Neil Davis
Sent: Friday, December 16, 2011 7:09 PM
To: FreeSWITCH-users@lists.freeswitch.org
Subject: [Freeswitch-users] Threads remain after calling close on Java client
Hi,
I built a web application that connects to Freeswitch using the org.freeswitch.esl.client.Client. I connect the Client object from a Spring annotated service that I call from a Spring controller. I put the connected client in my ServletContext, so I can access it later to call client.cancelEventSubscriptions() and client.close() from my ServletContextListener contextDestroyed method when Tomcat is shutting down.
The problem I'm having is that even after I call close on the client, there are still a bunch of active threads that the client has spawned in the background. These threads are causing Tomcat to hang when I'm shutting down. Can anyone suggest an approach that would enable my application to disconnect the Freeswitch client when Tomcat is shutting down that would allow Tomcat to shutdown gracefully?
Below are errors from my Tomcat log for the threads that I have identified as being related to the Freeswitch client. I don't know how I can get to these threads to interrupt them and Client.close() seems to leave them hanging.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-3-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-4-thread-1] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-2] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-4] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-5] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-6] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-7] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-8] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-9] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-10] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-11] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-12] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-13] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-14] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-15] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/socketspy] appears to have started a thread named [pool-5-thread-16] but has failed to stop it. This is very likely to create a memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalBoolean] (value [org.jboss.netty.util.internal.ThreadLocalBoolean@186e192]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@373f86}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.internal.ThreadLocalRandom$1] (value [org.jboss.netty.util.internal.ThreadLocalRandom$1@12bb519]) and a value of type [org.jboss.netty.util.internal.ThreadLocalRandom] (value [org.jboss.netty.util.internal.ThreadLocalRandom@7e9dbc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Dec 16, 2011 12:04:59 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/socketspy] created a ThreadLocal with key of type [org.jboss.netty.util.CharsetUtil$1] (value [org.jboss.netty.util.CharsetUtil$1@14d8e1]) and a value of type [java.util.IdentityHashMap] (value [{windows-1252=sun.nio.cs.MS1252$Encoder@a5b041}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Thanks,
Neil Davis