Kuali Rice Development
  1. Kuali Rice Development
  2. KULRICE-6585

Look into possible memory and thread local leaks reported by tomcat on a shutdown

    Details

    • Type: Bug Fix Bug Fix
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.1.0-m3, 2.1
    • Component/s: Analysis
    • Security Level: Public (Public: Anyone can view)
    • Labels:
      None
    • Similar issues:
      KULRICE-8970Memory leak errors when trying to shut down Tomcat
      KULRICE-4959On shutdown, KS' Rice outputs messages that may indicate Quartz isn't shutting down correctly
      KULRICE-4957Update surefire plugin due to memory leak
      KULRICE-6584GlobalResourceLoader shutdown happening before all background threads have shutdown
      KULRICE-8770easyXDM issues with Firefox and Tomcat
      KULRICE-3383KualiIncidentReport.jsp doesn't work on Tomcat 5.5.27+/Tomcat 6.0 (and possibly other screens?)
      KULRICE-13448Investigate Rice sampleapp out of memory exceptions
      KULRICE-9285AccountManagerMaintenanceDocumentTest fails with NPE locally and in CI with unable to get class for unknown documentTypeName 'AccountManagerMaintenanceDocument'
      KULRICE-5341UserSession gets bound to a thread, and eDocLite documents are not establishing user session properly
      KULRICE-13408Very large HistoryFlows cause Out Of Memory error.
    • KAI Review Status:
      Not Required
    • KTI Review Status:
      Not Required

      Description

      When I shutdown in Tomcat 7 (after using some KNS and KRAD screens) I see the following dumped to the screens:

      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [/kr-dev] appears to have started a thread named [JotmBatch] but has failed to stop it. This is very likely to create a memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [/kr-dev] appears to have started a thread named [JotmClock] but has failed to stop it. This is very likely to create a memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [/kr-dev] appears to have started a thread named [FileWatchdog] but has failed to stop it. This is very likely to create a memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [/kr-dev] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.core.api.util.ContextClassLoaderBinder$1] (value [org.kuali.rice.core.api.util.ContextClassLoaderBinder$1@3bb0b23f]) and a value of type [java.util.ArrayList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [com.sun.xml.bind.v2.ClassFactory$1] (value [com.sun.xml.bind.v2.ClassFactory$1@4d8664b6]) and a value of type [java.util.WeakHashMap] (value [{class org.kuali.rice.core.api.util.jaxb.QNameAsStringAdapter=java.lang.ref.WeakReference@323e107d, class org.kuali.rice.ksb.api.bus.support.SoapServiceConfiguration=java.lang.ref.WeakReference@8ac9850}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kew.engine.RouteContext$1] (value [org.kuali.rice.kew.engine.RouteContext$1@7b43b0a2]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.kew.engine.RouteContext@19019ae7]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [com.sun.xml.bind.v2.runtime.Coordinator$1] (value [com.sun.xml.bind.v2.runtime.Coordinator$1@3860c084]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@57ff3706]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.core.api.util.ContextClassLoaderBinder$1] (value [org.kuali.rice.core.api.util.ContextClassLoaderBinder$1@3bb0b23f]) and a value of type [java.util.ArrayList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@520d6b2b]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@4042bc63]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@50b7abe9]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@4a777765]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@53bdd907]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$1] (value [org.kuali.rice.kns.util.KNSGlobalVariables$1@3475980c]) and a value of type [org.kuali.rice.kns.util.MessageList] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.kns.util.KNSGlobalVariables$2] (value [org.kuali.rice.kns.util.KNSGlobalVariables$2@145156bc]) and a value of type [java.util.HashMap] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.kuali.rice.krad.util.GlobalVariables$1] (value [org.kuali.rice.krad.util.GlobalVariables$1@68155b0f]) and a value of type [java.util.LinkedList] (value [[org.kuali.rice.krad.util.GlobalVariables@3fc2c8ae]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [/kr-dev] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@640bb179]) and a value of type [java.util.Hashtable] (value [{}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jan 26, 2012 10:46:02 PM org.apache.coyote.AbstractProtocol stop

      From this we can see some threads not getting cleaned up as well as a bunch of dangling ThreadLocals hanging around. It would be good to try and clean these up as much as possible (to at least give people a shot to be able to hot deploy this sucker).

        Issue Links

          Activity

          Hide
          Aaron Hamid (Inactive) added a comment -

          The org.kuali.rice.* ThreadLocals are no longer noted in undeploy warnings, and the Jotm background threads are now getting shut down as well. com.sun.xml and log4j threadlocals are still present. Undeploy and redeploy appears to nominally work.

          Show
          Aaron Hamid (Inactive) added a comment - The org.kuali.rice.* ThreadLocals are no longer noted in undeploy warnings, and the Jotm background threads are now getting shut down as well. com.sun.xml and log4j threadlocals are still present. Undeploy and redeploy appears to nominally work.
          Hide
          Aaron Hamid (Inactive) added a comment -

          Peter do you know of an easy way to reproduce the xml-sec warning? Maybe just hit up a bus endpoint via a url in the browser?

          Show
          Aaron Hamid (Inactive) added a comment - Peter do you know of an easy way to reproduce the xml-sec warning? Maybe just hit up a bus endpoint via a url in the browser?
          Hide
          Aaron Hamid (Inactive) added a comment - - edited

          I have implemented a fix for the SignatureAlgorithm caches, keys are no longer printed on undeploy.

          Log4j warnings are related to mapped diagnostic contexts. We might want to include MDC.clear() in a servlet filter to ensure these get cleared after every request. However it appears there is still a bug in how the clearing is implemented in log4j https://issues.apache.org/bugzilla/show_bug.cgi?id=50486 so we might have to hack that as well.

          Show
          Aaron Hamid (Inactive) added a comment - - edited I have implemented a fix for the SignatureAlgorithm caches, keys are no longer printed on undeploy. Log4j warnings are related to mapped diagnostic contexts. We might want to include MDC.clear() in a servlet filter to ensure these get cleared after every request. However it appears there is still a bug in how the clearing is implemented in log4j https://issues.apache.org/bugzilla/show_bug.cgi?id=50486 so we might have to hack that as well.
          Hide
          Peter Giles (Inactive) added a comment - - edited

          Hi Aaron, sorry for the slow response. You've probably already got this figured out, but I think what you suggest for repro-ing the warning will work provided bus security is turned on for the service and the keystore is properly config-ed.

          Show
          Peter Giles (Inactive) added a comment - - edited Hi Aaron, sorry for the slow response. You've probably already got this figured out, but I think what you suggest for repro-ing the warning will work provided bus security is turned on for the service and the keystore is properly config-ed.
          Hide
          Aaron Hamid (Inactive) added a comment - - edited

          This is probably not a guaranteed bullet-proof solution but I think we are in much better shape:

          • org.kuali.rice.* ThreadLocals are getting cleaned up (notably the *GlobalVariables and RouteContext TLs)
          • JOTM background threads are getting shut down
          • Apache xml-sec SignatureAlgorithm key caching threadlocals are removed so private key info is not printed to log
          • Log4J MDCs (and NDCs) are removed on a request-basis by a servlet filter (in theory there may still be MDCs/NCDs on non-request/threadpool threads, but I didn't see any warnings)
          Show
          Aaron Hamid (Inactive) added a comment - - edited This is probably not a guaranteed bullet-proof solution but I think we are in much better shape: org.kuali.rice.* ThreadLocals are getting cleaned up (notably the *GlobalVariables and RouteContext TLs) JOTM background threads are getting shut down Apache xml-sec SignatureAlgorithm key caching threadlocals are removed so private key info is not printed to log Log4J MDCs (and NDCs) are removed on a request-basis by a servlet filter (in theory there may still be MDCs/NCDs on non-request/threadpool threads, but I didn't see any warnings)

            People

            • Assignee:
              Aaron Hamid (Inactive)
              Reporter:
              Eric Westfall
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1 week
                1w
                Remaining:
                Remaining Estimate - 1 week
                1w
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Structure Helper Panel