Wednesday, July 14, 2021

ThreadLocal Memory Leak in Java web application - Tomcat

ThreadLocal variables are infamous for creating memory leaks. A memory leak in Java is the amount of memory hold by an object which is not in use and should have been garbage collected, but because of unintended strong references, they still live in Java heap space. There are many ways memory leak can be caused in Java but when this memory leak is caused due to the ThreadLocal variable, it’s referred to as ThreadLocal memory leak. In our last post about the ThreadLocal variable, we have seen How the ThreadLocal variable can make SimpleDateFormat thread-safe and also raised points that in a managed environment like the J2EE application server or a web servers like Tomcat, Jetty, WebSphere or Weblogic use of ThreadLocal should be avoided.


In this post, we will go a little deep and find out How ThreadLocal variables create a memory leak in Java web applications, much like we have seen How to fix PermGen memory leak in Tomcat. If you are not familiar with ThreadLocal variables I suggest going through my previous post. Before going into detail let's recap How ThreadLocal class works in Java.




How ThreadLocal works in Java :

ThreadLocal memory leak in Java web application in tomcatThreadLocal in Java is a mechanism to provide a separate copy of the shared object to every Thread. So that they no longer shared between multiple Threads and remain thread-safe. ThreadLocal variables are stored in a special map called ThreadLocalMap which is designed to hold thread-local objects, it uses WeakReferences for keys. 

Since every Thread has a strong reference to there copy of ThreadLocal variables, they are not garbage collected until Thread is Alive and this is what creates memory leak in a typical J2EE web application. This is explained in detail in the next section “How ThreadLocal variable creates memory leak in Java”.



How ThreadLocal creates memory leak in Java

In web server and application servers like Tomcat or WebLogic, web-app is loaded by a different ClassLoader than which is used by Server itself. This ClassLoader loads and unloads classes from web applications. Web servers also maintain ThreadPool, which is a collection of the worker thread, to server HTTP requests. 

Now if one of the Servlets or any other Java class from web application creates a ThreadLocal variable during request processing and doesn't remove it after that, copy of that Object will remain with worker Thread and since life-span of worker Thread is more than web app itself, it will prevent the object and ClassLoader, which uploaded the web app, from being garbage collected. This will create a memory leak in Server. 

Now if you do this couple of time you may see java.lang.OutOfMemoryError: PermGen space . Now, this brings an important question, is it possible to use the ThreadLocal variable safely in a managed environment? The answer is Yes,, but that requires a careful usage of the ThreadLocal variable and making sure to remove the object from ThreadLocal once done.



How to use ThreadLocal safely in Java Web application

Many people use Filters to initialize and remove ThreadLocal variables. You can initialize ThreadLocal in the filter, put some expensive object as ThreadLocal and once request has been processed remove it from ThreadLocal as shown in below example:

public void doFilter(ServeletRequest request, ServletResponse){
        try{

          //set ThreadLocal variable
        chain.doFilter(request, response)

        }finally{
          //remove threadLocal variable.
        }
}

That’s all on How the ThreadLocal variable creates memory leak in Java. Having said that, If not necessary or You can manage without ThreadLocal variable than it’s best to avoid using ThreadLocal in managed environments like J2EE web and application servers.



9 comments:

  1. Could you explain why it causes the memory leak ? The variable is removed from the ThreadLocal at evey request, so it should prevent memory leaking.

    ReplyDelete
  2. "are not garbage collected until Thread is Alive"
    Don't you mean "Dead"

    Regards
    Ren

    ReplyDelete
  3. @Anonymous, yes if you remove ThreadLocal at every request that it won't cause memory leak. Memory leak caused when you forget to remove ThreadLocal variable once done.

    ReplyDelete
  4. Just to add on what Javin said, Yes this is true that threadlocal variables can prevent a class and subsequently classloader which is loaded them from being garbage collected, but this will only create issue if you store application classes in ThreadLocal, which is loaded by web-app classloader. If you store classes from JDK e.g. SimpleDateFormat, it will not create memory leak in Java web application, because they are loaded by BootStrap classloader, so they are not reloaded each time you deploy and un-deploy your web application. Having said that, it's best practice to remove any variable from thread local, as soon as you are done with that. For example if you are storing user specific information e.g. username as threadlocal in Filter class than you can also remove them, once request is processed.

    ReplyDelete
  5. If your server is pooling the threads then in that case it will create the memory leak. Basically after serving your request, thread will be returned to the pool(not dead) . So, the thread local object will be associated with the thread even after the utility of it. So, best way is to remove the thread local object from the thread as soon as you are finished with it.

    ReplyDelete
  6. Now this makes sense, threads in the pool managed by container doesn't gets destroyed when application reloads (Only destroyed when container re-starts again), this causes thread local variable to be still present, since worker threads are still present so as its associate Objects, which it is referencing to. and when you keep on reloading application again and again, every time new thread local objects created, but since they have strong reference with thread itself, they don't get GCed and causes Memory leaks. But these things are not true if you are creating threads inside application, since as soon as application terminates thread also gets destroyed and hence no reference to thread local object and hence can be GCed. Its not a Fault of Thread Local, Its because Container Thread Pool is not application scoped. Therefore when using Thread Local with Threads which are out of Scope of Application, even if application terminates threads still be there. Therefore as soon you are done with your thread local, clean the thread local variable from the thread..

    ReplyDelete
  7. What is the right way to remove the thread local object from the thread ?I see that ThreadLocal.remove() method is used in many posts. If ThreadLocal value is set to null or some default value like ThreadLocal.set(""), then also objects created in the application processors are GCed right ?

    ReplyDelete
  8. @Sriniva, yes, remove() is the right way to remove object from thread local but when you set null, the object should be freed which can hold the classloader and other stuff, but I have not tried that. May be it's not allowed and throw NPE somewhere.

    ReplyDelete
  9. When you use static ThreadLocal for SimpleDateFormat you can still get a class loader leak if you use double brace initialization for your SDF. For example in init method if you write: return new SimpleDateFormat(){{applyPattern("dd/MM/yyyy")}}; you will get class loader leak because you created an anonymous class that has a reference to your class loader.
    However in that case you could use a custom ThreadLocal that I created which would allow class loader to be GC-ed while preserving completely non-blocking access:
    https://github.com/codesinthedark/ImprovedThreadLocal

    ReplyDelete