What is Thread Pool?

Learn via video courses

Overview

We make use of Threads in Operating Systems to achieve parallel processing of tasks and to increase the throughput of our system. A Thread Pool is a collection of worker threads that efficiently execute asynchronous callbacks on behalf of the application. The thread pool is primarily used to reduce the number of application threads and provide management of the worker threads. Applications can queue work items, associate work with waitable handles, automatically queue based on a timer, and bind with I/O.

What is a Thread Pool?

As the name suggests, a Thread Pool is a pool or a collection of threads.

Let us take an example of a book library to understand thread pools better. A user comes to a library and requests for a book, if the book is not present in the library, the book has to be purchased first before allocating it to the user. However if the library anticipates in advance that there would be a demand for the book and stores the book before any user requests for it, they can save on the time delay of purchasing the book. Once the user visits the library, they can be immediately allocated the book. After consumption, the book will be returned to the library, ready for allocation to other users

The same concept can be applied to threads. Let us assume we have two tasks T1 and T2 which need to be executed one after the other. T1 requests the compiler/ application to spawn a thread where the task can execute, the compiler creates one, T1 does its processing, and then the compiler destroys one. Now T2 has to process, the same processs happens again. The total time of computation is as follows:

t(T1)+t(T2)+2creationTimeOfThread+2destructionTimeOfThreadt(T1) + t(T2) + 2 * creationTimeOfThread + 2 * destructionTimeOfThread

If the threads were already created in advance, and we do not destroy them after their use, instead return them to the pool, we can save up on a lot of compute time.

Thread Pool Architecture

The following diagram illustrates how a thread pool works Thread pool architecture

There are 3 threads in the pool: T1, T2 and T3.

Service1 and Service2 request for a thread from the thread pool, and are allocated thread T1 and T2 respectively

Service3 has finished its request and is ready to return thread T3 to the pool

Service4 has a request but since there are no threads available in the pool, its request cannot be served. The request will be parked to a waiting queue

Thread Pool in Java

In Java, a thread pool is created using ExecutorService

This will create a pool of 3 threads

Using the execute() function, a thread can be requested from the executorService

The Ideal Thread Pool

How many threads should be present in the pool? Let us understand the extreme cases first

  • If we create a lot of threads in the pool, we will face high memory utilization and degraded performance of the application
  • If we create too few threads, we would not get the benefits of using a thread pool

Java has an interesting way of creating a Thread pool using Executors.newChachedThreadPool(). This implementation creates an initial pool of zero threads, whenever there is a demand, more threads are created and added to the pool. Threads which remain idle for more than 60 seconds are removed from the pool

The parameters like number of threads in the pool, TTL of idle threads can be controlled. We can instantiate the pool with a starting number of threads, and more threads will be created dynamically if needed

How to know the starting thread count?
There is a formula to compute the ideal number of threads your pool should have:

Numberofthreads=NumberofAvailableCores(1+Waittime/Servicetime)Number of threads = Number of Available Cores * (1 + Wait time / Service time)

Wait Time: The total time spent when no processing is done, for example I/O operations, network calls Service Time: Actual processing time

For example, in a quad-core CPU, if a microservice spends 500ms on spent on I/O operations, and 50 ms on actual compute,

numberOfhreads=4(1+500/50)=44numberOfhreads = 4 * (1 + 500/50) = 44

Conclusion

  • Thread pools gives a way of creating multiple threads at start-up of an application
  • Thread pool manages the creation, allocation and destruction of threads
  • Java has a class ExecutorService which manages thread pools
  • User can control how many threads can be present in the thread pool using Executors.newChachedThreadPool() implementation of ExecutorService
  • We should not create a lot of threads in the pool, it will lead to high memory usage and slow performance overall. It might also impact the start-up time of the application
  • As a thumb rule we can use the formula Number of threads = Number of Available Cores * (1 + Wait time / Service time) to calculate the number of threads required in the pool