Briefly, this error occurs when Elasticsearch is unable to execute a scheduled task due to issues like high CPU usage, insufficient memory, or thread pool saturation. To resolve this, you can optimize your queries to reduce CPU usage, increase your server’s memory, or adjust the thread pool settings in Elasticsearch. Additionally, ensure that your tasks are not scheduled too closely together to avoid overloading the system.
This guide will help you check for common problems that cause the log ” failed to run scheduled task [{}] on thread pool [{}] ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: threadpool, task, pool, thread.
Overview
Elasticsearch uses threadpools to manage how requests are processed and to optimize the use of resources on each node in the cluster.
What it’s used for
The main threadpools are for search, get and write, but there are a number of others which you can see by running:
GET /_cat/thread_pool/?v&h=id,name,active,rejected,completed,size,type&pretty
You can see by running the above command that each node has a number of different thread pools, what the size and type of the thread pool are, and you can see which nodes have rejected operations. Elasticsearch automatically configures the threadpool management parameters based on the number of processors detected in each node.
Threadpool types
Fixed- a fixed number of threads, with a fixed queue size
thread_pool:
write:
size: 30
queue_size: 1000Scaling- a variable number of threads that Elasticsearch scales automatically according to workload.
thread_pool:
warmer:
core: 1
max: 8fixed_autoqueue_size- a fixed number of threads with a variable queue size which changes dynamically in order to maintain a target response time
thread_pool:
search:
size: 30
queue_size: 500
min_queue_size: 10
max_queue_size: 1000
auto_queue_frame_size: 2000
target_response_time: 1sExamples
To see which threads are using the highest CPU or taking the longest time you can use the following query. This may help find operations that are causing your cluster to underperform.
GET /_nodes/hot_threads
Notes and good things to know
In general it is not recommended to tweak threadpool settings. However, it is worth noting that the threadpools are set based upon the number of processors that Elasticsearch has detected on the underlying hardware. If that detection fails, then you should explicitly set the number of processors available in your hardware in elasticsearch.yml like this:
processors: 4
Most threadpools also have queues associated with them to enable Elasticsearch to store requests in memory while waiting for resources to become available to process the request. However the queues are usually of a finite size, and if that size becomes exceeded, then Elasticsearch will reject the request.
Sometimes you may be tempted to increase the queue size to prevent requests being rejected, but this will only treat the symptom and not the underlying cause of the problem. Indeed, it may even be counter productive, since by allowing a larger queue size, the node will need to use more memory to store the queue, and will have less space to actually manage requests. Furthermore increasing the queue size will also increase the length of time that operations are kept in the queue, resulting in client applications facing time out issues.
Usually, the only case where it can be justified to increase the queue size is where requests are received in uneven surges and you are unable to manage this process client-side.
You can monitor thread pools to better understand the performance of your Elasticsearch cluster. The Elasticsearch monitoring panel in Kibana shows your graphs of the search, get, and write thread queues and any queue rejections. Growing queues indicate that Elasticsearch is having difficulty keeping up with requests, and rejections indicate that queues have grown to the point that Elasticsearch rejects calls to the server.
Check the underlying causes of increases in queues. Try to balance activity across the nodes in the cluster and try to balance the demands on the cluster thread pool by taking actions on the client-side.
Overview
A task is an Elasticsearch operation, which can be any request performed on an Elasticsearch cluster, such as a delete by query request, a search request and so on. Elasticsearch provides a dedicated Task API for the task management which includes various actions, from retrieving the status of current running tasks to canceling any long running task.
Examples
Get all currently running tasks on all nodes of the cluster
Apart from other information, the response of the below request contains task IDs of all the tasks which can be used to get detailed information about the particular task in question.
GET _tasks
Get detailed information of a particular task
Where clQFAL_VRrmnlRyPsu_p8A:1132678759 is the ID of the task in below request
GET _tasks/clQFAL_VRrmnlRyPsu_p8A:1132678759
Get all the current tasks running on particular nodes
GET _tasks?nodes=nodeId1,nodeId2
Cancel a task
Where clQFAL_VRrmnlRyPsu_p8A:1132678759 is the ID of the task in the below request
POST /_tasks/clQFAL_VRrmnlRyPsu_p8A:1132678759/_cancel?pretty
Notes
- The Task API will be most useful when you want to investigate the spike of resource utilization in the cluster or want to cancel an operation.
Log Context
Log “failed to run scheduled task [{}] on thread pool [{}]” classname is ThreadPool.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
if (logger.isDebugEnabled()) {
logger.debug(() -> new ParameterizedMessage("scheduled task [{}] was rejected on thread pool [{}]";
command; executor); e);
}
};
(e) -> logger.warn(() -> new ParameterizedMessage("failed to run scheduled task [{}] on thread pool [{}]";
command; executor); e));
}
protected final void stopCachedTimeThread() {
cachedTimeThread.running = false;
