What it means
Slow searches might become a bottleneck and may cause a waiting queue to build.
There are a number of possible causes for slow search on particular nodes:
- Your application is not load balancing properly across all of the data nodes.
- Search and/or indexing operations are concentrated on specific nodes because of the way shards are allocated.
- The queries running on certain indices (concentrated on the nodes in question) are slow and need optimization.
- There are other processes (such as merges or snapshots) running on the nodes in question which are using resources on the slow nodes.
How to resolve
Look at the monitoring data to determine which indices are receiving most search / indexing operations. Often you will see that certain nodes have a high CPU usage, and that there are certain “hot” index shards that exist on those nodes. For example, imagine the following scenario:
- Index A has a high search rate,
- Index A has shards only on node 3 and 4
- Nodes 3 and 4 have higher cpu usage than the other data nodes.
index A is a strong candidate to be one of the causes of the problem.
Check to see whether you have any of the issues described in loaded data nodes.
Activate slow logging
If you have slow logging activated, look to see which logs are taking the most time. In particular look for aggregations with large “size” values or nested queries which are the most common culprits of slow queries.
To activate slow logging for a given index you can run the command below.
PUT /my-index/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.level": "info"
}To save resources it is best to leave slow logging deactivated when you have finished using the value of -1.
PUT /my-index/_settings
{
"index.search.slowlog.threshold.query.warn": "-1",
"index.search.slowlog.threshold.query.info": "-1",
"index.search.slowlog.level": "info"
}Once you have activated slow logging you can see the queries which are taking a long time. Look for the issues mentioned in the article 10 Important Tips to Improve Search in OpenSearch.
Further reading on slow searches
OpenSearch Slow Log Search Queries – A Complete Guide.
OpenSearch Slow Indexing in Nodes.
