Briefly, this error occurs when Elasticsearch encounters an issue while trying to mark a primary shard as failed. This could be due to network issues, disk space problems, or a bug in the Elasticsearch software. To resolve this issue, you can try to restart the Elasticsearch node, check for network connectivity issues, ensure there is enough disk space, or upgrade Elasticsearch to the latest version to fix potential bugs.
This guide will help you check for common problems that cause the log ” unexpected failure while failing primary [{}] ” to appear. To understand the issues related to this log, read the explanation below about the following Elasticsearch concepts: replication.
Overview
Replication refers to storing a redundant copy of the data. Starting from version 7.x, Elasticsearch creates one primary shard with a replication factor set to 1. Replicas never get assigned to the same node on which primary shards are assigned, which means you should have at least two nodes in the cluster to assign the replicas. If a primary shard goes down, the replica automatically acts as a primary shard.
What it is used for
Replicas are used to provide high availability and failover. A higher number of replicas is also helpful for faster searches.
Examples
Update replica count
PUT /api-logs/_settings?pretty
{
"index" : {
"number_of_replicas" : 2
}
}Common problems
- By default, new replicas are not assigned to nodes with more than 85% disk usage. Instead, Elasticsearch throws a warning.
- Creating too many replicas may cause a problem if there are not enough resources available in the cluster.
Log Context
Log “unexpected failure while failing primary [{}]” classname is ReplicationOperation.java.
We extracted the following from Elasticsearch source code for those seeking an in-depth context :
@Override
public void onFailure(Exception e) {
e.addSuppressed(failure);
assert false : e;
logger.error(new ParameterizedMessage("unexpected failure while failing primary [{}]"; primary.routingEntry()); e);
finishAsFailed(
new RetryOnPrimaryException(
primary.routingEntry().shardId();
String.format(Locale.ROOT; "unexpected failure while failing primary [%s]"; primary.routingEntry());
e
