Unable to retrieve node FS stats for – How to solve this Elasticsearch error

Briefly, this error occurs when Elasticsearch cannot access the file system (FS) statistics for a specific node. This could be due to insufficient permissions, a non-responsive node, or a network connectivity issue. To resolve this, you can check the node’s status and network connectivity. If the node is down, restart it. If it’s a permissions issue, adjust the permissions to allow Elasticsearch access. If it’s a network issue, check your network settings and connections. Also, ensure that the Elasticsearch process has the necessary system-level permissions to access FS stats.

When you have connectivity issues between Kibana & Elasticsearch you may get this log: unable to retrieve version information from Elasticsearch nodes, and in this case we recommend you try running the Elasticsearch Error Check-Up. The tool can help you pinpoint the source for connectivy issues and offer personalized recommendations on how to improve your ELK performance.

Overview

This is actually a Kibana error log message. Kibana, as well as other Elastic products (Beats, APM server…), interacts with an Elasticsearch cluster playing the role of a client application. This means you have to configure Kibana to access the Elasticsearch service as you normally would for any other client application, by setting up: host addresses for the service, ports, credentials and more. If your cluster is configured to encrypt communications with its clients, you’ll probably want to set some TLS related properties in Kibana’s configuration.

All of those properties can be set in Kibana’s configuration file, kibana.yml, which is typically located at the $KIBANA_HOME/config path. The location of this file can be easily changed by setting the KBN_PATH_CONF environment variable, as such:

KBN_PATH_CONF=/home/kibana/config ./bin/kibana

When it comes to connectivity with the cluster, the following properties are relevant:

• elasticsearch.hosts: The URLs of the Elasticsearch instances to use for all your queries. All nodes listed here must be on the same cluster. Default: [ “http://localhost:9200” ]. To enable SSL/TLS for outbound connections to Elasticsearch, use the https protocol in this setting.
• elasticsearch.username and elasticsearch.password: If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Kibana server uses to perform maintenance on the Kibana index at startup. Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.
• elasticsearch.ssl.*: several configurations related to connection encryption.

In its bootstrap process, Kibana will try to connect to all Elasticsearch endpoints configured in elasticsearch.hosts in order to check their versions. This is important because Kibana obeys a strict compatibility matrix, so it can guarantee that there will be no API request errors.

What it means

Typically, you’ll see the “Unable to version information Elasticsearch nodes” message when Kibana can’t connect to one of the hosts included in elasticsearch.hosts for whatever reason. This can be a little misleading, because not only could it not retrieve the specific information (node version), but it actually couldn’t establish a connection at all with the node.

Possible causes

Here are some reasons why this might occur:

• If not all the endpoints included in elasticsearch.hosts are reachable. Check for connectivity and misspellings.
• If the KBN_PATH_CONF environment variable is set and pointing to a different config file.
• If there is a firewall between the Kibana instance and the Elasticsearch cluster preventing the connection from being established.

How to resolve it

To solve this issue just make sure your can establish a connection with all endpoints included in the elasticsearch.hosts configuration parameter, like this:

curl http://es01:9200/

Or in this way, if your cluster is available through https:

# Insecure
curl -u elastic -k https://es01:9200/
 
# Secure
curl -u elastic --cacert ~/certs/ca/ca.crt https://es01:9200/

You should receive a response that looks like this:

{
  "name" : "node01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "fxP-R0FTRcmTl_AWs7-DiA",
  "version" : {
    "number" : "7.13.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "5d21bea28db1e89ecc1f66311ebdec9dc3aa7d64",
    "build_date" : "2021-07-02T12:06:10.804015202Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Log context

For those seeking for more of an in-depth context, you can look at the exact point of Kibana’s source code where this log message is emitted:

export function mapNodesVersionCompatibility(
  nodesInfoResponse: NodesInfo & { nodesInfoRequestError?: Error },
  kibanaVersion: string,
  ignoreVersionMismatch: boolean
): NodesVersionCompatibility {
  if (Object.keys(nodesInfoResponse.nodes ?? {}).length === 0) {
    // Note: If the a nodesInfoRequestError is present, the message contains the nodesInfoRequestError.message as a suffix
    let message = `Unable to retrieve version information from Elasticsearch nodes.`;
    if (nodesInfoResponse.nodesInfoRequestError) {
      message = message + ` ${nodesInfoResponse.nodesInfoRequestError.message}`;
    }
    return {
      isCompatible: false,
      message,
      incompatibleNodes: [],
      warningNodes: [],
      kibanaVersion,
      nodesInfoRequestError: nodesInfoResponse.nodesInfoRequestError,
    };
  }
...

Here you can see that Kibana will test every endpoint in elasticsearch.hosts:

 const incompatibleNodes = nodes.filter(
    (node) => !esVersionCompatibleWithKibana(node.version, kibanaVersion)
  );

You might also be interested in the code that tests Kibana release version against Elasticsearch’s node release version:

export function esVersionCompatibleWithKibana(esVersion: string, kibanaVersion: string) {
  const esVersionNumbers = {
    major: semver.major(esVersion),
    minor: semver.minor(esVersion),
    patch: semver.patch(esVersion),
  };
 
  const kibanaVersionNumbers = {
    major: semver.major(kibanaVersion),
    minor: semver.minor(kibanaVersion),
    patch: semver.patch(kibanaVersion),
  };
 
  // Reject mismatching major version numbers.
  if (esVersionNumbers.major !== kibanaVersionNumbers.major) {
    return false;
  }
 
  // Reject older minor versions of ES.
  if (esVersionNumbers.minor < kibanaVersionNumbers.minor) {
    return false;
  }
 
  return true;
}