Quick links
- Introduction
- Updating certificates with the same CA
- Updating certificates with a different CA
- Notes and good things to know
Introduction
This article explains the various ways in which you can update your Elasticsearch security certificates. The method that you select depends on whether you are updating certificates with the same or different certificate authority (CA). It also depends on whether you are using p12 or PEM and key. Each of these different methods is detailed below.
Updating certificates with the same CA
If you have access to the CA used to sign your existing certificates, you only need to change the certificates and keys for each node in your cluster. If you change your existing certificates and keys on each node and use the same filenames, Elasticsearch will reload the files and start using the new certificates and keys.
You don’t need to reboot every node, but doing so will force new transport layer security (TLS) connections and is a recommended practice when updating certificates. Therefore, the following steps involve restarting a node after updating each certificate.
The steps below also provide instructions for generating new node certificates and keys for both the TLS and the HTTP layer. Depending on which of your certificates are expiring, you may only need to replace one or the other layer.
Updating certificates with the same CA (p12)
To use this method, you need to have an admin p12 certificate.
The first step is to create a new certificate using your CA certificate:
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ca-pass "" --days 365 --out "new_cert.p12" --pass ""
Then, paste the new certificate in place of the old certificate on all nodes:
cp new_cert.p12 /etc/elasticsearch/config/elastic-certificates.p12
That completes the process. All you need to do now is check the response from Elastic logs:
[][INFO ][o.e.x.c.s.SSLConfigurationReloader] [] reloaded [es01/config/elastic-certificates.p12]
And updated ssl contexts using this file:
#Example elasticsearch.yml for .p12 file cluster.name: certificate node.name: es01 cluster.initial_master_nodes: ["es01", "es02"] xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.verification_mode: certificate xpack.security.http.ssl.keystore.path: elastic-certificates.p12 xpack.security.http.ssl.truststore.path: elastic-certificates.p12
Updating certificates with the same CA (PEM and key)
To use this method, you need to have an admin certificate and key.
The first step is to create a new .crt certificate and .key private key using the CA certificate and private key:
./bin/elasticsearch-certutil cert --ca-cert ca/ca.crt --ca-key ca/ca.key --pem --name new_cert --out new_cert.zip
Next, unzip and paste the new certificate in place of the old certificate on all nodes:
unzip new_cert.zip cp new_cert/new_cert.crt /etc/elasticsearch/config/client.crt cp new_cert/new_cert.key /etc/elasticsearch/config/client.key
Example elasticsearch.yml for .crt and .key file
cluster.name: certificate node.name: es01 cluster.initial_master_nodes: ["es01", "es02"] xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.key: client.key xpack.security.transport.ssl.certificate: client.crt xpack.security.transport.ssl.certificate_authorities: ca.crt xpack.security.http.ssl.enabled: true xpack.security.http.ssl.verification_mode: certificate xpack.security.http.ssl.key: client.key xpack.security.http.ssl.certificate: client.crt xpack.security.http.ssl.certificate_authorities: ca.crt
It is important to note that after updating the certificate(s) you don’t need to restart nodes. Elasticsearch checks the certificates every five seconds.
If you have indexers like Logstash, Filebeat, or any other client application, you should update that certificate for external communication using the xpack.security.http.ssl* settings.
Updating certificates with a different CA
If you don’t have the root (admin) certificate p12 or PEM and the associated private key, you can’t create a client certificate. But you can create a new root CA certificate and then a new client certificate.
Updating certificates with a different CA (p12)
The first step is to create a p12 certificate (which includes a key and PEM file in it).
The following command will generate elastic-stack-ca.p12, which is the new p12 root CA certificate:
./bin/elasticsearch-certutil ca
Then, this command will generate the new client certificate using the newly generated root CA certificate:
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ca-pass "" --days 365 --out "new_cert.p12" --pass ""
When that is complete, the new certificates are ready to use. Simply paste the new certificate in place of the old certificate on all nodes.
Then, check elasticsearch.yml and ensure the certificate’s name is the same as the one in the configuration file:
cp new_cert.p12 /etc/elasticsearch/config/elastic-certificates.p12
Remember to store the new root CA certificate safely because you might need it in the future.
Updating certificates with a different CA (PEM and key)
To follow this method, the first step is to create a PEM certificate and key.
The following command will generate ca.crt and ca.key, which are the new root CA certificate and key, respectively:
./bin/elasticsearch-certutil ca -pem unzip elastic-stack-ca.zip
Next, this command will generate the new client certificate and key:
./bin/elasticsearch-certutil cert --ca-cert ca/ca.crt --ca-key ca/ca.key --pem --name new_cert --out new_cert.zip
When that is completed, the new certificates are ready to use. Paste the new certificate in place of the old certificate on all nodes.
Remember to check the elasticsearch.yml and make sure the name of the certificate is the same as the one in the configuration file:
unzip new_cert.zip cp ca/ca.crt config/ca.crt cp new_cert/new_cert.crt /etc/elasticsearch/config/client.crt cp new_cert/new_cert.key /etc/elasticsearch/config/client.key
WARNING: Do not restart any of the nodes before copying the certificates to all nodes. If you do restart a node before copying the certificate, the restarted node will not be able to join the cluster because it has a different certificate.
If you use the wrong CA certificate with the above method, you can face any of the following errors after node restart (emphasis added):
[][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [] client did not trust this server’s certificate, closing connection Netty4TcpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9300, remoteAddress=/[0:0:0:0:0:0:0:1]:, profile=default}
[][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/127.0.0.1:62282, remoteAddress=/127.0.0.1:9300, profile=default}]
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
If you have experienced the above errors, you need to check that you are using the same certificates on all your nodes.
Notes and good things to know
Here are some key considerations to bear in mind when updating your Elasticsearch security certificates:
- This article assumes that you don’t want to change passwords in elasticsearch-keystore or set a password for it.
- If you have a CA certificate that has not expired, you should follow the steps under the heading “Updating certificates with the same CA” above.
- If you don’t have a certificate, or if your certificate has expired, then you should follow the steps under the heading “Updating certificates with a different CA” above.
- The same certificate can be used for the HTTP and transport layer, or you can create two separate certificates using the same method.
- Once you create the new certificate, copy and paste it to all nodes. Do not create certificates on each node individually because it will be easier to use the same certificate for all nodes.
- If you do not change the name of the certificate, you do not need to make any changes in elasticsearch.yml.
- You do not need to restart Elasticsearch after updating the certificate. But best practice recommends a rolling restart. After updating the certificates on all nodes, restart one of the coordinator nodes to test it. Everything is working properly if the node can connect to the cluster after the restart. If that is the case, you don’t need to restart other nodes.
- After renewing all the certificates, check if the certificates on all nodes are the same. You can do this check with shasum <certificate> to get the hash of the certificates.
- If the applications that query and index Elasticsearch (like Filebeat, Logstash, and Kibana) use secure communication, you should also update their certificates.
