When generating a new certificate for the transport layer, receive fatal exception while booting Elasticsearch

[juparker37]

Elasticsearch Version

8.17

Installed Plugins

None

Java Version

bundled

OS Version

Linux ipr-ost-netflow 4.18.0-553.34.1.el8_10.x86_64

Problem Description

When generating a new certificate for the transport layer, receive fatal exception while booting Elasticsearch

Following the setup guide at https://www.elastic.co/guide/en/elastic-stack/8.17/install-stack-demo-secure.html#install-stack-demo-secure-transport

and completing all steps, then starting elasticsearch, elasticsearch fails to boot.

Steps to Reproduce

Generated password three times following https://www.elastic.co/guide/en/elastic-stack/8.17/install-stack-demo-secure.html#install-stack-demo-secure-transport

No issues when reading the file after decryption:

/usr/share/elasticsearch/jdk/bin/keytool -keystore elastic-stack-ca.p12 -list
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

new-ca, Jan 19, 2025, trustedCertEntry,
Certificate fingerprint (SHA-256): 40:EA:AD:A9:FB:ED:C1:16:95:CA:85:B6:45:4D:0C:CD:39:FD:0B:39:3B:E3:08:0A:82:C1:43:94:E9:13:3E:F8

Logs (if relevant)

[2025-01-19T02:06:06,371][ERROR][o.e.b.Elasticsearch ] [ipr-ost-netflow-node-1] fatal exception while booting Elasticsearch org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/elastic-stack-ca.p12] - this is usually caused by an incorrect password; (a keystore password was provided) at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:620) ~[?:?] at java.util.HashMap.forEach(HashMap.java:1430) ~[?:?] at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1708) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:616) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:160) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:496) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:325) ~[?:?] at org.elasticsearch.node.NodeConstruction.lambda$construct$16(NodeConstruction.java:894) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:254) ~[elasticsearch-8.17.0.jar:?] at java.util.stream.ReferencePipeline$7$1FlatMap.accept(ReferencePipeline.java:289) ~[?:?] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215) ~[?:?] at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:636) ~[?:?] at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:656) ~[?:?] at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:662) ~[?:?] at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:667) ~[?:?] at org.elasticsearch.node.NodeConstruction.construct(NodeConstruction.java:916) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:291) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.node.Node.<init>(Node.java:200) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:247) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:247) ~[elasticsearch-8.17.0.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:76) ~[elasticsearch-8.17.0.jar:?] Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/certs/elastic-stack-ca.p12] - this is usually caused by an incorrect password; (a keystore password was provided) at org.elasticsearch.common.ssl.SslFileUtil.ioException(SslFileUtil.java:57) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:99) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:83) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:479) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1229) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:618) ~[?:?] ... 24 more Caused by: java.io.IOException: keystore password was incorrect at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2112) ~[?:?] at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?] at java.security.KeyStore.load(KeyStore.java:1499) ~[?:?] at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:73) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:95) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:83) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:479) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1229) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:618) ~[?:?] ... 24 more Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2112) ~[?:?] at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?] at java.security.KeyStore.load(KeyStore.java:1499) ~[?:?] at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:73) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:95) ~[?:?] at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:83) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:479) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1229) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:618) ~[?:?] ... 24 more

[juparker37]

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 19-01-2025 02:02:59
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
  truststore.path: /etc/elasticsearch/certs/elastic-stack-ca.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ipr-ost-netflow"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

[juparker37]

/usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
Returns correct decrypted password

/usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password
Returns correct decrypted password

/usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.truststore.secure_password

Returns correct decrypted password