etcdctl check datascale does not work when the etcd client uses TLS

[prabhakarp-dev]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request, support requests should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

etcdctl check datascale does not work when the etcd client uses TLS

$ /etcdctl check datascale fetch error: Get "http://168.254.5.3:2379/metrics": EOF FAIL: Could not read process_resident_memory_bytes before the put operations.

Not sure why it tries to access the /metrics endpoint in non-tls mode. We could confirm that metrics are available

$ curl -s --cert $ETCDCTL_CERT --key $ETCDCTL_KEY --cacert $ETCDCTL_CACERT https://168.254.5.3:2379/metrics|grep process_resident_memory_bytes HELP process_resident_memory_bytes Resident memory size in bytes. TYPE process_resident_memory_bytes gauge process_resident_memory_bytes 3.96034048e+08

`$ /etcdctl version

etcdctl version: 3.5.7
API version: 3.5`

What did you expect to happen?

etcdctl check datascale to use TLS mode to collect the metrics and run the checks accordingly

How can we reproduce it (as minimally and precisely as possible)?

Please bring up etcd cluster with client running in TLS mode and run "etcdctl check datascale"

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcdctl version
etcdctl version: 3.5.7
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table

/run/etcdctl member list -w table
+------------------+---------+----------+--------------------------------------------------------------------+--------------------------+------------+
|        ID        | STATUS  |   NAME   |                             PEER ADDRS                             |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+----------+--------------------------------------------------------------------+--------------------------+------------+
| 2561290d9e8ed912 | started | member-3 | https://member-3:2380 | https://168.254.5.3:2379 |      false |
| 737cc990093aacde | started | member-1 | https://member-1:2380 | https://168.254.5.4:2379 |      false |
| e96f7993fc6b27ee | started | member-2 | https://member-2:2380 | https://168.254.5.5:2379 |      false |
+------------------+---------+----------+--------------------------------------------------------------------+--------------------------+------------+

etcdctl endpoint status -w table
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 168.254.5.5:2379 | e96f7993fc6b27ee |   3.5.7 |  189 MB |      true |      false |        35 |   13669230 |           13669230 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

No response

[jmhbnz]

Hey @prabhakar-oracle - Thanks for raising this, can you please confirm what happens if you run the following which explicitly specifies an https endpoint?

Note I have used the same ip address as your report above, so please adjust if this has now changed.

etcdctl check datascale --endpoints=https://168.254.5.3:2379

[prabhakarp-dev]

@jmhbnz

[root@host-10-64-223-33 ~]# export|grep ETCDCTL
declare -x ETCDCTL_API="3"
declare -x ETCDCTL_CACERT="/etc/kubernetes/secrets/etcd-client-ca.crt"
declare -x ETCDCTL_CERT="/etc/kubernetes/secrets/etcd-client.crt"
declare -x ETCDCTL_KEY="/etc/kubernetes/secrets/etcd-client.key"

[root@host-10-64-223-33 ~]# /run/etcdctl check datascale --endpoints=https://168.254.5.3:2379
fetch error: Get "https://168.254.5.3:2379/metrics": x509: certificate signed by unknown authority
FAIL: Could not read process_resident_memory_bytes before the put operations.

[jmhbnz]

[root@host-10-64-223-33 ~]# /run/etcdctl check datascale --endpoints=https://168.254.5.3:2379 fetch error: Get "https://168.254.5.3:2379/metrics": x509: certificate signed by unknown authority FAIL: Could not read process_resident_memory_bytes before the put operations.

Thanks for the update @prabhakar-oracle. That error looks like a certificate issue in relation to the ca cert. Can you please confirm those certificates are valid? Perhaps use something like curl so we can rule out certificates issues in general.