Patroni connect etcd with TLS

[EdithChenLi]

What happened?

we are trying to integrate Patroni with ETCD, but found patroni can not connect with etcd using TLS with CN name specificly, the ETCD enabled client-cert-auth and enabled authentication. Patroni export told me it's becuase the gRPC-gateway connection.

The patroni log:
DEBUG: New etcd client created for htps://127.0.0.1:4001
DEBUG: Starting new HTTPS connection(1): xxxxx:2379
DEBUG: https:xxxx:2379 "GET /version HTTP/1.1" 200 45
..........
DEBUG:https:xxxx:2379 "POST /v3alpha/kv/rang HTTP/1.1" 403 50
CRITICAL: Username or password not set, authentication is not possible
ERROR: KVCache.run <PermissionDEnied errro: 'etcdserver: permission denied, code: 7>

What did you expect to happen?

But we do not want use u/p in the patroni configuraiton file, so I am thinking to give up the etcd RBAC. I did some CTL and API call test, seems everyone use any client certification assigned by same root CA can read/audit/delete to change the ETCD keys. is this expected logical? if etcd auth is not enabled, everyone by default can control the ETCD by API/CTL

How can we reproduce it (as minimally and precisely as possible)?

ETCD created with --client-auth-auth=true
Enable the etcd basic auth (patroni_usr user with root role), patroni_usr also is patroni service owner
start patroni with client ceritification for user patroni_usr

Anything else we need to know?

etcd3:
hosts:xx:2379,xx:2379, xx:2379
protocol:https
cacert: $PATH/ca.pem --(one root CA)
cert: $PATH/patroni_usr.pem
key: $PATH/patroni_usr.key

Etcd version (please run commands below)

3.2 etcdserver
3.2/3.5 etcdctl (we have 2 versions to do testing)

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

The debug info for API calls using different certificates

D | auth: found common name patroni_usr
D | etcdserver/api/v3rpc: start time =xxxxx, ......., request type= /etcdserverpb.KV/Range, request content =key: "/" range_end: "0" keys_only:true
D | auth: found common name 10.112.132.117
D | etcdserver/api/v3rpc: start time =xxxxx, ......., request type= /etcdserverpb.KV/DeleteRange, request content =key: "/service/test/sync"
D | auth: found common name 10.112.132.117
D | etcdserver/api/v3rpc: start time =xxxxx, ......., request type= /etcdserverpb.KV/Range, request content =key: "/" range_end: "0" keys_only:true
.................

Relevant log output

No response

[jmhbnz]

Hi @EdithChenLi - Thanks for your question, in future please raise as a github discussion, refer: https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md#support-requests

I believe this section of documentation is relevant in your case: https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#using-tls-common-name

As of version v3.2 if an etcd server is launched with the option --client-cert-auth=true, the field of Common Name (CN) in the client’s TLS cert will be used as an etcd user. In this case, the common name authenticates the user and the client does not need a password. Note that if both of 1. --client-cert-auth=true is passed and CN is provided by the client, and 2. username and password are provided by the client, the username and password based authentication is prioritized.

Note that this feature cannot be used with gRPC-proxy and gRPC-gateway. This is because gRPC-proxy terminates TLS from its client so all the clients share a cert of the proxy. gRPC-gateway uses a TLS connection internally for transforming HTTP request to gRPC request so it shares the same limitation. Therefore the clients cannot provide their CN to the server correctly. gRPC-proxy will cause an error and stop if a given cert has non empty CN. gRPC-proxy returns an error which indicates that the client has an non empty CN in its cert.

[EdithChenLi]

Thanks for the support link and details. James
you mentioned the username and password based authentication is prioritized. If we create one custom user(Ex. patroni_usr) with root role and enable the basic auth in ETCD side. could we change password periodically? when the password changed, at that moment, all client connection will corrupted or survival for "a while"?

[jmhbnz]

Hi @EdithChenLi - Please refer to this documentation covering how etcd auth time-of-check/time-of-use works https://etcd.io/docs/v3.5/learning/design-auth-v3/#authentication

Additionally please refer to this documentation for password change instructions: https://etcd.io/docs/v3.5/op-guide/authentication/rbac/#working-with-users

[EdithChenLi]

Thanks James
We enabled the etcd auth and create patroni_usr and root users with root role, and update the patroni configuration file, but got bad certificate warnings in patroni logs.
is this may related the "--client-cert-auth=true" configuration(we used client TLS connection but not enable the auth before)? As my understanding 2 methods can manage etcd after enable etcd auth

1. Use --client-cert-auth=true + client certificate +CN name is one of etcd user with no password and have root role
2. Username with password and have root role

$ etcdctl user list
patroni_usr
root

But we got certification error when trying to use username/password as howing below, did I misunderstand something?

etcd3:
hosts:xx:2379,xx:2379, xx:2379
protocol:https
username: patroni_usr
password: 1234

The patroni log:
DEBUG: New etcd client created for htps://127.0.0.1:4001
DEBUG: Starting new HTTPS connection(1): xxxxx:2379
DEBUG: Failed to get list of machines from https://xxxxx:2379/v2: MaxRetryError("HTTPSConnectionPool (host='xxxx', port=2379)........ sslv3 alert bad certificate (xxx")
..........

$ /home/chen/etcd35/etcdctl user list --endpoints=$ENDPOINTS --user patroni_usr:1234
{"level": "warn",'ts": "xxx", "logger": "etcd-client", "caller": "[email protected]/retry_interceptoer.go:62", "msg":"retrying of unary invoker failed", "target":"etcd-endpoints"://xxxx/:2379", "error": "rpc error: code=DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}
Error: contenxt deadline exceeded

[jmhbnz]

Please provide the configuration file or full list of flags you are using to configure etcd. You can redact hostnames if necessary I just want to understand what flags you are passing in relation to tls in context with other etcd configuration options.

Edit: This docs page may be of assistance in relation to general purpose tls configuration: https://etcd.io/docs/v3.5/op-guide/security/

[EdithChenLi]

Yes, please see below env file for etcd. The etcd.crt CN included all 3 ETCD nodes IP/hostnames

ETCD_NAME=etcd_1
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://1.1.1.1:2379
ETCD_LISTEN_CLIENT_URLS=https://1.1.1.1:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://1.1.1.1:2380
ETCD_LISTEN_PEER_URLS=https://1.1.1.1:2380
ETCD_INITIAL_CLUSTER=etcd_1=https://1.1.1.1:2380,etcd_2=https://1.1.1.2:2380,etcd_3=https://1.1.1.3:2380
ETCD_INITIAL_CLUSTER_TOKEN=etcd_token
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_AUTO_COMPACTION_RETENTION=1

ETCD_CERT_FILE=/etc/etcd.crt
ETCD_KEY_FILE=/etc/etcd.key
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/ca.pem
ETCD_AUTO_TLS=false

ETCD_PEER_CERT_FILE=/etc/etcd.crt
ETCD_PEER_KEY_FILE=/etc/etcd.key
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_TRUSTED_CA_FILE=/etc/ca.pem
ETCD_PEER_AUTO_TLS=false

The patroni original configuraiton before enable etcd auth:
etcd3:
hosts:xx:2379,xx:2379, xx:2379
protocol:https
cacert: /etc/ca.pem
cert: /etc/etcd.crt
key: /etc/etcd.key

To connect etcd using etcdctl before:
$ etcdctl user list --endpoints=$ENDPOINTS --cert=/tmp/etcd.crt --key=/tmp/etcd.key

[karfrank]

@EdithChenLi I ran into the same error when I set up a Patroni cluster with client cert authentication today and found this issue when I looked for a solution.

The error is triggered by the following code:

if len(chains[0].Subject.CommonName) != 0 {
      http.Error(rw, "CommonName of client sending a request against gateway will be ignored and not used as expected", http.StatusBadRequest)
      return
}

https://github.com/etcd-io/etcd/blob/0073fd422528e7cb56fdf8b8a61d8751e3a281db/server/embed/serve.go#L419-L422C4

Clearly, etcd doesn't like the CommonName in the client certificate, so I issued one without a CN and everything is now working fine for me.

Hope this helps!

[EdithChenLi]

Thanks @karfrank I did not realised the client certificate can generate without a CN with PKI, if you don't mind, can you pls share the certificate configuraiton file? And I think I got the reason why I got the permission error using etcd auth. There is one line warning in patroni log missed(not sure why did not see similar warning/error in etcd log, maybe because the log level)

Patroni log:
WARNING: Detected Etcd Version 3.2.0 is lower than 3.3.0, authentication is not supported
DEBUG: https://xxxx:2379 "POST /v3alpha/cluster/memeber/list HTTP/1.1" 200 520
.............
DEBUG: https://xxxx:2379" POST /v3palpha/kv/range HTTP/1.1" 403 50
ERROR: KVCache.run <PermissionDenied error: 'etcdserver: permission denied>

I downloaded version 3.5 and do same configuraiton then got same error you mentioned for CN:
ERROR: Failed to get list of machines from https://xxx/v3: <Unknow erro: 'CommonName of client sending a request against gateway will be ignored and not used as expected', code:2>

So before 3.3, ETCD authenticate is not supported for real? because I can see the etcd authenticate page for 3.2 version(for some reason, I need use version 3.2)
https://etcd.io/docs/v3.2/op-guide/authentication/

[karfrank]

@EdithChenLi I used the following minimalistic cfssl configuration file to generate the client certificates for Patroni:

{
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "O": "patroni",
      "OU": "postgres",
      "L": "production"
    }
  ]
}

[EdithChenLi]

Thanks for the sharing @karfrank

@karfrank @jmhbnz Regarding the warning message in patroni when use username/password to connect ETCD. if possible you know where/who we can confirm if 3.2.0 support etcd auth or not? I am checking with Patroni Eng as well

Patroni log:
WARNING: Detected Etcd Version 3.2.0 is lower than 3.3.0, authentication is not supported
DEBUG: https://xxxx:2379/ "POST /v3alpha/cluster/memeber/list HTTP/1.1" 200 520
.............
DEBUG: https://xxxx:2379/" POST /v3palpha/kv/range HTTP/1.1" 403 50
ERROR: KVCache.run <PermissionDenied error: 'etcdserver: permission denied>

The patroni code which trigger the WARNING message for reference

https://github.com/zalando/patroni/blob/dc9ff4cb8ae974ae5cc5b2521a12a53143936c2c/patroni/dcs/etcd3.py#L222-L247

[jmhbnz]

@karfrank - Thanks for your insights.

Hey @EdithChenLi - Etcd 3.2 certainly supported authentication, auth was introduced in etcd 2.1, refer https://etcd.io/docs/v3.2/op-guide/authentication/

However the python-etcd client used by Patroni might be the issue judging by this commit message: patroni/patroni@3341c89

Following the breadcrumbs it appears Patroni is using an etcd python client that has not been updated for a long time, refer requirements.txt, https://github.com/zalando/patroni/blob/master/requirements.txt#L5. I might be wrong but going off pip that appears to point to https://github.com/jplana/python-etcd.

I've moved this into a discussion as I don't think we have any bug on the etcd side however this might be a case for a known issue or faq entry once I have a chance to read back through this thread properly.

[EdithChenLi]

Much appreciated @karfrank @jmhbnz
ll try to use higher version to enable ETCD Auth or setup allow list