Migration from 3.4.14 with auth enable to 3.5.9

[kikijolicoeur]

Hello.
I have a five nodes etcd cluster with auth enable in 3.4.16.
I want to migrate this cluster to 3.5.9.
The etcd documentation says (https://etcd.io/docs/v3.5/upgrades/upgrade_3_5/) :

"NOTE: If your cluster enables auth, rolling upgrade from 3.4 or older version isn’t supported because 3.5 changes a format of WAL entries related to auth (#11943)"

So what is the process to migrate under these conditions ?
Should I :

1. create a new etcd cluster in 3.5.10 (with auth enable too)
2. Stop all the apps that use 3.4.16 cluster
3. use etcdctl snapshot save to backup the old cluster
4. use etcdctl snapshot restore to restore old cluster data to new 3.5.10 cluster
5. reconfigure all my apps to point to the 3.5.9 cluster
6. start all my apps

Are roles (RBAC) with passwords and grants backuped with snapshot or the snapshot concerns only etcd key space ?
How to backup roles if snapshot is only for keys space ?

What about if I disable auth on old cluster ? Can I use rolling upgrade then re-enable auth when the cluster will be in 3.5.9 ?

Thanks for your advices.

Christian

[ahrtr]

It should be fine as long as one of the followings is true,

• at least a v2 snapshot was created after you changed any user's password. [ignore this if you have never changed users' password or no password is needed]
• jwt instead of the simple token is used. FYI. https://etcd.io/docs/v3.4/op-guide/configuration/#--auth-token
• TLS common name is used so that password isn't needed. FYI. https://etcd.io/docs/v3.4/op-guide/configuration/#--auth-token

cc @mitake

[mitake]

Yeah I think it's fine, but using simple token is also acceptable because it keeps states only in memory. Thanks for answering @ahrtr , please let me know if you find issues @kikijolicoeur .

[kikijolicoeur]

Unfortunately I must use auth because for security reason, I use role with grant and the appplications that connect to etcd don't use jwt. For example one of this application is Patroni (PostgreSQL HA). It doesn't support jwt but username/password only.

[mitake]

It doesn't support jwt but username/password only.

If patroni uses username and password I think it's using jwt (not sure though). Or does this mean the use case rely on simple token? Simple token is for testing purpose so using it in prod isn't recommended.

[kikijolicoeur]

I need clarification.
Is it in case you change a password or add a user while the rolling upgrade can't be used? Is it possible to use the rolling upgrade if no changes are made to passwords or users for the duration of the rolling upgrade?

[mitake]

Is it possible to use the rolling upgrade if no changes are made to passwords or users for the duration of the rolling upgrade?

Yes that's possible, but because of the WAL format change updating auth config during rolling upgrade isn't supported.

[kikijolicoeur]

Thanks. So I will use rolling updrade process.
I think the documentation should be more precise on this subject.
Best regards

[kikijolicoeur]

I deactivated users modifications in the applications and I could use rolling upgrade process to migrate sucessfully my etcd clusters from version 3.4.16 to 3.5.9.
Thanks for your advices.
Best regards