Etcd member down

[pulsarrrr]

Hi everyone，
I have a k8s cluster(ver. 1.21.10) contains 3 masters(OS: Centos7) 2 workers(OS: Centos7) and 2 LB
I changed the master's os one by one to Rocky Linux 9, it goes well when I changed the first 2 masters and joined in my cluster, but after changing the 3rd master's os and join the cluster for 30mins or longer, there's something wrong with it:

1. usually the 3rd etcd pod keeps CrashLoopBackOff (sometimes random one);
2. the machine which etcd keeps CrashLoopBackOff become very slow(I type a letter it appers after 10s and ping more than 1000ms), I checked the CPU and memory usage, neither of them is high;
3. my cluster recovers after deleting the 3rd etcd member(but in my circumstance I need 3 masters).

etcd Version: 3.4.13

Any help is much appreciated!

[jmhbnz]

Hey @pulsarrrr - Thanks for your question.

Are you certain that the issue you are encountering is specific to etcd? It sounds as though you are having general operating system issues, you mention overall system performance problems.

If you are certain this is specific to etcd or that etcd is the cause please provide the logs for an impacted instance and a copy of the configuration you are specifying.

Additionally - please note that etcd v3.4.13 is three years old so the best advice we can likely give is to update as soon as possible. This would extend to your kubernetes version as well, v1.21 ended support over a year ago, refer https://endoflife.date/kubernetes.

[pulsarrrr]

@jmhbnz hello，thanks for reply.
I'm very confused because my cluster can work well when there are 2 masters, as long as the 3rd master join the cluster then it becomes unstable
I'll try updating my etcd version, I'll let you know if there is still something wrong or it could be resolved by your advice.
Thanks a lot.