You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a three node etcd cluster setup (version 3.5.1) in our project.
We recently ran a failover test where one of the nodes (not the cluster leader) was stopped.
Shortly after a weird sequence of events unfolded:
At 11:43:40:772, node 84d31e249fe9a16f reported that it has lost the leader 64e76369dd53b2d2:
{"level":"info","ts":"2023-07-26T11:43:40.772Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 84d31e249fe9a16f lost leader 64e76369dd53b2d2 at term 32"}
Shortly after, at 11:43:41.079 node 64e76369dd53b2d2(the leader) reported it has lost the leader, itself:
{"level":"info","ts":"2023-07-26T11:43:41.079Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 64e76369dd53b2d2 lost leader 64e76369dd53b2d2 at term 32"}
In the next 13 seconds, both nodes try to become pre-candidates and send each other the preVote requests but none seem to receive the pre-vote request from the other...until 13 seconds later when they all seem to come all at once:
Example:
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e24
9fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Finally, at 11:45:53.289 re-election succeeded:
Jul 26, 2023 @ 11:45:53.289 srv-ets-02.ets.xvp.frequentis.frq {"level":"info","ts":"2023-07-26T11:45:53.289Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 84d31e249fe9a16f elected leader 64e76369dd53b2d2 at term 33"}
Questions are:
Why did the cluster leader report that it has lost connection to itself?
Why did the election take so long? Does this look like symptoms of some network/cpu bottleneck that we might have?
If yes, do you have some recommendations on how to prove/monitor such resource starvation?
Important note: We have the etcd cluster running in the real time slice, so I tend to think this is not a CPU starvation issue.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello,
We have a three node etcd cluster setup (version 3.5.1) in our project.
We recently ran a failover test where one of the nodes (not the cluster leader) was stopped.
Shortly after a weird sequence of events unfolded:
At 11:43:40:772, node 84d31e249fe9a16f reported that it has lost the leader 64e76369dd53b2d2:
{"level":"info","ts":"2023-07-26T11:43:40.772Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 84d31e249fe9a16f lost leader 64e76369dd53b2d2 at term 32"}
Shortly after, at 11:43:41.079 node 64e76369dd53b2d2(the leader) reported it has lost the leader, itself:
{"level":"info","ts":"2023-07-26T11:43:41.079Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 64e76369dd53b2d2 lost leader 64e76369dd53b2d2 at term 32"}
In the next 13 seconds, both nodes try to become pre-candidates and send each other the preVote requests but none seem to receive the pre-vote request from the other...until 13 seconds later when they all seem to come all at once:
Example:
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.626 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e24
9fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.627Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Jul 26, 2023 @ 11:43:53.627 {"level":"info","ts":"2023-07-26T11:43:53.626Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"64e76369dd53b2d2 [logterm: 32, index: 359485262, vote: 64e76369dd53b2d2] rejected MsgPreVote from 84d31e249fe9a16f [logterm: 32, index: 359485189] at term 32"} etcd-core-1
Finally, at 11:45:53.289 re-election succeeded:
Jul 26, 2023 @ 11:45:53.289 srv-ets-02.ets.xvp.frequentis.frq {"level":"info","ts":"2023-07-26T11:45:53.289Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 84d31e249fe9a16f elected leader 64e76369dd53b2d2 at term 33"}
Questions are:
Important note: We have the etcd cluster running in the real time slice, so I tend to think this is not a CPU starvation issue.
Thank you in advance,
Radu.
PS: attached are the raft logs from the two etcd nodes:
etcd-core-3.log
etcd-core-1.log
Beta Was this translation helpful? Give feedback.
All reactions