TestCurrentLeader failed

[w-zr]

Can be reproduced ?

Yes

Steps:

cd components/prophet/election
CGO_ENABLED=1 go test -race -run=TestCurrentLeader

Expected behavior:

No error.

Actual behavior:

{"level":"info","ts":"2022-03-11T18:23:16.326+0800","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:30001"]}
{"level":"info","ts":"2022-03-11T18:23:16.327+0800","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:30000"]}
{"level":"info","ts":"2022-03-11T18:23:16.327+0800","caller":"embed/etcd.go:307","msg":"starting an etcd server","etcd-version":"3.5.0","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.17.8","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":false,"name":"p1","data-dir":"/tmp/prophet/test-1646994196324664300","wal-dir":"","wal-dir-dedicated":"","member-dir":"/tmp/prophet/test-1646994196324664300/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:30001"],"listen-peer-urls":["http://127.0.0.1:30001"],"advertise-client-urls":["http://127.0.0.1:30000"],"listen-client-urls":["http://127.0.0.1:30000"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"p1=http://127.0.0.1:30001","initial-cluster-state":"new","initial-cluster-token":"etcd-cluster","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2022-03-11T18:23:16.336+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/tmp/prophet/test-1646994196324664300/member/snap/db","took":"8.2396ms"}
{"level":"info","ts":"2022-03-11T18:23:16.340+0800","caller":"etcdserver/raft.go:448","msg":"starting local member","local-member-id":"885c6fd809d1cb2d","cluster-id":"19fc93efab8e466f"}
{"level":"info","ts":"2022-03-11T18:23:16.340+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d switched to configuration voters=()"}
{"level":"info","ts":"2022-03-11T18:23:16.341+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d became follower at term 0"}
{"level":"info","ts":"2022-03-11T18:23:16.341+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 885c6fd809d1cb2d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}
{"level":"info","ts":"2022-03-11T18:23:16.341+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d became follower at term 1"}
{"level":"info","ts":"2022-03-11T18:23:16.341+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d switched to configuration voters=(9825851460683942701)"}
{"level":"warn","ts":"2022-03-11T18:23:16.343+0800","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2022-03-11T18:23:16.346+0800","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2022-03-11T18:23:16.347+0800","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2022-03-11T18:23:16.349+0800","caller":"etcdserver/server.go:843","msg":"starting etcd server","local-member-id":"885c6fd809d1cb2d","local-server-version":"3.5.0","cluster-version":"to_be_decided"}
{"level":"info","ts":"2022-03-11T18:23:16.349+0800","caller":"etcdserver/server.go:728","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"885c6fd809d1cb2d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"info","ts":"2022-03-11T18:23:16.352+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d switched to configuration voters=(9825851460683942701)"}
{"level":"info","ts":"2022-03-11T18:23:16.353+0800","caller":"membership/cluster.go:393","msg":"added member","cluster-id":"19fc93efab8e466f","local-member-id":"885c6fd809d1cb2d","added-peer-id":"885c6fd809d1cb2d","added-peer-peer-urls":["http://127.0.0.1:30001"]}
{"level":"info","ts":"2022-03-11T18:23:16.358+0800","caller":"embed/etcd.go:580","msg":"serving peer traffic","address":"127.0.0.1:30001"}
{"level":"info","ts":"2022-03-11T18:23:16.358+0800","caller":"embed/etcd.go:552","msg":"cmux::serve","address":"127.0.0.1:30001"}
{"level":"info","ts":"2022-03-11T18:23:16.358+0800","caller":"embed/etcd.go:276","msg":"now serving peer/client/metrics","local-member-id":"885c6fd809d1cb2d","initial-advertise-peer-urls":["http://127.0.0.1:30001"],"listen-peer-urls":["http://127.0.0.1:30001"],"advertise-client-urls":["http://127.0.0.1:30000"],"listen-client-urls":["http://127.0.0.1:30000"],"listen-metrics-urls":[]}
{"level":"info","ts":"2022-03-11T18:23:16.543+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d is starting a new election at term 1"}
{"level":"info","ts":"2022-03-11T18:23:16.543+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d became pre-candidate at term 1"}
{"level":"info","ts":"2022-03-11T18:23:16.543+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d received MsgPreVoteResp from 885c6fd809d1cb2d at term 1"}
{"level":"info","ts":"2022-03-11T18:23:16.544+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d became candidate at term 2"}
{"level":"info","ts":"2022-03-11T18:23:16.544+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d received MsgVoteResp from 885c6fd809d1cb2d at term 2"}
{"level":"info","ts":"2022-03-11T18:23:16.544+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"885c6fd809d1cb2d became leader at term 2"}
{"level":"info","ts":"2022-03-11T18:23:16.544+0800","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 885c6fd809d1cb2d elected leader 885c6fd809d1cb2d at term 2"}
{"level":"info","ts":"2022-03-11T18:23:16.548+0800","caller":"etcdserver/server.go:2027","msg":"published local member to cluster through raft","local-member-id":"885c6fd809d1cb2d","local-member-attributes":"{Name:p1 ClientURLs:[http://127.0.0.1:30000]}","request-path":"/0/members/885c6fd809d1cb2d/attributes","cluster-id":"19fc93efab8e466f","publish-timeout":"7s"}
{"level":"info","ts":"2022-03-11T18:23:16.547+0800","caller":"etcdserver/server.go:2476","msg":"setting up initial cluster version using v2 API","cluster-version":"3.5"}
{"level":"info","ts":"2022-03-11T18:23:16.548+0800","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2022-03-11T18:23:16.550+0800","caller":"membership/cluster.go:531","msg":"set initial cluster version","cluster-id":"19fc93efab8e466f","local-member-id":"885c6fd809d1cb2d","cluster-version":"3.5"}
{"level":"info","ts":"2022-03-11T18:23:16.551+0800","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2022-03-11T18:23:16.551+0800","caller":"etcdserver/server.go:2500","msg":"cluster version is updated","cluster-version":"3.5"}
{"level":"info","ts":"2022-03-11T18:23:16.563+0800","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:30000"}
{"level":"info","ts":"2022-03-11 18:23:16.708","logger":"elector","caller":"election/election_leadship.go:168","msg":"ready to next loop","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.709","logger":"elector","caller":"election/election_leadship.go:177","msg":"ready to load current leader","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.724","logger":"elector","caller":"election/election_leadship.go:188","msg":"current leader loaded","tag":"[0/1]","purpose":"0","step":"main-loop","leader":""}
{"level":"info","ts":"2022-03-11 18:23:16.724","logger":"elector","caller":"election/election_leadship.go:221","msg":"start checkExpectLeader","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.727","logger":"elector","caller":"election/election_leadship.go:446","msg":"expect leader loaded","tag":"[0/1]","purpose":"0","step":"campaign","expect-leader":""}
{"level":"info","ts":"2022-03-11 18:23:16.727","logger":"elector","caller":"election/election_leadship.go:230","msg":"end checkExpectLeader, and start campaign","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.727","logger":"elector","caller":"election/election_leadship.go:247","msg":"start campaign","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:16.732","logger":"elector","caller":"election/election_leadship.go:258","msg":"grant lease ok","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:16.736","logger":"elector","caller":"election/election_leadship.go:277","msg":"start keep lease alive","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:16.736","logger":"elector","caller":"election/election_leadship.go:168","msg":"ready to next loop","tag":"[0/2]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.737","logger":"elector","caller":"election/election_leadship.go:177","msg":"ready to load current leader","tag":"[0/2]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.742","logger":"elector","caller":"election/election_leadship.go:188","msg":"current leader loaded","tag":"[0/2]","purpose":"0","step":"main-loop","leader":"1"}
{"level":"info","ts":"2022-03-11 18:23:16.742","logger":"elector","caller":"election/election_leadship.go:206","msg":"start to watch current leader","tag":"[0/2]","purpose":"0","step":"main-loop","leader":"1"}
{"level":"info","ts":"2022-03-11 18:23:16.748","logger":"elector","caller":"election/election_leadship.go:422","msg":"leader updated","tag":"[0/2]","purpose":"0","step":"watch-leader","value":"1"}
{"level":"info","ts":"2022-03-11 18:23:16.752","logger":"elector","caller":"election/election_leadship.go:422","msg":"leader updated","tag":"[0/2]","purpose":"0","step":"watch-leader","value":"2"}
{"level":"info","ts":"2022-03-11 18:23:16.752","logger":"elector","caller":"election/election_leadship.go:482","msg":"expect leader added","tag":"[0/1]","purpose":"0","expect-leader":"2","ttl-in-seconds":1}
{"level":"info","ts":"2022-03-11 18:23:16.755","logger":"elector","caller":"election/election_leaser.go:85","msg":"lease closed","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:16.787","logger":"elector","caller":"election/election_leadship.go:345","msg":"exit due to lease expired","tag":"[0/1]","purpose":"0","step":"keepalive"}
{"level":"info","ts":"2022-03-11 18:23:16.787","logger":"elector","caller":"election/election_leadship.go:239","msg":"end campaign","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.787","logger":"elector","caller":"election/election_leaser.go:112","msg":"lease exit due to context done","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:16.787","logger":"elector","caller":"election/election_leaser.go:147","msg":"keep lease exit due to context done","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:16.888","logger":"elector","caller":"election/election_leadship.go:168","msg":"ready to next loop","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.888","logger":"elector","caller":"election/election_leadship.go:177","msg":"ready to load current leader","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:16.895","logger":"elector","caller":"election/election_leadship.go:188","msg":"current leader loaded","tag":"[0/1]","purpose":"0","step":"main-loop","leader":"2"}
{"level":"info","ts":"2022-03-11 18:23:16.895","logger":"elector","caller":"election/election_leadship.go:206","msg":"start to watch current leader","tag":"[0/1]","purpose":"0","step":"main-loop","leader":"2"}
{"level":"info","ts":"2022-03-11 18:23:16.951","logger":"elector","caller":"election/election_leadship.go:422","msg":"leader updated","tag":"[0/1]","purpose":"0","step":"watch-leader","value":"2"}
{"level":"info","ts":"2022-03-11 18:23:18.854","logger":"elector","caller":"election/election_leadship.go:412","msg":"leader deleted","tag":"[0/1]","purpose":"0","step":"watch-leader"}
{"level":"info","ts":"2022-03-11 18:23:18.855","logger":"elector","caller":"election/election_leadship.go:412","msg":"leader deleted","tag":"[0/2]","purpose":"0","step":"watch-leader"}
{"level":"info","ts":"2022-03-11 18:23:18.856","logger":"elector","caller":"election/election_leadship.go:214","msg":"current leader out","tag":"[0/1]","purpose":"0","step":"main-loop","leader":"2"}
{"level":"info","ts":"2022-03-11 18:23:18.856","logger":"elector","caller":"election/election_leadship.go:214","msg":"current leader out","tag":"[0/2]","purpose":"0","step":"main-loop","leader":"1"}
{"level":"info","ts":"2022-03-11 18:23:18.856","logger":"elector","caller":"election/election_leadship.go:221","msg":"start checkExpectLeader","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:18.856","logger":"elector","caller":"election/election_leadship.go:221","msg":"start checkExpectLeader","tag":"[0/2]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:18.864","logger":"elector","caller":"election/election_leadship.go:446","msg":"expect leader loaded","tag":"[0/1]","purpose":"0","step":"campaign","expect-leader":""}
{"level":"info","ts":"2022-03-11 18:23:18.864","logger":"elector","caller":"election/election_leadship.go:230","msg":"end checkExpectLeader, and start campaign","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:18.864","logger":"elector","caller":"election/election_leadship.go:247","msg":"start campaign","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:18.865","logger":"elector","caller":"election/election_leadship.go:446","msg":"expect leader loaded","tag":"[0/2]","purpose":"0","step":"campaign","expect-leader":""}
{"level":"info","ts":"2022-03-11 18:23:18.865","logger":"elector","caller":"election/election_leadship.go:230","msg":"end checkExpectLeader, and start campaign","tag":"[0/2]","purpose":"0","step":"main-loop"}
{"level":"info","ts":"2022-03-11 18:23:18.865","logger":"elector","caller":"election/election_leadship.go:247","msg":"start campaign","tag":"[0/2]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:18.872","logger":"elector","caller":"election/election_leadship.go:258","msg":"grant lease ok","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:18.874","logger":"elector","caller":"election/election_leadship.go:258","msg":"grant lease ok","tag":"[0/2]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:18.886","logger":"elector","caller":"election/election_leadship.go:277","msg":"start keep lease alive","tag":"[0/1]","purpose":"0","step":"campaign"}
{"level":"info","ts":"2022-03-11 18:23:18.892","logger":"elector","caller":"election/election_leaser.go:85","msg":"lease closed","tag":"[0/2]","purpose":"0","tag":"[0/2]","purpose":"0"}
{"level":"error","ts":"2022-03-11 18:23:18.892","logger":"elector","caller":"election/election_leadship.go:232","msg":"fail to campaign leader","tag":"[0/2]","purpose":"0","step":"main-loop","error":"campaign leader failed, other server may campaign ok"}
{"level":"info","ts":"2022-03-11 18:23:19.764","logger":"elector","caller":"election/election_leadship.go:115","msg":"begin to stop","tag":"[0/2]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.764","logger":"elector","caller":"election/election_leaser.go:112","msg":"lease exit due to context done","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.764","logger":"elector","caller":"election/election_leaser.go:147","msg":"keep lease exit due to context done","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.764","logger":"elector","caller":"election/election_leadship.go:362","msg":"exit due to context done","tag":"[0/1]","purpose":"0","step":"keepalive"}
{"level":"info","ts":"2022-03-11 18:23:19.765","logger":"elector","caller":"election/election_leadship.go:239","msg":"end campaign","tag":"[0/1]","purpose":"0","step":"main-loop"}
{"level":"warn","ts":"2022-03-11T18:23:19.769+0800","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"324.3µs","request":"header:<ID:14640498119188903196 > lease_revoke:<id:4b2d7f7881077516>","response":"size:27","error":"lease not found"}
{"level":"warn","ts":"2022-03-11T18:23:19.772+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000198540/#initially=[http://127.0.0.1:30000]","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: requested lease not found"}
{"level":"info","ts":"2022-03-11 18:23:19.772","logger":"elector","caller":"election/election_leaser.go:79","msg":"fail to close lease","tag":"[0/2]","purpose":"0","tag":"[0/2]","purpose":"0","error":"etcdserver: requested lease not found"}
{"level":"info","ts":"2022-03-11 18:23:19.772","logger":"elector","caller":"election/election_leadship.go:121","msg":"begin to stop stopper","tag":"[0/2]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.773","logger":"elector","caller":"election/election_leadship.go:115","msg":"begin to stop","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.778","logger":"elector","caller":"election/election_leaser.go:85","msg":"lease closed","tag":"[0/1]","purpose":"0","tag":"[0/1]","purpose":"0"}
{"level":"info","ts":"2022-03-11 18:23:19.778","logger":"elector","caller":"election/election_leadship.go:121","msg":"begin to stop stopper","tag":"[0/1]","purpose":"0"}
--- FAIL: TestCurrentLeader (3.45s)
    election_test.go:259:
                Error Trace:    election_test.go:259
                Error:          Not equal:
                                expected: "2"
                                actual  : "1"

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -2
                                +1
                Test:           TestCurrentLeader
                Messages:       current leader failed, false
FAIL
exit status 1
FAIL    github.com/matrixorigin/matrixcube/components/new_prophet/election      3.524s

Environment:

• Version or commit-id (e.g. v0.1.0 or 8b23a93):
• Hardware parameters:
• OS type:
• Others:

Configuration file:

Additional context:

• Error message from client:
• Server log:
• Other information: