Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed head offset in follower after a snapshot load #415

Merged
merged 1 commit into from
Dec 28, 2023

Conversation

merlimat
Copy link
Collaborator

The head snapshot in follower was not updated after loading a snapshot. This can make it look like the follower is not caught up with the leader, when in fact it is.

Dec 27 22:43:34.180 |INF| Successfully moved ensemble to a new term component='shard-controller' followers=[{ server-address={ public='oxia-2.oxia-svc.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-sv
c:6649' } entry-id={ term=-1 offset=-1 } },{ server-address={ public='oxia-1.oxia-svc.oxia.svc.cluster.local:6648' internal='oxia-1.oxia-svc:6649' } entry-id={ term=-1 offset=-1 } }] namespace='default' n
ew-leader={ public='oxia-5.oxia-svc.oxia.svc.cluster.local:6648' internal='oxia-5.oxia-svc:6649' } shard=1 term=19
Dec 27 22:43:34.229 |INF| Successfully deleted shard component='shard-controller' namespace='default' server={ public='oxia-4.oxia-svc.oxia.svc.cluster.local:6648' internal='oxia-4.oxia-svc:6649' } shard=
1
Dec 27 22:43:34.233 |INF| Elected new leader component='shard-controller' leader={ public='oxia-5.oxia-svc.oxia.svc.cluster.local:6648' internal='oxia-5.oxia-svc:6649' } namespace='default' shard=1 term=1
9
Dec 27 22:43:34.241 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:34.324 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:34.406 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:34.672 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:34.922 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:35.621 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1
Dec 27 22:43:36.113 |INF| Follower is *not* caught-up yet with the leader component='shard-controller' follower-head-offset=-1 leader-head-offset=27556 namespace='default' server={ public='oxia-2.oxia-svc
.oxia.svc.cluster.local:6648' internal='oxia-2.oxia-svc:6649' } shard=1

The node had indeed correctly applied the snapshot:

Dec 27 22:38:34.193 |INF| Received SendSnapshot request component='internal-rpc-server' namespace='default' peer='10.244.0.130:50430' shard=1
Dec 27 22:38:34.193 |INF| Closing leader controller component='leader-controller' namespace='default' shard=1 term=18
Dec 27 22:38:34.194 |INF| [JOB 1] WAL file /data/db/default/shard-1/000002.log with log number 000002 stopped reading at offset: 0; replayed 0 keys in 0 batches component='pebble' shard=1
Dec 27 22:38:34.196 |INF| Created follower commit-offset=-1 component='follower-controller' head-offset=-1 namespace='default' shard=1 term=18
Dec 27 22:38:34.208 |INF| Successfully applied snapshot commit-offset=27556 component='follower-controller' namespace='default' shard=1 snapshot-size=449137 term=18
Dec 27 22:38:34.220 |INF| Received Replicate request component='internal-rpc-server' namespace='default' peer='10.244.0.130:50430' shard=1

@merlimat merlimat merged commit 174b459 into streamnative:main Dec 28, 2023
4 checks passed
@merlimat merlimat deleted the fix-last-entry-after-snapshot branch December 28, 2023 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant