Spreading KVs for half of the dead node grace period. #114
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fulmicoton
force-pushed
the
transmitting-dead-node-kvs
branch
4 times, most recently
from
db503f9 to
488820c
Compare
fulmicoton
commented
Feb 13, 2024
fulmicoton
force-pushed
the
transmitting-dead-node-kvs
branch
from
488820c to
b2d0765
Compare
fulmicoton
changed the title
fulmicoton
force-pushed
the
transmitting-dead-node-kvs
branch
3 times, most recently
from
166dd41 to
c349456
Compare
guilload
approved these changes
Feb 14, 2024
There was a problem hiding this comment.
I find that the code remains difficult to read despite your explanations in the README.
I would go with the following naming/explanations.
Node deletion
- Heartbeats are fed into a phi-accrual detector.
- Detector tells
livenodes fromfailednodes apart. - Failed nodes are GCed after GC_GRACE_PERIOD.
Reliable broadcast
- In order to ensure reliable broadcast, we must propagate info about
failednodes for some time shorter thanGC_GRACE_PERIODbefore deleting them. - To do so,
failednodes are split into two categories:zombieanddead. - First, upon failure,
failednodes becomezombienodes, and we keep sharing data about them. - After
ZOMBIE_GRACE_PERIOD,zombienodes transition todeadnodes, and we stop sharing data about them. - ZOMBIE_GRACE_PERIOD is set to GC_GRACE_PERIOD / 2
|
|
||
| The chitchat library does not include any mechanism to prevent this from happening. They should however eventually get deleted (after a bit more than `DEAD_NODE_GRACE_PERIOD`) if the node is really dead. | ||
|
|
||
| If the node is alive, it should be able to fix everyone's state via reset or regular delta. |
There was a problem hiding this comment.
At this point, chitchat is so specific to Quickwit, I'd rather move it to quickwit-gossip inside quickwit. Updates will be easier. Same for mrecorlog.
There was a problem hiding this comment.
I think it is fine outside of quickwit: we don't modify it much and I like that it is really uncoupled.
Also, it is surprisingly used by a couple of external projects.
In the future I'd love to improve it, and make it into a protocol to replicate pure operation based Crdts.
Quickwit would get cleaner.
fulmicoton
force-pushed
the
transmitting-dead-node-kvs
branch
from
633fb6a to
9582399
Compare
See README for more information. This change is made because we use chitchat as a reliable broadcast to update published position in Quickwit.
fulmicoton
force-pushed
the
transmitting-dead-node-kvs
branch
from
9582399 to
73fd3e2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See README for more information.
This change is made because we use chitchat as a reliable broadcast to
update published position in Quickwit.