Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: merger should process late arriving one-block files (to copy to forked blocks) #81

Open
matthewdarwin opened this issue Feb 4, 2025 · 0 comments

Comments

@matthewdarwin
Copy link

Copy from: streamingfast/merger#31

In the case where a one-block arrives late (late defined as after merger has already created a merged-block), the merger should evaluate the one-block as to if the one-block is already included in the merged-blocks or if it is a "forked block" and should be written to forked blocks.

When this happens: readers might be sending blocks to live firehose users but those blocks are delayed being send to the merger due to network issues. When you run different firehose components in different parts of the planet, weird things can happen.

If the one-block file is older than the "retain forked blocks" time period, then the one-block should simply be discarded (as today).

If not, read the one-block, read the merged-block, then take the decision on what to do (throw away or write to forked blocks)

@matthewdarwin matthewdarwin changed the title Bug: merger should process late arriving one-block files Bug: merger should process late arriving one-block files (to copy to forked blocks) Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant