Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Inconsistent DHT #44

Merged
merged 2 commits into from
Jan 23, 2025
Merged

Fix Inconsistent DHT #44

merged 2 commits into from
Jan 23, 2025

Conversation

rishiad
Copy link
Contributor

@rishiad rishiad commented Jan 23, 2025

This PR addresses the issue where larger files took an extended time to have all their pieces inserted into the Distributed Hash Table (DHT). The solution involves the following changes:

  1. DHT Insertion Timing: The DHT insertion process has been moved to the piece processing stage, which improves efficiency.
  2. Tagging Process: Tagging is now handled by the worker threads. This change ensures that only the validator who has tagged the pieces can send the challenge, making it redundant to publish the tags in the DHT.
  3. Local Tag Table: Instead of relying on the DHT for tags, we now maintain a local table of tags.

closes #41

@rishiad
feat: move dht insert durring piece proccsing
25418d2 
only tagging is done on the worker threads and put into a local table
rayokamoto
rayokamoto requested changes Jan 23, 2025
View reviewed changes
db/migrations/20241212075345_validator_db.sql Outdated Show resolved Hide resolved
db/migrations/20241212075345_validator_db.sql Outdated Show resolved Hide resolved
storb/validator/routes.py Outdated Show resolved Hide resolved
storb/validator/validator.py Outdated Show resolved Hide resolved
storb/validator/validator.py Show resolved Hide resolved
storb/validator/piece_processing.py Show resolved Hide resolved
@rishiad rishiad requested review from rayokamoto and removed request for Shr1ftyy January 23, 2025 12:41
@rishiad
refactor: clean up whitespace and variable naming in piece processing…
5b4d6da 
… logic

fix: correct formatting in start_piece_consumers docstring
@rishiad rishiad force-pushed the fix/inconsistent-dht branch from e118397 to 5b4d6da Compare January 23, 2025 12:49
@rishiad rishiad merged commit 25d24c4 into main Jan 23, 2025
1 check passed
@rayokamoto rayokamoto deleted the fix/inconsistent-dht branch January 23, 2025 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rayokamoto rayokamoto rayokamoto approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Inconsistent DHT Storage with Multiple Piece Queue Consumers
2 participants
@rishiad @rayokamoto