Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(trie): use local ThreadPool in Parallel::multiproof #13416

Merged
merged 3 commits into from
Dec 19, 2024

Conversation

fgimenez
Copy link
Member

@fgimenez fgimenez commented Dec 16, 2024

Aiming to prevent deadlocks when enabling ParallelProof::multiproof ref #13260 it allows us to process proofs in chunks, so that we don't spin up tasks for all the existing proofs at once, potentially causing resource exhaustion and rayon contention.

the goal of this change is just to prevent the deadlock we were experimenting with when enabling parallel multiproof, but I also wanted to verify that splitting the workload in chunks doesn't add a significant overhead, fixing the time calculation for the state root task we have consistently better times compared to the regular calculation (state root task time is usually a bit above half of the parallel calculation time)

Dec 17 07:39:37 reth-5 reth-holesky[344736]: 2024-12-17T07:39:37.597759Z  INFO engine::tree: State root task finished block=NumHash { number: 2953384, hash: 0xc80f1ab43ca7cbc67c22edd9e32ae6b039efe06dfccc588e1160906259ff616d } total_time=102.27362ms task_state_root=0x1b50b96acf75949b0ace02fcad63ad56242c94cc93ef51f64d8c87309359d330
Dec 17 07:39:37 reth-5 reth-holesky[344736]: 2024-12-17T07:39:37.664179Z DEBUG engine::tree: Calculated state root root_elapsed=191.658237ms block=NumHash { number: 2953384, hash: 0xc80f1ab43ca7cbc67c22edd9e32ae6b039efe06dfccc588e1160906259ff616d }

@fgimenez fgimenez changed the title perf(trie): split Parallel::multiproof workload in chunks perf(trie): use local ThreadPool in Parallel::multiproof Dec 17, 2024
@fgimenez fgimenez force-pushed the fgimenez/chunked-parallel-multiproof branch from 90d68f7 to 4e45575 Compare December 17, 2024 09:50
@fgimenez fgimenez force-pushed the fgimenez/chunked-parallel-multiproof branch from 1fdf1ca to d2372a3 Compare December 17, 2024 15:26
@fgimenez fgimenez force-pushed the fgimenez/chunked-parallel-multiproof branch 2 times, most recently from 4514574 to 37fb95f Compare December 18, 2024 06:22
@fgimenez fgimenez added C-perf A change motivated by improving speed, memory usage or disk footprint A-blockchain-tree Related to sidechains, reorgs and pending blocks A-trie Related to Merkle Patricia Trie implementation labels Dec 18, 2024
@fgimenez fgimenez force-pushed the fgimenez/chunked-parallel-multiproof branch from 37fb95f to 52f9ae3 Compare December 18, 2024 20:52
Copy link
Collaborator

@shekhirin shekhirin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits, @rkrasiuk PTAL

crates/trie/parallel/src/proof.rs Show resolved Hide resolved
crates/trie/parallel/src/proof.rs Outdated Show resolved Hide resolved
@fgimenez fgimenez added this pull request to the merge queue Dec 19, 2024
Merged via the queue into main with commit d1b3dee Dec 19, 2024
42 checks passed
@fgimenez fgimenez deleted the fgimenez/chunked-parallel-multiproof branch December 19, 2024 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-blockchain-tree Related to sidechains, reorgs and pending blocks A-trie Related to Merkle Patricia Trie implementation C-perf A change motivated by improving speed, memory usage or disk footprint
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants