Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2304 enhancement announced blocks parallel download #2320

Open
wants to merge 26 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
41a51d1
Parallel blocks download
ErakhtinB Dec 26, 2024
0ded858
Merge remote-tracking branch 'origin/master' into 2304-enhancement-an…
ErakhtinB Dec 26, 2024
d65d680
Correct TODO format
ErakhtinB Dec 26, 2024
7714e61
Ranges for indices
ErakhtinB Dec 26, 2024
8d9f2c6
Merge remote-tracking branch 'origin/master' into 2304-enhancement-an…
ErakhtinB Dec 26, 2024
2aa0cf0
Build fix
ErakhtinB Dec 26, 2024
35daeb7
No peer manager is used to work with active peers
ErakhtinB Dec 26, 2024
dd28c19
clang tidy fixes
ErakhtinB Dec 26, 2024
7d99519
Merge branch 'master' into 2304-enhancement-announced-blocks-parallel…
ErakhtinB Dec 26, 2024
5c6d66d
Better erasing from active_peers
ErakhtinB Dec 26, 2024
4f9a09b
random gen is member of sync impl
ErakhtinB Dec 27, 2024
0807169
Just log, remove
ErakhtinB Dec 27, 2024
c8e46b6
Multiple blocks download by announce
ErakhtinB Dec 27, 2024
9604a31
Random peers downloading only in syncByHeader
ErakhtinB Dec 27, 2024
ab41109
Check for peer id while iteratring over peer states
ErakhtinB Dec 27, 2024
b5630f4
Maximum downloads are handled correct
ErakhtinB Dec 31, 2024
4a3ddce
Best way to handle
ErakhtinB Dec 31, 2024
6164c3b
Only first block of a pack checked in load_blocks
ErakhtinB Jan 6, 2025
63bdc91
Merge remote-tracking branch 'origin/master' into 2304-enhancement-an…
ErakhtinB Jan 6, 2025
3b621de
Better request start res handling
ErakhtinB Jan 6, 2025
d533fb5
Better iteration over blocks
ErakhtinB Jan 8, 2025
e749488
Merge branch 'master' into 2304-enhancement-announced-blocks-parallel…
ErakhtinB Jan 8, 2025
632eadb
Merge branch 'master' into 2304-enhancement-announced-blocks-parallel…
ErakhtinB Jan 8, 2025
49f3845
Erase load_blocks Logic adjustment
ErakhtinB Jan 10, 2025
6df59ac
better init
ErakhtinB Jan 10, 2025
9337b28
Logging load blocks bad result
ErakhtinB Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions core/application/app_configuration.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,8 @@ namespace kagome::application {
const = 0;

virtual std::optional<PrecompileWasmConfig> precompileWasm() const = 0;

virtual uint32_t maxParallelDownloads() const = 0;
};

} // namespace kagome::application
14 changes: 10 additions & 4 deletions core/application/impl/app_configuration_impl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ namespace {
#endif
const uint32_t def_db_cache_size = 1024;
const uint32_t def_parachain_runtime_instance_cache_size = 100;
const uint32_t def_max_parallel_downloads = 5;

/**
* Generate once at run random node name if form of UUID
Expand Down Expand Up @@ -177,12 +178,11 @@ namespace {

static constexpr std::array<std::string_view,
1 + KAGOME_WASM_COMPILER_WASM_EDGE>
interpreters {
interpreters{
#if KAGOME_WASM_COMPILER_WASM_EDGE == 1
"WasmEdge",
"WasmEdge",
#endif
"Binaryen"
};
"Binaryen"};

static const std::string interpreters_str =
fmt::format("[{}]", fmt::join(interpreters, ", "));
Expand Down Expand Up @@ -845,6 +845,9 @@ namespace kagome::application {
("rpc-methods", po::value<std::string>(), R"("auto" (default), "unsafe", "safe")")
("no-mdns", po::bool_switch(), "(unused, zombienet stub)")
("prometheus-external", po::bool_switch(), "alias for \"--prometheus-host 0.0.0.0\"")
("max-parallel-downloads", po::value<uint32_t>()->default_value(def_max_parallel_downloads),
"Maximum number of peers from which to ask for the same blocks in parallel."
"This allows downloading announced blocks from multiple peers. Decrease to save traffic and risk increased latency.")
;

po::options_description development_desc("Additional options");
Expand Down Expand Up @@ -1606,6 +1609,9 @@ namespace kagome::application {
runtime_exec_method_ = RuntimeExecutionMethod::Compile;
}

max_parallel_downloads_ =
find_argument<uint32_t>(vm, "max-parallel-downloads")
.value_or(def_max_parallel_downloads);
// if something wrong with config print help message
if (not validate_config()) {
std::cout << desc << '\n';
Expand Down
5 changes: 5 additions & 0 deletions core/application/impl/app_configuration_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,10 @@ namespace kagome::application {
return precompile_wasm_;
}

uint32_t maxParallelDownloads() const override {
return max_parallel_downloads_;
}

private:
void parse_general_segment(const rapidjson::Value &val);
void parse_blockchain_segment(const rapidjson::Value &val);
Expand Down Expand Up @@ -386,6 +390,7 @@ namespace kagome::application {
std::max<size_t>(std::thread::hardware_concurrency(), 1)};
bool disable_secure_mode_{false};
std::optional<PrecompileWasmConfig> precompile_wasm_;
uint32_t max_parallel_downloads_;
};

} // namespace kagome::application
Expand Down
1 change: 0 additions & 1 deletion core/blockchain/impl/block_tree_impl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,6 @@ namespace kagome::blockchain {
std::move(justification_storage_policy),
state_pruner,
main_thread_pool));

// Add non-finalized block to the block tree
for (auto &e : collected) {
const auto &block = e.first;
Expand Down
Loading
Loading