Feat: ancillary archive creation #2191

dlachaume · 2024-12-23T17:52:32Z

Content

This PR updates the AncillaryArtifactBuilder to include the creation of an archive from the ancillary files of the Cardano database. The archive contains:

The /ledger directory and its content.
The /volatile directory and its content.
The /immutable directory, including the latest trio of immutable files (.chunk, .primary, .secondary).

Additionally, the returned locations for downloading the archive are now referenced in the artifact.

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- No clippy warnings in the CI
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- Update README file (if relevant)
- Update documentation website (if relevant)
- Add dev blog post (if relevant)

Issue(s)

Relates to #2151

mithril-common/src/digesters/dummy_cardano_db.rs

github-actions · 2024-12-23T18:00:37Z

Test Results

4 files ±0 52 suites ±0 10m 25s ⏱️ +6s
1 479 tests +9 1 479 ✅ +9 0 💤 ±0 0 ❌ ±0
1 735 runs +9 1 735 ✅ +9 0 💤 ±0 0 ❌ ±0

Results for commit 8d374ca. ± Comparison against base commit 00567fe.

This pull request removes 3 and adds 12 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_archive_should_return_all_uploaders_returned_locations
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_archive_should_return_empty_locations_with_no_uploader
mithril-aggregator ‑ file_uploaders::local_uploader::tests::should_extract_digest_to_deduce_location

mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::create_ancillary_builder_should_error_when_no_uploader
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::create_archive_should_embed_ledger_volatile_directories_and_last_immutables
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_ancillary_archive_should_error_when_no_location_is_returned
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_ancillary_archive_should_log_upload_errors
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_ancillary_archive_should_return_all_uploaders_returned_locations
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_ancillary_archive_should_return_location_even_with_uploaders_errors
mithril-aggregator ‑ artifact_builder::cardano_database_artifacts::ancillary::tests::upload_should_return_error_and_not_upload_when_archive_creation_fails
mithril-aggregator ‑ file_uploaders::local_snapshot_uploader::tests::should_copy_file_to_target_location
mithril-aggregator ‑ file_uploaders::local_snapshot_uploader::tests::should_error_if_path_is_a_directory
mithril-aggregator ‑ file_uploaders::local_snapshot_uploader::tests::should_extract_digest_to_deduce_location
…

♻️ This comment has been updated with latest results.

mithril-aggregator/src/dependency_injection/builder.rs

mithril-aggregator/src/file_uploaders/mod.rs

mithril-aggregator/src/file_uploaders/utils.rs

jpraynaud

LGTM 👍

mithril-aggregator/src/artifact_builder/cardano_database_artifacts/ancillary.rs

Co-authored-by: Sébastien Fauvel <[email protected]>

…rl` instead of `String` Co-authored-by: Jean-Philippe Raynaud <[email protected]> Co-authored-by: Sébastien Fauvel <[email protected]>

Co-authored-by: Sébastien Fauvel <[email protected]>

…gator builder

… uploader Co-authored-by: Sébastien Fauvel <[email protected]>

…ned while trying to upload the archive Co-authored-by: Sébastien Fauvel <[email protected]>

Co-authored-by: Sébastien Fauvel <[email protected]>

jpraynaud

LGTM 👍

Alenar

Lgtm 🚀

…one is returning a location

Co-authored-by: Sébastien Fauvel <[email protected]>

* mithril-aggregator from `0.6.8` to `0.6.9` * mithril-client from `0.10.6` to `0.10.7` * mithril-common from `0.4.100` to `0.4.101` * mithril-end-to-end from `0.4.59` to `0.4.60`

dlachaume self-assigned this Dec 23, 2024

github-advanced-security bot found potential problems Dec 23, 2024

View reviewed changes

mithril-common/src/digesters/dummy_cardano_db.rs Fixed Show fixed Hide fixed

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch 2 times, most recently from 66638f7 to d412a36 Compare December 24, 2024 15:57

dlachaume temporarily deployed to testing-preview December 24, 2024 16:34 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet December 24, 2024 16:34 — with GitHub Actions Inactive

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from 9290e74 to d412a36 Compare January 2, 2025 09:00

Base automatically changed from ensemble/2151/extend-snapshotter-for-specific-files-directories to main January 2, 2025 09:22

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch 2 times, most recently from c89cd05 to fa7a3c6 Compare January 2, 2025 11:32

github-advanced-security bot found potential problems Jan 2, 2025

View reviewed changes

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch 2 times, most recently from 5fa147a to 3944b85 Compare January 3, 2025 14:03

dlachaume temporarily deployed to testing-preview January 3, 2025 14:15 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 3, 2025 14:15 — with GitHub Actions Inactive

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from 3944b85 to bc32bd9 Compare January 3, 2025 14:17

dlachaume marked this pull request as ready for review January 3, 2025 14:17

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from bc32bd9 to 7e1e7a3 Compare January 3, 2025 14:18

dlachaume temporarily deployed to testing-preview January 3, 2025 14:27 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 3, 2025 14:27 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-preview January 3, 2025 14:58 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 3, 2025 14:58 — with GitHub Actions Inactive

dlachaume assigned sfauvel Jan 3, 2025

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from 1f1f31a to 16ba5af Compare January 3, 2025 15:33

dlachaume temporarily deployed to testing-preview January 3, 2025 15:42 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 3, 2025 15:42 — with GitHub Actions Inactive

dlachaume requested review from jpraynaud and Alenar January 3, 2025 15:43

jpraynaud approved these changes Jan 3, 2025

View reviewed changes

dlachaume and others added 13 commits January 6, 2025 15:46

feat: create archive for ancillary files

3ba6e22

Co-authored-by: Sébastien Fauvel <[email protected]>

feat: integrate ancillary archive creation into the upload process

b294513

feat: reference the ancillary artifact location in the artifact

6f95567

feat: make the ancillary archive name more precise

86ef2ab

refactor: remove http_server dependency from LocalUploader

6294a1e

Co-authored-by: Sébastien Fauvel <[email protected]>

refactor: rename LocalUploader to LocalSnapshotUploader

9298db2

refactor: type server_url_prefix from LocalSnapshotUploader to `U…

135c6ef

…rl` instead of `String` Co-authored-by: Jean-Philippe Raynaud <[email protected]> Co-authored-by: Sébastien Fauvel <[email protected]>

feat: implement standard LocalUploader

b6cf1b2

Co-authored-by: Sébastien Fauvel <[email protected]>

refactor: create function to build the server url prefix in the aggre…

b8668e8

…gator builder

refactor: rewording and adjusting values in tests

536dd40

feat: return error when AncillaryArtifactBuilder is created without…

f7c5bad

… uploader Co-authored-by: Sébastien Fauvel <[email protected]>

feat: log uploader errors and return an error if no location is retur…

a041e51

…ned while trying to upload the archive Co-authored-by: Sébastien Fauvel <[email protected]>

feat: enable Snapshotter to create archives in subdirectories

bda4968

Co-authored-by: Sébastien Fauvel <[email protected]>

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from c58f738 to 8a8caed Compare January 6, 2025 14:47

jpraynaud approved these changes Jan 6, 2025

View reviewed changes

dlachaume temporarily deployed to testing-preview January 6, 2025 14:56 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 6, 2025 14:56 — with GitHub Actions Inactive

fix: remove useless async attribute in test

e723715

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from 8a8caed to d79b231 Compare January 6, 2025 15:29

dlachaume temporarily deployed to testing-preview January 6, 2025 15:38 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 6, 2025 15:38 — with GitHub Actions Inactive

Alenar approved these changes Jan 6, 2025

View reviewed changes

dlachaume and others added 3 commits January 6, 2025 17:41

test: verify upload behavior when uploaders are failing and at least …

1eee194

…one is returning a location

chore: reference the feature in the CHANGELOG

1e77688

Co-authored-by: Sébastien Fauvel <[email protected]>

chore: upgrade crate versions

8d374ca

* mithril-aggregator from `0.6.8` to `0.6.9` * mithril-client from `0.10.6` to `0.10.7` * mithril-common from `0.4.100` to `0.4.101` * mithril-end-to-end from `0.4.59` to `0.4.60`

dlachaume force-pushed the ensemble/2151/create-ancillary-archive branch from d79b231 to 8d374ca Compare January 6, 2025 16:41

dlachaume temporarily deployed to testing-preview January 6, 2025 16:50 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet January 6, 2025 16:50 — with GitHub Actions Inactive

dlachaume merged commit bf038eb into main Jan 6, 2025
51 checks passed

dlachaume deleted the ensemble/2151/create-ancillary-archive branch January 6, 2025 16:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: ancillary archive creation #2191

Feat: ancillary archive creation #2191

dlachaume commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

jpraynaud left a comment

jpraynaud left a comment

Alenar left a comment

Feat: ancillary archive creation #2191

Feat: ancillary archive creation #2191

Conversation

dlachaume commented Dec 23, 2024 • edited Loading

Content

Pre-submit checklist

Issue(s)

github-actions bot commented Dec 23, 2024 • edited Loading

Test Results

jpraynaud left a comment

Choose a reason for hiding this comment

jpraynaud left a comment

Choose a reason for hiding this comment

Alenar left a comment

Choose a reason for hiding this comment

dlachaume commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading