Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rust] legacy compatibility for forensic db #329

Merged
merged 106 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
af3f8a2
add fixtures and scaffold compatibility structs
0o-de-lally Oct 2, 2024
82dd710
builds
0o-de-lally Oct 2, 2024
93714d7
parsese manifest file
0o-de-lally Oct 2, 2024
8db6fc3
try port HashValue from V5
0o-de-lally Oct 3, 2024
b157095
create compatibility crate
0o-de-lally Oct 4, 2024
076a975
can parse DiemAccountResource
0o-de-lally Oct 5, 2024
d71db2d
names
0o-de-lally Oct 5, 2024
3bf6e45
readme
0o-de-lally Oct 5, 2024
36b1721
no one reads this anyway
0o-de-lally Oct 5, 2024
07d5e2e
cleanup and document decoding test
0o-de-lally Oct 5, 2024
ac5c688
test manifest can be loaded
0o-de-lally Oct 5, 2024
9ed6ef0
include move_resource_v5
0o-de-lally Oct 6, 2024
88e2eca
add transaction backup manifest parsing
0o-de-lally Oct 6, 2024
d702701
wip port all transaction types
0o-de-lally Oct 7, 2024
3440b33
can get remaining Input error
0o-de-lally Oct 7, 2024
cd66993
wip filling structs
0o-de-lally Oct 7, 2024
6253782
wip
0o-de-lally Oct 7, 2024
4a41416
tx parsing working
0o-de-lally Oct 8, 2024
1bcb815
scaffold warehouse cli
0o-de-lally Oct 8, 2024
9f0240b
wip scanning dir for manifests
0o-de-lally Oct 8, 2024
47f83cb
test for version
0o-de-lally Oct 8, 2024
e1a8887
patch x25519 dalek pinned dependency zeroize
0o-de-lally Oct 8, 2024
6ad7547
using patched diem dependencies
0o-de-lally Oct 9, 2024
d354995
add sqlx
0o-de-lally Oct 9, 2024
c69c12a
sqlx works in test harness
0o-de-lally Oct 9, 2024
d7ff8dc
testing sqlx migration voodoo
0o-de-lally Oct 10, 2024
b979eea
scaffold init
0o-de-lally Oct 10, 2024
ed1ca2c
clippy
0o-de-lally Oct 10, 2024
0363220
scaffold loading with sqlite
0o-de-lally Oct 10, 2024
8d28528
can init two tables with user account
0o-de-lally Oct 10, 2024
f7f3440
extract v5 state into warehouse table structs
0o-de-lally Oct 23, 2024
69a2970
account loading with deduplication
0o-de-lally Oct 23, 2024
3aa83d0
patch e2e test
0o-de-lally Oct 23, 2024
355891b
scaffold batch loading
0o-de-lally Oct 24, 2024
0f3f90e
prevents duplicates
0o-de-lally Oct 24, 2024
9f84641
scaffold loading coin state
0o-de-lally Oct 24, 2024
82a348f
batch insert coins and tests
0o-de-lally Oct 24, 2024
9ffedf4
balance queries can coerce types
0o-de-lally Oct 25, 2024
69d40d7
add test for incremental balances
0o-de-lally Oct 25, 2024
e517803
implement first test with testcontainers
0o-de-lally Oct 28, 2024
f067943
fixes false positive for container ready
0o-de-lally Oct 28, 2024
7e07e3e
can migrate a pg db
0o-de-lally Oct 28, 2024
e5eb10c
wip converting to postgres
0o-de-lally Oct 28, 2024
2361f6e
all tests passing in postgres
0o-de-lally Oct 29, 2024
8010005
extract and load v7 account state working and tested
0o-de-lally Oct 29, 2024
664f1ae
AGE graph db works, accepts cypher query
0o-de-lally Oct 30, 2024
0119867
adds transaction archive parsing
0o-de-lally Oct 30, 2024
5f1a4f6
can parse transaction archive, and detect transfer type transactions
0o-de-lally Oct 30, 2024
c72ca85
add framework_sdk_builder to compatibility
0o-de-lally Oct 31, 2024
c23cbfc
able to parse v7 transaction logs
0o-de-lally Oct 31, 2024
7186327
try to get known events
0o-de-lally Oct 31, 2024
4eef151
add V6 transaction backup fixtures
0o-de-lally Oct 31, 2024
d7c1c6e
clippy
0o-de-lally Oct 31, 2024
f1dd28d
unzip v6 files
0o-de-lally Oct 31, 2024
bff43ea
clean
0o-de-lally Oct 31, 2024
eec796e
parsing events from all transactions
0o-de-lally Oct 31, 2024
14ca58f
scaffold gzip of archives
0o-de-lally Nov 4, 2024
48bd503
create unzip helpers, add zip files as fixtures
0o-de-lally Nov 4, 2024
cb74b5a
scaffold end to end ingest of archives
0o-de-lally Nov 4, 2024
044db4d
add earlier snapshots for v6
0o-de-lally Nov 4, 2024
6ea291a
can insert account data from two snapshot archives
0o-de-lally Nov 4, 2024
6ecb7c0
document AGE voodoo
0o-de-lally Nov 5, 2024
7becd10
add neo4j testcontainer
0o-de-lally Nov 5, 2024
241e494
test neo4j connection
0o-de-lally Nov 5, 2024
a25b4b8
cypher queries to insert txs
0o-de-lally Nov 6, 2024
ccef148
Squashed commit of the following:
0o-de-lally Nov 6, 2024
5482239
neo4j indexing
0o-de-lally Nov 6, 2024
2ac47e7
update transaction parsing to include v6 types
0o-de-lally Nov 6, 2024
7c800ba
scaffold e2e test for load tx to neo4j
0o-de-lally Nov 6, 2024
fd7a244
batch inserts with cypher queries (or bolt types)
0o-de-lally Nov 7, 2024
348183a
cli entry point to load txs to remote
0o-de-lally Nov 7, 2024
67cb275
allow ingest-all to select types of content to ingest e.g. transaction
0o-de-lally Nov 8, 2024
76ba384
catch other tx types besides entry functions
0o-de-lally Nov 8, 2024
4bdadca
adds parsing of supporting data
0o-de-lally Nov 8, 2024
7005c14
wip cypher for supporting data
0o-de-lally Nov 9, 2024
5adeb3b
add neo4j meta tests for timestamp
0o-de-lally Nov 9, 2024
0faf197
cli and e2e tests for swap data
0o-de-lally Nov 11, 2024
dc6b04d
add transaction data
0o-de-lally Nov 12, 2024
75ec349
include tx_data in relation
0o-de-lally Nov 14, 2024
b74757a
patch tests
0o-de-lally Nov 15, 2024
c27d385
move postgres features to test::experimental
0o-de-lally Nov 15, 2024
2903b0f
patch neo4j constraints
0o-de-lally Nov 16, 2024
933d229
add more data to OL transactions
0o-de-lally Nov 16, 2024
2d632cd
short hash code display collision
0o-de-lally Nov 16, 2024
7e6b832
add note on neo4j data science
0o-de-lally Nov 18, 2024
fb77c72
add simple task queue for loading
0o-de-lally Nov 18, 2024
e283b48
removes warehouse
0o-de-lally Nov 18, 2024
e61318e
update diem to release
0o-de-lally Nov 18, 2024
e978e16
patch cargo.toml
0o-de-lally Nov 18, 2024
4ec31a1
update cargo toml
0o-de-lally Nov 18, 2024
0165443
add coin register event
0o-de-lally Nov 19, 2024
9a5a4c2
nit
0o-de-lally Nov 19, 2024
4ba3415
add ol specific Move resource bindings
0o-de-lally Nov 22, 2024
8209e16
add moveresourcev5
0o-de-lally Nov 22, 2024
ed272be
patch
0o-de-lally Nov 22, 2024
4b82808
make public mods
0o-de-lally Nov 23, 2024
fa947e4
clippy
0o-de-lally Nov 23, 2024
ed61d60
v5 transaction builder for entry function decoding
0o-de-lally Nov 24, 2024
54cd7f9
clean
0o-de-lally Nov 24, 2024
9ed2dd5
clean
0o-de-lally Nov 24, 2024
6ac622c
add transaction_view_v5
0o-de-lally Nov 24, 2024
489f9d9
make raw transaction public
0o-de-lally Nov 25, 2024
8058976
add 5.0.0 genesis code
0o-de-lally Nov 25, 2024
8b20710
add serde to script function calls
0o-de-lally Nov 25, 2024
c57d5d5
add v5 new epoch event type
0o-de-lally Dec 4, 2024
2259d95
Merge branch 'main' into legacy-compat
0o-de-lally Jan 8, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .github/actions/build_env/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,6 @@ runs:
shell: bash
run: rustup component add rustfmt clippy

- name: also rust cache
uses: Swatinem/[email protected]
with:
shared-key: "libra"
cache-on-failure: "true"

# call own action for libra_cli
# known issue with needing to hard code the @version
# https://github.com/orgs/community/discussions/41927
Expand Down
46 changes: 32 additions & 14 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,13 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

# fail fast if types doesnt compile, everything else will fail.
# fail fast if types doesn't compile, everything else will fail.
- name: types
working-directory: ./types
run: cargo test --no-fail-fast
Expand All @@ -67,10 +68,11 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- name: wallet
if: always()
Expand All @@ -94,10 +96,11 @@ jobs:
name: framework-build
path: framework/

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

# Check test suite meta tests
- name: smoke-tests # NOTE: needs working DIEM_FORGE_NODE_BIN_PATH
Expand All @@ -114,10 +117,11 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- uses: actions/[email protected]
with:
Expand All @@ -140,10 +144,11 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- uses: actions/[email protected]
with:
Expand All @@ -165,10 +170,11 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- uses: actions/[email protected]
with:
Expand Down Expand Up @@ -204,10 +210,14 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- name: Run sccache-cache
uses: mozilla-actions/[email protected]

- uses: actions/[email protected]
with:
Expand Down Expand Up @@ -258,17 +268,20 @@ jobs:
- uses: actions/checkout@v3

- name: setup env
# if: ${{runner.environment == 'self-hosted'}}
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- uses: actions/[email protected]
# if: ${{runner.environment == 'self-hosted'}}
with:
name: framework-build
path: framework/
- name: rescue
if: always()
# if: ${{runner.environment == 'self-hosted'}}
working-directory: ./tools/rescue
run: RUST_MIN_STACK=104857600 cargo test --no-fail-fast -- --test-threads=1
storage:
Expand All @@ -277,16 +290,18 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- name: storage
working-directory: ./tools/storage
# TODO: not testing the V6 file parsing
run: cargo test --no-fail-fast

twin-testsuite:
Expand All @@ -295,13 +310,16 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- name: twin
working-directory: ./testsuites/twin
run: cargo test --no-fail-fast
3 changes: 2 additions & 1 deletion .github/workflows/cleanliness.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,11 @@ jobs:
- name: setup env
uses: ./.github/actions/build_env

- uses: Swatinem/rust-cache@v2.7.3
- uses: Swatinem/rust-cache@v2
with:
shared-key: "libra-framework"
cache-all-crates: true
cache-on-failure: true

- name: format
uses: actions-rs/cargo@v1
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,7 @@ on:

jobs:
publish:
permissions:
write-all
permissions: write-all
name: publish
runs-on: ubuntu-latest
steps:
Expand Down
Loading
Loading