Add transparent address gap limit handling & general address rotation functionality. #1673

nuttycom · 2024-12-23T20:43:44Z

No description provided.

codecov · 2024-12-28T05:55:12Z

Codecov Report

Attention: Patch coverage is 62.25166% with 342 lines in your changes missing coverage. Please review.

Project coverage is 54.36%. Comparing base (4ded512) to head (fae0271).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
.../init/migrations/transparent_gap_limit_handling.rs	44.80%	85 Missing ⚠️
zcash_client_sqlite/src/wallet.rs	57.79%	65 Missing ⚠️
zcash_client_sqlite/src/lib.rs	55.30%	59 Missing ⚠️
zcash_client_sqlite/src/wallet/transparent.rs	75.67%	54 Missing ⚠️
zcash_client_sqlite/src/wallet/orchard.rs	0.00%	27 Missing ⚠️
.../src/wallet/init/migrations/ephemeral_addresses.rs	65.38%	9 Missing ⚠️
zcash_client_sqlite/src/wallet/sapling.rs	68.96%	9 Missing ⚠️
..._client_sqlite/src/wallet/transparent/ephemeral.rs	59.09%	9 Missing ⚠️
zcash_client_sqlite/src/error.rs	0.00%	7 Missing ⚠️
...let/init/migrations/fix_broken_commitment_trees.rs	0.00%	5 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1673      +/-   ##
==========================================
+ Coverage   54.33%   54.36%   +0.02%     
==========================================
  Files         173      174       +1     
  Lines       20282    20830     +548     
==========================================
+ Hits        11021    11324     +303     
- Misses       9261     9506     +245

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…handling-prep Preparatory refactoring for #1673

zcash_client_sqlite/src/wallet/db.rs

zcash_client_sqlite/src/error.rs

zcash_client_sqlite/src/lib.rs

str4d

Flushing some comments from an old review.

zcash_transparent/src/keys.rs

zcash_client_sqlite/src/error.rs

zcash_client_backend/src/data_api.rs

str4d

Flushing partial review comments as of 4cbefae.

zcash_client_backend/src/data_api/testing/pool.rs

zcash_client_sqlite/src/wallet/db.rs

str4d · 2025-02-04T05:00:57Z

zcash_client_sqlite/src/wallet/init/migrations/ephemeral_addresses.rs

+                // NB: we have reduced the initial space of generated ephemeral addresses
+                // from 20 addresses to 3, as ephemeral addresses should always be used in
+                // a transaction immediatly after being reserved, and as a consequence
+                // there is no significant benefit in having a larger gap limit.


This is altering an existing migration.

If the new migration does not remove unnecessary gap-limit-generated addresses from previously-migrated databases, then we end up with data inconsistencies between wallets that we need to be aware of and handle (somehow) basically indefinitely.

If the new migration does remove unnecessary gap-limit-generated addresses from previously-migrated databases, then altering this existing migration is unnecessary.

So I don't think we want to alter the migration in either case, and we should instead remove the unneeded gap limit addresses from the removed table (which we can reliably do leveraging its structure).

The rationale for altering this migration is the same as the rationale for a 3-address ephemeral gap limit being acceptable in the first place: in practice, no such gaps should appear; therefore, the existing allocation of 20 addresses just means that those addresses are persisted to the wallet "early".

In some sense, all 2^31 possible addresses already "exist"; it's just a question of whether the wallet has written them down locally.

zcash_client_sqlite/src/wallet.rs

str4d

Flushing review comments as of f07321e. I only have zcash_client_sqlite/src/wallet/transparent/ephemeral.rs and zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs left to review.

zcash_client_backend/src/data_api.rs

zcash_client_sqlite/src/wallet/transparent.rs

str4d · 2025-02-06T10:07:44Z

zcash_client_sqlite/src/wallet/transparent.rs

+/// The provided [`UnifiedAddressRequest`] is used to pregenerate unified addresses that correspond
+/// to the transparent address index in question; such unified addresses need not internally
+/// contain a transparent receiver, and may be overwritten when these addresses are exposed via the
+/// [`WalletWrite::get_next_available_address`] or [`WalletWrite::get_address_for_index`] methods.


TODO: Confirm that they do indeed get overwritten, rather than having their receivers merged.

get_next_available_address calls upsert_address, and there's actually a problem here; the current semantics of that method are that they won't alter the composition of the address, so what will happen is that the generated address won't actually be persisted (though it will be returned to the user.) What should probably happen is that some placeholder value for the address should be inserted into the address field, and the update should occur only if the value is that placeholder; otherwise get_next_available_address should return an error.

The other thing here is that we actually want get_next_available_address to alter its behavior depending upon the address request; if the request contains a transparent element, then its diversifier index should be drawn from within the transparent range, i.e. it should be the first unreserved address within the gap; if it doesn't contain a transparent element, then it should correspond to the first unreserved diversifier index in the range of values greater than 2^32 (I presume we don't want to do anything in the space between 2^31 and 2^32 because that's the "hardened transparent" range?)

zcash_client_sqlite/src/wallet/transparent.rs

This is a large change that unifies the handling of ephemeral transparent addresses for ZIP 320 support with generalized "gap limit" handling for transparent wallet recovery. The best way to understand this commit is to start from the `transparent_gap_limit_handling` database migration that drives the change in behavior.

…limit handling.

Co-authored-by: Jack Grigg <[email protected]>

…resses. The rationale behind this change is that current implementations of UTXO retrieval do not attempt to avoid revealing that multiple addresses belong to the same wallet when using a light wallet server to check for transparent UTXOs. A comprehensive solution to this problem requires changes to the light wallet protocol such that wallets no longer need to submit batches of addresses to the light wallet server, and can instead determine whether outputs exist for their addresses locally. By setting the default gap limit to 10 addresses, we make it possible for wallets that add privacy-preserving UTXO checking in the future (either by connecting to full node indexes directly in a privacy-preserving manner or via changes that take advantage of an updated light wallet protocol) to then start generating new addresses within a range of the industry-standard 20 address gap limit that have not yet been revealed to belong to the wallet.

nuttycom · 2025-02-07T01:13:42Z

force-pushed to address comments from code review

Then force-pushed to rebase on main

In the presence of gap-limit addresses, querying for UTXOs based exclusively upon the last height at which such a query was executed is inadequate to fully recover the contents of a wallet; when a transaction is discovered in the past, this may advance the gap limit, and newly exposed addresses may receive outputs immediately following the discovered transaction. We can expect that when a transparent output belonging to the wallet is discovered, that the height at which that transaction was mined was checked in a query that included _all_ of addresses having child indices less than or equal to that of the involved address. Therefore, it's safe to start looking for addresses in the gap as of that point.

…ess exposure height.

nuttycom · 2025-02-07T17:55:23Z

force-pushed to fix bugs caught in CI

str4d

Probably utACK fae0271 but the exposed_at_height needs either addressing or reasoning.

str4d · 2025-02-08T09:00:56Z

zcash_client_sqlite/src/error.rs

    #[cfg(feature = "transparent-inputs")]
-    ReachedGapLimit(AccountUuid, u32),
+    ReachedGapLimit(u32, KeyScope),


Nit: KeyScope is a level above the address index, so I'd prefer the tuple being in the opposite order.

str4d · 2025-02-08T09:17:10Z

zcash_client_sqlite/src/wallet/transparent/ephemeral.rs

@@ -1,184 +1,68 @@
-//! Functions for wallet support of ephemeral transparent addresses.
-use std::cmp::{max, min};
+//! Functikjns for wallet support of ephemeral transparent addresses.


No idea how this got broken.

Suggested change

//! Functikjns for wallet support of ephemeral transparent addresses.

//! Functions for wallet support of ephemeral transparent addresses.

str4d · 2025-02-08T09:39:06Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+        conn.execute_batch(&format!(
+            r#"
+            ALTER TABLE addresses ADD COLUMN key_scope INTEGER NOT NULL DEFAULT {external_scope_code};
+            ALTER TABLE addresses ADD COLUMN transparent_child_index INTEGER;


An exposed_at_height column is not added here, so AFAICT that column will be null for every existing address in the table, while ephemeral addresses that get inserted have this set correctly. Given that going forward we set this for all address scopes (not just ephemeral AFAIR), we should set this correctly for existing addresses as well (to the best of our knowledge, i.e. no worse than if the wallet were recovered from seed).

str4d · 2025-02-08T09:41:02Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+
+        conn.execute_batch(&format!(
+            r#"
+            ALTER TABLE addresses ADD COLUMN key_scope INTEGER NOT NULL DEFAULT {external_scope_code};


Confirming that we did indeed only store external addresses in here previously? Definitely the case for shielded, and I believe we weren't generating change addresses for transparent.

str4d · 2025-02-08T09:42:14Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+            CREATE TABLE addresses_new (
+                id INTEGER NOT NULL PRIMARY KEY,
+                account_id INTEGER NOT NULL,
+                key_scope INTEGER NOT NULL DEFAULT {external_scope_code},


Do we need to leave the default set? It's convenient for the migration (so could be retained above), but it could cause us issues during future query refactoring if the final table auto-fills the column (fixing a data error here would not be pleasant).

nuttycom marked this pull request as draft December 23, 2024 20:43

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 16 times, most recently from 62c1394 to bd2df86 Compare December 28, 2024 04:13

nuttycom changed the title ~~WIP: Add transparent address gap limit handling & general address rotation functionality.~~ Add transparent address gap limit handling & general address rotation functionality. Dec 28, 2024

nuttycom marked this pull request as ready for review December 28, 2024 04:14

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from bd2df86 to 4b99663 Compare December 28, 2024 04:27

nuttycom requested a review from str4d December 28, 2024 04:27

nuttycom mentioned this pull request Dec 30, 2024

Preparatory refactoring for zcash/librustzcash#1673 #1675

Merged

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from 1386dd1 to 49230de Compare December 30, 2024 22:20

str4d added a commit that referenced this pull request Dec 30, 2024

Merge pull request #1675 from nuttycom/feature/transparent_gap_limit_…

5651d80

…handling-prep Preparatory refactoring for #1673

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from b39c6c3 to 3ac0396 Compare December 30, 2024 22:33

str4d reviewed Dec 30, 2024

View reviewed changes

zcash_client_sqlite/src/wallet/db.rs Outdated Show resolved Hide resolved

str4d reviewed Dec 30, 2024

View reviewed changes

zcash_client_sqlite/src/wallet/db.rs Outdated Show resolved Hide resolved

daira reviewed Jan 29, 2025

View reviewed changes

zcash_client_sqlite/src/error.rs Show resolved Hide resolved

daira reviewed Jan 29, 2025

View reviewed changes

zcash_client_sqlite/src/lib.rs Show resolved Hide resolved

str4d reviewed Jan 31, 2025

View reviewed changes

zcash_transparent/src/keys.rs Outdated Show resolved Hide resolved

zcash_client_sqlite/src/error.rs Outdated Show resolved Hide resolved

zcash_client_sqlite/src/error.rs Outdated Show resolved Hide resolved

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from 8796acb to 0fa362f Compare January 31, 2025 03:37

str4d reviewed Jan 31, 2025

View reviewed changes

zcash_client_backend/src/data_api.rs Outdated Show resolved Hide resolved

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 3 times, most recently from 3a91e3c to 4cbefae Compare February 4, 2025 18:10

str4d reviewed Feb 4, 2025

View reviewed changes

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from 021fa6b to f07321e Compare February 6, 2025 00:32

str4d requested changes Feb 6, 2025

View reviewed changes

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from ec4c558 to 3bba679 Compare February 7, 2025 01:09

nuttycom and others added 4 commits February 6, 2025 18:12

zcash_client_sqlite: Add a test for external transparent address gap …

8ac7a6f

…limit handling.

Apply suggestions from code review

0794232

Co-authored-by: Jack Grigg <[email protected]>

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from 3bba679 to 15063c6 Compare February 7, 2025 01:12

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from 15063c6 to a7f04c8 Compare February 7, 2025 16:53

nuttycom added 6 commits February 7, 2025 10:46

Move transparent gap limit tests to zcash_client_backend

004b6e0

zcash_client_sqlite: Add birthday height to Account type.

a370d41

zcash_client_sqlite: Implement get_address_at_index

0c06c65

zcash_client_sqlite: Use ephemeral_addresses.used_in_tx to set addr…

c772b43

…ess exposure height.

Address comments from code review.

fae0271

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from a7f04c8 to fae0271 Compare February 7, 2025 17:54

str4d requested changes Feb 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add transparent address gap limit handling & general address rotation functionality. #1673

Add transparent address gap limit handling & general address rotation functionality. #1673

nuttycom commented Dec 23, 2024

codecov bot commented Dec 28, 2024 •

edited

Loading

str4d left a comment

str4d left a comment

str4d Feb 4, 2025

nuttycom Feb 6, 2025 •

edited

Loading

str4d left a comment

str4d Feb 6, 2025

nuttycom Feb 6, 2025 •

edited

Loading

nuttycom Feb 6, 2025

nuttycom commented Feb 7, 2025 •

edited

Loading

nuttycom commented Feb 7, 2025

str4d left a comment

str4d Feb 8, 2025

str4d Feb 8, 2025

str4d Feb 8, 2025

str4d Feb 8, 2025

str4d Feb 8, 2025

	//! Functikjns for wallet support of ephemeral transparent addresses.
	//! Functions for wallet support of ephemeral transparent addresses.

Add transparent address gap limit handling & general address rotation functionality. #1673

Are you sure you want to change the base?

Add transparent address gap limit handling & general address rotation functionality. #1673

Conversation

nuttycom commented Dec 23, 2024

codecov bot commented Dec 28, 2024 • edited Loading

Codecov Report

str4d left a comment

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuttycom Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuttycom Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuttycom commented Feb 7, 2025 • edited Loading

nuttycom commented Feb 7, 2025

str4d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 28, 2024 •

edited

Loading

nuttycom Feb 6, 2025 •

edited

Loading

nuttycom Feb 6, 2025 •

edited

Loading

nuttycom commented Feb 7, 2025 •

edited

Loading