Add transparent address gap limit handling & general address rotation functionality. #1673

nuttycom · 2024-12-23T20:43:44Z

No description provided.

codecov · 2024-12-28T05:55:12Z

Codecov Report

Attention: Patch coverage is 59.70549% with 301 lines in your changes missing coverage. Please review.

Project coverage is 52.08%. Comparing base (5651d80) to head (6238af4).

Files with missing lines	Patch %	Lines
.../init/migrations/transparent_gap_limit_handling.rs	46.37%	74 Missing ⚠️
zcash_client_sqlite/src/wallet.rs	60.99%	55 Missing ⚠️
zcash_client_sqlite/src/lib.rs	52.33%	51 Missing ⚠️
zcash_client_sqlite/src/wallet/transparent.rs	73.97%	51 Missing ⚠️
zcash_client_sqlite/src/wallet/orchard.rs	0.00%	27 Missing ⚠️
.../src/wallet/init/migrations/ephemeral_addresses.rs	56.00%	11 Missing ⚠️
zcash_client_sqlite/src/wallet/sapling.rs	68.96%	9 Missing ⚠️
..._client_sqlite/src/wallet/transparent/ephemeral.rs	59.09%	9 Missing ⚠️
zcash_client_backend/src/data_api.rs	0.00%	3 Missing ⚠️
zcash_client_sqlite/src/error.rs	0.00%	3 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1673      +/-   ##
==========================================
- Coverage   52.22%   52.08%   -0.15%     
==========================================
  Files         179      180       +1     
  Lines       21353    21759     +406     
==========================================
+ Hits        11152    11333     +181     
- Misses      10201    10426     +225

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…handling-prep Preparatory refactoring for #1673

str4d · 2024-12-30T22:43:05Z

zcash_client_sqlite/src/wallet/db.rs

+/// - `cached_transparent_receiver_address`: the transparent receiver component of address (which
+///   may be the same as `address` in the case of an internal-scope transparent change address or a
+///   ZIP 320 interstitial address). It is cached directly in the table to make account lookups for
+///   transparent outputs more efficient, enabling joins to [`TABLE_TRANSPARENT_RECEIVED_OUTPUTS`].


This documentation is now AFAICT incorrect. It is not "the transparent receiver component of address" because there are cases where address does not contain a transparent receiver, and yet we still populate this column.

Assuming that this is guaranteed to always be the transparent receiver derived from the same diversifier index as address:

Suggested change

/// - `cached_transparent_receiver_address`: the transparent receiver component of address (which

/// may be the same as `address` in the case of an internal-scope transparent change address or a

/// ZIP 320 interstitial address). It is cached directly in the table to make account lookups for

/// transparent outputs more efficient, enabling joins to [`TABLE_TRANSPARENT_RECEIVED_OUTPUTS`].

/// - `cached_transparent_receiver_address`: the transparent address derived from the same

/// viewing key and at the same diversifier index as `address`. This may be the same as

/// `address` in the case of an internal-scope transparent change address or a

/// ZIP 320 interstitial address, and it may be a receiver within `address` in the case of a

/// Unified Address with transparent receiver. It is cached directly in the table to make account lookups for

/// transparent outputs more efficient, enabling joins to [`TABLE_TRANSPARENT_RECEIVED_OUTPUTS`].

str4d · 2024-12-30T22:53:24Z

zcash_client_sqlite/src/wallet/db.rs

+/// - `transparent_child_index`: the diversifier index, if it is in the range of a non-hardened
+///   transparent address index. This is used for gap limit handling and is always populated if the
+///   diversifier index is in that range; since the diversifier index is stored as a byte array we
+///   cannot use SQL integer operations on it and thus need it as an integer as well.


Suggested change

/// - `transparent_child_index`: the diversifier index, if it is in the range of a non-hardened

/// transparent address index. This is used for gap limit handling and is always populated if the

/// diversifier index is in that range; since the diversifier index is stored as a byte array we

/// cannot use SQL integer operations on it and thus need it as an integer as well.

/// - `transparent_child_index`: the diversifier index in integer form, if it is in the range of a `u31`

/// (i.e. a non-hardened transparent address index). It is used for gap limit handling, and is set

/// whenever a transparent address at a given index should be scanned at receive time. This

/// includes:

/// - Unified Addresses with transparent receivers (at any valid index).

/// - Unified Addresses without transparent receivers, but within the gap limit of potential

/// sequential transparent addresses.

/// - Transparent change addresses.

/// - ZIP 320 ephemeral addresses.

/// This column exists because the diversifier index is stored as a byte array, meaning that we

/// cannot use SQL integer operations on it for gap limit calculations, and thus need it as an

/// integer as well.

str4d · 2024-12-30T23:04:45Z

zcash_client_sqlite/src/wallet/db.rs

+/// - `exposed_at_height`: The chain tip height at the time that the address was generated by an
+///   explicit request by the user or reserved for use in a ZIP 320 transaction. In the case of an
+///   address with its first use discovered in a transaction obtained by scanning the chain, this
+///   will be set to the mined height of that transaction.


Suggested change

/// - `exposed_at_height`: The chain tip height at the time that the address was generated by an

/// explicit request by the user or reserved for use in a ZIP 320 transaction. In the case of an

/// address with its first use discovered in a transaction obtained by scanning the chain, this

/// will be set to the mined height of that transaction.

/// - `exposed_at_height`: Our best knowledge as to when this address was first exposed to

/// the wider ecosystem.

/// - For user-generated addresses, this is the chain tip height at the time that the address

/// was generated by an explicit request by the user or reserved for use in a ZIP 320

/// transaction. These heights are not recoverable from chain.

/// - In the case of an address with its first use discovered in a transaction obtained by

/// scanning the chain, this will be set to the mined height of that transaction. In recover

/// from seed cases, this is what user-generated addresses will be assigned.

str4d · 2024-12-30T23:18:51Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+                SELECT account_id, accounts.uivk AS uivk, diversifier_index_be
+                FROM addresses
+                JOIN accounts ON accounts.id = account_id
+                GROUP BY account_id, uivk, diversifier_index_be


This should already be distinct, because all existing addresses were external, and (account_id, diversifier_index_be) was already constrained to be unique.

str4d · 2024-12-30T23:20:38Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+                            cached_transparent_receiver_address = :t_addr
+                        WHERE account_id = :account_id
+                        AND diversifier_index_be = :diversifier_index_be
+                        AND key_scope = :external_scope_code


Not technically required as we just added this column so everything has external scope.

str4d · 2024-12-30T23:24:23Z

zcash_client_sqlite/src/wallet/init/migrations/transparent_gap_limit_handling.rs

+                let diversifier_index = DiversifierIndex::from(
+                    u32::try_from(transparent_child_index).map_err(|_| {
+                        WalletMigrationError::CorruptedData(
+                            "ephermeral address indices must be in the range of `u32`".to_owned(),


Maybe go via NonHardenedChildIndex before DiversifierIndex to detect any "u32 but not u31" corruption.

str4d · 2024-12-30T23:26:39Z

zcash_client_sqlite/src/wallet/db.rs

@@ -216,6 +186,7 @@ CREATE TABLE "sapling_received_notes" (
    memo BLOB,
    commitment_tree_position INTEGER,
    recipient_key_scope INTEGER,
+    address_id INTEGER REFERENCES addresses(id),


Document this new column (and its nullability).

str4d · 2024-12-30T23:26:43Z

zcash_client_sqlite/src/wallet/db.rs

@@ -269,6 +240,7 @@ CREATE TABLE orchard_received_notes (
    memo BLOB,
    commitment_tree_position INTEGER,
    recipient_key_scope INTEGER,
+    address_id INTEGER REFERENCES addresses(id),


Document this new column (and its nullability).

str4d · 2024-12-30T23:26:54Z

zcash_client_sqlite/src/wallet/db.rs

@@ -339,6 +311,7 @@ CREATE TABLE transparent_received_outputs (
    script BLOB NOT NULL,
    value_zat INTEGER NOT NULL,
    max_observed_unspent_height INTEGER,
+    address_id INTEGER REFERENCES addresses(id),


Document this new column (and the reason it was left nullable).

zcash_client_sqlite/src/wallet.rs

str4d · 2024-12-30T23:47:07Z

zcash_client_sqlite/src/wallet.rs

+            :cached_transparent_receiver_address,
+            :exposed_at_height
+        )
+        ON CONFLICT (account_id, diversifier_index_be, key_scope) DO UPDATE


In the case that this upsert is triggered by a note that was received in a pool that the conflicting row's address does not contain, we should upgrade the row to contain that receiver (similar to what we do in zcashd), because this is evidence of that receiver being exposed to the ecosystem. It's not technically evidence of other receivers in address being exposed, so we might only want to add the one receiver triggering the upsert to the conflicting row.

str4d · 2024-12-30T23:57:56Z

zcash_client_sqlite/src/wallet/transparent.rs

+            WITH offsets AS (
+                SELECT
+                    a.transparent_child_index,
+                    LEAD(a.transparent_child_index)


Check what SQLite version introduced this syntax.

This was introduced in 3.25.0

str4d · 2024-12-31T00:05:06Z

zcash_client_sqlite/src/wallet/transparent.rs

+                transparent_child_index + 1,
+                next_child_index - transparent_child_index - 1 AS gap_len
+            FROM offsets
+            WHERE gap_len >= :gap_limit OR next_child_index IS NULL


Add a comment here clarifying that if gap_len is equal to gap_limit (and by extension, or greater than), then next_child_index would not be detected by gap-limit-following scanning logic, and thus we've found the start of the corresponding gap.

str4d · 2024-12-31T00:06:17Z

zcash_client_sqlite/src/wallet/transparent.rs

+            )
+            SELECT
+                transparent_child_index + 1,
+                next_child_index - transparent_child_index - 1 AS gap_len


Add a comment here clarifying that both next_child_index and transparent_child_index are used, so the unused gap between them is one less than their separation.

This is a large change that unifies the handling of ephemeral transparent addresses for ZIP 320 support with generalized "gap limit" handling for transparent wallet recovery. The best way to understand this commit is to start from the `transparent_gap_limit_handling` database migration that drives the change in behavior.

…limit handling.

nuttycom marked this pull request as draft December 23, 2024 20:43

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 16 times, most recently from 62c1394 to bd2df86 Compare December 28, 2024 04:13

nuttycom changed the title ~~WIP: Add transparent address gap limit handling & general address rotation functionality.~~ Add transparent address gap limit handling & general address rotation functionality. Dec 28, 2024

nuttycom marked this pull request as ready for review December 28, 2024 04:14

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from bd2df86 to 4b99663 Compare December 28, 2024 04:27

nuttycom requested a review from str4d December 28, 2024 04:27

nuttycom mentioned this pull request Dec 30, 2024

Preparatory refactoring for zcash/librustzcash#1673 #1675

Merged

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from 1386dd1 to 49230de Compare December 30, 2024 22:20

str4d added a commit that referenced this pull request Dec 30, 2024

Merge pull request #1675 from nuttycom/feature/transparent_gap_limit_…

5651d80

…handling-prep Preparatory refactoring for #1673

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from b39c6c3 to 3ac0396 Compare December 30, 2024 22:33

str4d reviewed Dec 30, 2024

View reviewed changes

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from 3ac0396 to 24d611e Compare December 30, 2024 23:01

str4d reviewed Dec 30, 2024

View reviewed changes

zcash_client_sqlite/src/wallet.rs Outdated Show resolved Hide resolved

str4d reviewed Dec 30, 2024

View reviewed changes

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from 24d611e to 68e1864 Compare December 30, 2024 23:47

str4d reviewed Dec 30, 2024

View reviewed changes

str4d reviewed Dec 31, 2024

View reviewed changes

nuttycom force-pushed the feature/transparent_gap_limit_handling branch 2 times, most recently from ac23d07 to 81b05c1 Compare December 31, 2024 00:31

nuttycom added 2 commits December 30, 2024 21:04

zcash_client_sqlite: Add a test for external transparent address gap …

6238af4

…limit handling.

nuttycom force-pushed the feature/transparent_gap_limit_handling branch from 81b05c1 to 6238af4 Compare December 31, 2024 04:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add transparent address gap limit handling & general address rotation functionality. #1673

Add transparent address gap limit handling & general address rotation functionality. #1673

nuttycom commented Dec 23, 2024

codecov bot commented Dec 28, 2024 •

edited

Loading

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024 •

edited

Loading

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

str4d Dec 30, 2024

nuttycom Dec 31, 2024

str4d Dec 31, 2024

str4d Dec 31, 2024

-/// - `transparent_child_index`: the diversifier index, if it is in the range of a non-hardened
-///   transparent address index. This is used for gap limit handling and is always populated if the
-///   diversifier index is in that range; since the diversifier index is stored as a byte array we
-///   cannot use SQL integer operations on it and thus need it as an integer as well.
+/// - `transparent_child_index`: the diversifier index in integer form, if it is in the range of a  `u31`
+///   (i.e. a non-hardened transparent address index). It is used for gap limit handling, and is set
+///   whenever a transparent address at a given index should be scanned at receive time. This
+///   includes:
+///   - Unified Addresses with transparent receivers (at any valid index).
+///   - Unified Addresses without transparent receivers, but within the gap limit of potential
+///     sequential transparent addresses.
+///   - Transparent change addresses.
+///   - ZIP 320 ephemeral addresses.
+///   This column exists because the diversifier index is stored as a byte array, meaning that we
+///   cannot use SQL integer operations on it for gap limit calculations, and thus need it as an
+///   integer as well.

-/// - `exposed_at_height`: The chain tip height at the time that the address was generated by an
-///   explicit request by the user or reserved for use in a ZIP 320 transaction. In the case of an
-///   address with its first use discovered in a transaction obtained by scanning the chain, this
-///   will be set to the mined height of that transaction.
+/// - `exposed_at_height`: Our best knowledge as to when this address was first exposed to
+///   the wider ecosystem.
+///   - For user-generated addresses, this is the chain tip height at the time that the address
+///     was generated by an explicit request by the user or reserved for use in a ZIP 320
+///     transaction. These heights are not recoverable from chain.
+///   - In the case of an address with its first use discovered in a transaction obtained by
+///     scanning the chain, this will be set to the mined height of that transaction. In recover
+///     from seed cases, this is what user-generated addresses will be assigned.

Add transparent address gap limit handling & general address rotation functionality. #1673

Are you sure you want to change the base?

Add transparent address gap limit handling & general address rotation functionality. #1673

Conversation

nuttycom commented Dec 23, 2024

codecov bot commented Dec 28, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

str4d Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 28, 2024 •

edited

Loading

str4d Dec 30, 2024 •

edited

Loading