[rpc] Fatal `getSignaturesForAddress()` when Bigtable errors #3700

steveluscher · 2024-11-19T01:20:20Z

Problem

Consider a request to getSignaturesForAddress(). Imagine that there are no signatures in blockstore, but there are signatures in long-term storage (ie. Bigtable).

Currently, if we fail to reach Bigtable – because of a timeout or a connection failure – we return whatever signatures we have. This means that people who query the RPC can't distinguish between the following cases:

No signatures were found because there are none, or
No signatures were found because Bigtable was temporarily unavailable

Summary of Changes

When doing a range query in get_confirmed_signatures_for_address and the before/until can't be found, throw a SignatureNotFound error instead of RowNotFound.
Now that we can match on actual connection errors – separate from SignatureNotFound errors – return a JSON-RPC error in the event that long-term storage errors out.

Test Plan

With the Bigtable emulator

> gcloud beta emulators bigtable start
> ./init-bigtable.sh

> tail -f validator.log | grep [Bb]ig[Tt]able
[2024-11-19T23:06:05.994676924Z INFO  solana_rpc::rpc_service] rpc configuration: JsonRpcConfig { enable_rpc_transaction_history: true, enable_extended_tx_metadata_storage: true, faucet_addr: Some(0.0.0.0:9900), health_check_slot_distance: 0, skip_preflight_health_check: false, rpc_bigtable_config: Some(RpcBigtableConfig { enable_bigtable_ledger_upload: false, bigtable_instance_name: "solana-ledger", bigtable_app_profile_id: "default", timeout: None, max_message_size: 67108864 }), max_multiple_accounts: None, account_indexes: AccountSecondaryIndexes { keys: None, indexes: {} }, rpc_threads: 0, rpc_niceness_adj: 0, full_api: true, rpc_scan_and_fix_roots: false, max_request_body_size: None, disable_health_check: true }
[2024-11-19T23:06:05.995145895Z INFO  solana_storage_bigtable::bigtable] Connecting to bigtable emulator at localhost:8086
[2024-11-19T23:06:05.995411312Z INFO  solana_rpc::rpc_service] BigTable ledger storage initialized

Fetching a signature for an address that does not exist locally

Observe that the RPC goes out to Bigtable and fetches successfully (finds nothing)

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ"
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature for an address and before signature that don't exist locally

Observe that Bigtable can't find the before signature, but doesn't fatal the request

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ",
    {"before":"31AUDAUXgD4B5DqqtFZZe8udgHUfbJX9dQbiXQGmNiRqeKRkzRTYitrRbJDtpt4DMd4P3G8haaXMJ8TU6wwQmf3h"}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature for an address that is completely available locally

Observe Bigtable is never contacted.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[{"blockTime":1732058393,"confirmationStatus":"finalized","err":null,"memo":null,"signature":"3mZUQjhaGA7ytwb3g9PnjYL4d1ErsWf7nheMCqnSuwW6x2hF8tx4DuuvG1u7j6qVCC2J9JqpRHbJ7z57ayhJZyW2","slot":9284}],"id":"1"}

Fetching a signature for an address that does exist but with a bad `before` signature

Observe that Bigtable can't find the before signature, but doesn't fatal the request

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"before":"419mQJ2ZpM1uJ2VJyHckWNzmEpRKkyuirvRQH9NfgKvzQJxTcTa5v8U7tXJ4VkqEH8GsTtG7iqSsQBJe8VxxC2XQ","limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

Observe that the RPC goes out to Bigtable and fetches successfully (finds nothing)

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {
      "before":"45z1V6qdGUXRHid4YDzSSqV13pegyP2naBaA3aG4ZoS4xMdKjC3eBvhwTkbXrqEXJT38grfRJ11wCTn5Qop67AEv",
      "limit": 1
    }
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":[],"id":"1"}

With the Bigtable emulator shut down, simulating a connection failure

Fetching a signature for an address that does not exist locally

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ"
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature for an address and before signature that don't exist locally

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "BzszvNFvokG4v4qb1ipYSDGtKuhen8PMtYjevfZeAkKZ",
    {"before":"31AUDAUXgD4B5DqqtFZZe8udgHUfbJX9dQbiXQGmNiRqeKRkzRTYitrRbJDtpt4DMd4P3G8haaXMJ8TU6wwQmf3h"}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature for an address that is completely available locally

Observe Bigtable is never contacted.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","result":{"jsonrpc":"2.0","result":[{"blockTime":1732058393,"confirmationStatus":"finalized","err":null,"memo":null,"signature":"3mZUQjhaGA7ytwb3g9PnjYL4d1ErsWf7nheMCqnSuwW6x2hF8tx4DuuvG1u7j6qVCC2J9JqpRHbJ7z57ayhJZyW2","slot":9284}],"id":"1"}

Fetching a signature for an address that does exist but with a bad `before` signature

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {"before":"419mQJ2ZpM1uJ2VJyHckWNzmEpRKkyuirvRQH9NfgKvzQJxTcTa5v8U7tXJ4VkqEH8GsTtG7iqSsQBJe8VxxC2XQ","limit": 1}
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

Observe that Bigtable experiences a connection error.

curl localhost:8899 -d '{
  "id": "1",
  "jsonrpc": "2.0",
  "method": "getSignaturesForAddress",
  "params": [
    "jZ2L6sHvF862QMKYxD7ho7JQAX5m1rT9VzVKvzSSbin",
    {
      "before":"45z1V6qdGUXRHid4YDzSSqV13pegyP2naBaA3aG4ZoS4xMdKjC3eBvhwTkbXrqEXJT38grfRJ11wCTn5Qop67AEv",
      "limit": 1
    }
  ]
}' --header 'Content-Type: application/json'
{"jsonrpc":"2.0","error":{"code":-32019,"message":"Failed to query long-term storage; please try again"},"id":"1"}

Fixes #3696

…able) can't be reached

mergify · 2024-11-19T01:20:54Z

If this PR represents a change to the public RPC API:

Make sure it includes a complementary update to rpc-client/ (example)
Open a follow-up PR to update the JavaScript client @solana/web3.js (example)

Thank you for keeping the RPC clients in sync with the server API @steveluscher.

steveluscher · 2024-11-19T01:22:34Z

How would I go about testing/mocking a Bigtable failure. I wanted to at least write something like this:

#[test]
fn test_signatures_for_address_blockstore_query_failure() {
    let rpc = RpcHandler::start_with_config(JsonRpcConfig {
        enable_rpc_transaction_history: true,
        rpc_bigtable_config: # ???
        ..Default::default()
    });
    let pubkey = Pubkey::new_unique();
    // This address is guaranteed to have no signatures, thereby forcing
    // `getSignaturesForAddress` to go looking in long-term storage.
    let address = pubkey.to_string();
    let request = create_test_request(
        "getSignaturesForAddress",
        Some(json!([address, {"limit": 10}])),
    );
    let (code, _message) = parse_failure_response(rpc.handle_request_sync(request));

    assert_eq!(code, JSON_RPC_SERVER_ERROR_LONG_TERM_STORAGE_UNREACHABLE);
}

…but I can't figure out how to supply ‘a bigtable instance that will always fail.’

…NotFound` instead of `RowNotFound`

…ut can't be reached

steveluscher · 2024-11-19T23:43:55Z

rpc/src/rpc.rs

+        if !self.config.enable_rpc_transaction_history {
+            return Err(RpcCustomError::TransactionHistoryNotAvailable.into());
+        }


This is the meat of the unindenting. All of the code below is unchanged.

steveluscher · 2024-11-19T23:44:25Z

rpc-client-api/src/custom_error.rs

@@ -26,6 +26,7 @@ pub const JSON_RPC_SERVER_ERROR_UNSUPPORTED_TRANSACTION_VERSION: i64 = -32015;
 pub const JSON_RPC_SERVER_ERROR_MIN_CONTEXT_SLOT_NOT_REACHED: i64 = -32016;
 pub const JSON_RPC_SERVER_ERROR_EPOCH_REWARDS_PERIOD_ACTIVE: i64 = -32017;
 pub const JSON_RPC_SERVER_ERROR_SLOT_NOT_EPOCH_BOUNDARY: i64 = -32018;
+pub const JSON_RPC_SERVER_ERROR_LONG_TERM_STORAGE_UNREACHABLE: i64 = -32019;


Will dutifully add this to @solana/web3.js once landed.

steveluscher · 2024-11-19T23:44:48Z

rpc-client-api/src/custom_error.rs

@@ -244,6 +247,11 @@ impl From<RpcCustomError> for Error {
                ),
                data: None,
            },
+            RpcCustomError::LongTermStorageUnreachable => Self {
+                code: ErrorCode::ServerError(JSON_RPC_SERVER_ERROR_LONG_TERM_STORAGE_UNREACHABLE),
+                message: "Failed to query long-term storage; please try again".to_string(),


I figured I'd say ‘please try again’ to indicate that the failure is transient.

steveluscher · 2024-11-19T23:47:00Z

storage-bigtable/src/lib.rs

+                    .map_err(|err| match err {
+                        bigtable::Error::RowNotFound => Error::SignatureNotFound,
+                        _ => err.into(),


Something similar was added to get_signature_status in 71e9958 but we could use it here, to disambiguate between ‘couldn't find the thing’ and ‘o no bigtable down bad.’

steveluscher · 2024-11-19T23:47:45Z

rpc/src/rpc.rs

@@ -1761,8 +1761,8 @@ impl JsonRpcRequestProcessor {
                            bigtable_before = None;
                        }
                        Err(err) => {
-                            warn!("{:?}", err);
-                            return Ok(map_results(results));


Previously, this code would catch all manner of Bigtable failure, including connection failures, and return whatever results we have so far.

steveluscher · 2024-11-19T23:48:52Z

rpc/src/rpc.rs

-                            warn!("{:?}", err);
-                            return Ok(map_results(results));
+                            warn!("Failed to query Bigtable: {:?}", err);
+                            return Err(RpcCustomError::LongTermStorageUnreachable.into());


Now that storage_bigtable/lib.rs properly throws SignatureNotFound errors, we can be sure that when this arm is matched, it's because of a Bigtable connection error.

steveluscher · 2024-11-19T23:49:25Z

rpc/src/rpc.rs

@@ -1791,8 +1791,10 @@ impl JsonRpcRequestProcessor {
                            }
                        }
                    }
+                    Err(StorageError::SignatureNotFound) => {}


A failure to find the before or until signatures continues to behave like the old code: skip and move on.

steveluscher · 2024-11-19T23:49:38Z

rpc/src/rpc.rs

                    Err(err) => {
-                        warn!("{:?}", err);
+                        warn!("Failed to query Bigtable: {:?}", err);
+                        return Err(RpcCustomError::LongTermStorageUnreachable.into());


Now that storage_bigtable/lib.rs properly throws SignatureNotFound errors, we can be sure that when this arm is matched, it's because of a Bigtable connection error.

godmodegalactus

LGTM

KirillLykov · 2024-11-22T09:13:10Z

rpc/src/rpc.rs

+                .map(|x| {
+                    let mut item: RpcConfirmedTransactionStatusWithSignature = x.into();
+                    if item.slot <= highest_super_majority_root {
+                        item.confirmation_status = Some(TransactionConfirmationStatus::Finalized);


looks like there was wrong indentation, weird that cargo fmt didn't complain on the old code.

KirillLykov

I looked through the code, looks like a really good change to me.

mergify · 2025-01-13T19:29:49Z

Backports to the stable branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule.

mergify · 2025-01-13T19:29:51Z

Backports to the beta branch are to be avoided unless absolutely necessary for fixing bugs, security issues, and perf regressions. Changes intended for backport should be structured such that a minimum effective diff can be committed separately from any refactoring, plumbing, cleanup, etc that are not strictly necessary to achieve the goal. Any of the latter should go only into master and ride the normal stabilization schedule. Exceptions include CI/metrics changes, CLI improvements and documentation updates on a case by case basis.

* Unindent code in `get_signatures_for_address` * Add a custom JSON-RPC error to throw when long-term storage (ie. Bigtable) can't be reached * When the `before`/`until` signatures can't be found, throw `SignatureNotFound` instead of `RowNotFound` * Fatal `getSignaturesForAddress` calls when Bigtable must be queried but can't be reached (cherry picked from commit 52f132c)

…ackport of #3700) (#4443) [rpc] Fatal `getSignaturesForAddress()` when Bigtable errors (#3700) * Unindent code in `get_signatures_for_address` * Add a custom JSON-RPC error to throw when long-term storage (ie. Bigtable) can't be reached * When the `before`/`until` signatures can't be found, throw `SignatureNotFound` instead of `RowNotFound` * Fatal `getSignaturesForAddress` calls when Bigtable must be queried but can't be reached (cherry picked from commit 52f132c) Co-authored-by: Steven Luscher <[email protected]>

…ackport of #3700) (#4442) [rpc] Fatal `getSignaturesForAddress()` when Bigtable errors (#3700) * Unindent code in `get_signatures_for_address` * Add a custom JSON-RPC error to throw when long-term storage (ie. Bigtable) can't be reached * When the `before`/`until` signatures can't be found, throw `SignatureNotFound` instead of `RowNotFound` * Fatal `getSignaturesForAddress` calls when Bigtable must be queried but can't be reached (cherry picked from commit 52f132c) Co-authored-by: Steven Luscher <[email protected]>

steveluscher added 2 commits November 19, 2024 00:30

Unindent code in get_signatures_for_address

e1fa697

Add a custom JSON-RPC error to throw when long-term storage (ie. Bigt…

3dd5947

…able) can't be reached

steveluscher changed the title ~~Fatal when there is no bigtable~~ [rpc] Fatal getSignaturesForAddress() when Bigtable errors Nov 19, 2024

steveluscher added 2 commits November 19, 2024 14:51

When the before/until signatures can't be found, throw `Signature…

f2ca0d0

…NotFound` instead of `RowNotFound`

Fatal getSignaturesForAddress calls when Bigtable must be queried b…

054273e

…ut can't be reached

steveluscher commented Nov 19, 2024

View reviewed changes

steveluscher marked this pull request as ready for review November 19, 2024 23:49

godmodegalactus approved these changes Nov 20, 2024

View reviewed changes

steveluscher added the rust Pull requests that update Rust code label Nov 20, 2024

KirillLykov reviewed Nov 22, 2024

View reviewed changes

KirillLykov approved these changes Nov 22, 2024

View reviewed changes

steveluscher merged commit 52f132c into anza-xyz:master Nov 25, 2024
42 checks passed

steveluscher deleted the fatal-when-there-is-no-bigtable branch November 25, 2024 17:45

steveluscher added v2.0 Backport to v2.0 branch v2.1 Backport to v2.1 branch labels Jan 13, 2025

This was referenced Jan 13, 2025

v2.0: [rpc] Fatal getSignaturesForAddress() when Bigtable errors (backport of #3700) #4442

Merged

v2.1: [rpc] Fatal getSignaturesForAddress() when Bigtable errors (backport of #3700) #4443

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[rpc] Fatal `getSignaturesForAddress()` when Bigtable errors #3700

[rpc] Fatal `getSignaturesForAddress()` when Bigtable errors #3700

steveluscher commented Nov 19, 2024 •

edited

Loading

mergify bot commented Nov 19, 2024

steveluscher commented Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

steveluscher Nov 19, 2024

godmodegalactus left a comment

KirillLykov Nov 22, 2024

KirillLykov left a comment

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

[rpc] Fatal getSignaturesForAddress() when Bigtable errors #3700

[rpc] Fatal getSignaturesForAddress() when Bigtable errors #3700

Conversation

steveluscher commented Nov 19, 2024 • edited Loading

Problem

Summary of Changes

Test Plan

With the Bigtable emulator

Fetching a signature for an address that does not exist locally

Fetching a signature for an address and before signature that don't exist locally

Fetching a signature for an address that is completely available locally

Fetching a signature for an address that does exist but with a bad before signature

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

With the Bigtable emulator shut down, simulating a connection failure

Fetching a signature for an address that does not exist locally

Fetching a signature for an address and before signature that don't exist locally

Fetching a signature for an address that is completely available locally

Fetching a signature for an address that does exist but with a bad before signature

Fetching a signature before the last signature, forcing the RPC to go to Bigtable where there is no data

mergify bot commented Nov 19, 2024

steveluscher commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

godmodegalactus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KirillLykov left a comment

Choose a reason for hiding this comment

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

[rpc] Fatal `getSignaturesForAddress()` when Bigtable errors #3700

[rpc] Fatal `getSignaturesForAddress()` when Bigtable errors #3700

steveluscher commented Nov 19, 2024 •

edited

Loading

Fetching a signature for an address that does exist but with a bad `before` signature

Fetching a signature for an address that does exist but with a bad `before` signature