> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockops.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Continuity and recovery

> What is backed up, how a failed signing node is restored without exposing key material, and how the platform behaves during partial failure.

The platform assumes that machines fail, networks partition, workers restart and receipts go missing. Each layer keeps enough durable state to recover without guessing, and no recovery step requires anyone to handle a complete private key.

## Losing a signing node

A two-of-three cluster keeps signing with one node down; the loss is reduced redundancy, not an outage. Restoring the node:

1. Re-provision the machine or pod.
2. Restore the node's identity from the approved secret store.
3. Restore the encrypted share store from backup.
4. Supply the store's passphrase from the runtime secret store.
5. Start the node; it registers as ready and rejoins signing sessions.

At no point is a share decrypted outside the node, and the person performing the recovery never sees key material.

For **planned** replacement (retiring a machine, moving to a new substrate, or adding an institution-operated node) the cluster is reshared to the new set of nodes instead. Resharing preserves the wallet's public key and every address; nothing moves on chain. See [Threshold signing](/security/threshold-signing).

## What is backed up

| Component                              | Backup                                                                                 | Notes                                                                                                                       |
| :------------------------------------- | :------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------- |
| **Signing node share store**           | Encrypted backups written on a schedule to storage appropriate to the node's substrate | Encrypted at the application layer before any storage-level encryption. A backup alone cannot sign or reconstruct a wallet. |
| **Node identities and initiator keys** | Held in the secret store                                                               | Rotated under separate controls from the share stores.                                                                      |
| **Operation ledger (Postgres)**        | Regular backups with point-in-time recovery                                            | Contains no key material.                                                                                                   |
| **Messaging state**                    | Durable streams                                                                        | If lost, in-flight operations are reconciled from the ledger and the chain before publishing resumes.                       |

The chain code used for address derivation is never regenerated for existing wallets; derived addresses depend on it.

## Recovering the control plane

The API is restored from the ledger backup and brought up with write paths disabled. Operators then review non-terminal operations (wallet creations, signing requests and withdrawals that were in flight) and reconcile any submitted withdrawal against the chain before re-enabling result consumption, broadcast and confirmation. Idempotency records are preserved: repeating a request with the same key returns the existing operation rather than creating a second one.

## Partial failure during normal operation

| Situation                                                 | Behaviour                                                                                                                                                        |
| :-------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A key-generation or signing request receives no result    | It is marked failed after a configured timeout. A result that arrives later is ignored; it cannot revive a failed operation.                                     |
| A signed transaction was broadcast but no receipt appears | The withdrawal stays *submitted*. The same signed transaction can be rebroadcast; a receipt below the confirmation depth is never treated as final or as failed. |
| The chain reorganises                                     | Deposits in the affected blocks are marked failed, the scanner rewinds, and canonical blocks are rescanned.                                                      |
| The messaging plane is unreachable                        | The API reports not-ready and refuses new commands rather than accepting work that cannot complete.                                                              |
| The kill switch state cannot be read                      | Withdrawals are blocked. The check fails closed.                                                                                                                 |

## Operator visibility

Operators can list stuck operations (stale key generations, signing requests and submitted withdrawals) through an admin-only view, and every operation carries the identifiers needed to trace it: request ID, workspace, wallet, withdrawal, signing request, cluster correlation ID and transaction hash.