Durability and recovery

A node recovers from validated disk state, which can lag its latest executed or finalized result. The normal path overlaps execution with storage: a bounded writer makes account changes durable in order, then QMDB applies them. A successful RPC receipt is therefore not a substitute for the durable-height and storage acknowledgements.

This explanation answers one question: which completed barrier makes a block recoverable after a crash? Follow the height-100/101 example, then use the crash-position table to reason about a restore. For commands that inspect an actual host, use incident diagnosis. For the account model and different root types, use state ownership.

Walk one block to disk

Suppose QMDB has acknowledged height 100 and the reporter executes finalized block 101. The common commit constructs a deterministic account batch and updates the live execution view. It enqueues the batch to a separate FIFO durability writer.

That writer has a channel bound of sixteen batches, with blocking backpressure. It appends block 101's full account writes and deletions to the WAL and waits for the WAL durability operation. It then publishes the checksummed durable consensus height. Only after those steps does it forward the batch to the QMDB drain. A full queue slows execution; the bound is not a wall-clock recovery objective. Writer order and failure contract.

The QMDB drain checks ordering. Ordinary blocks must follow a completed height-zero bootstrap and advance contiguously. An authenticated full snapshot installation has an explicit replacement path that can jump ahead. After applying a batch, the drain waits for QMDB commit/flush, updates the known-key inventory when membership changes, and persists an applied marker containing both consensus height and the physical QMDB metadata sequence.

Only after those artifacts are durable may it rotate the WAL and return the root acknowledgement. This preserves the replay material until the persistent database and its key inventory are ready to stand on their own. Drain ordering, post-commit barriers.

Explore crash boundaries.

A crash is a question about the last completed barrier

These are proof requirements, not instructions to remove inconvenient files. A surviving WAL tail is not automatically accepted merely because its checksum passes. The height sidecar is the local recovery anchor; a record above it is unproven and rejected. Likewise, applying the same logical record again can change QMDB's physical serial/version history and root, so recovery uses the applied marker rather than assuming all replay is harmless. Recovery validation.

The implementation explicitly documents a residual exposure: finalized payloads beyond the durable position are not stored as a local executable payload log. A node missing that suffix must state-sync from peers before voting on a mismatched recovery anchor. A simultaneous quorum crash inside the deferred window is therefore a different failure from one lagging disk recovering against healthy peers. The code does not establish zero data loss across that event.

Fail closed when a barrier cannot complete

A WAL append error, durable-height publication error or disconnected drain sets the shared halt state. The affected root acknowledgement is dropped. The deferred Merkle handle reports failure and later commits refuse to advance.

A QMDB post-commit failure is also fatal even if the database has already flushed some work. The runtime retains WAL evidence instead of acknowledging a later state and risking loss of the only replay record. Recovery must reconcile the database, applied marker and retained history as a set.

For operators, a drain halt means recent client observations need to be interpreted against the recovered durable state. Capture logs, release identity, epoch and the complete data directory before intervention. Repeatedly restarting or deleting the marker is not a repair procedure. The deployment guide explains why a cutover failure after storage opens may require manual recovery rather than automatic binary rollback.

Startup proves a complete bridge before mutation

The production node requires persistent QMDB. When a height sidecar exists, it first probes the database in a fresh process, then attempts the full open. Either an unsuccessful probe or a later open failure can enter validated snapshot/WAL recovery. If that recovery fails, startup exits instead of serving a new in-memory chain.

When the height sidecar is missing, startup inventories persistent artifacts. It creates a fresh store only when it can establish that the directory is pristine. Existing WAL, snapshot or database artifacts without their height anchor cause a fatal error. This prevents a damaged node from silently bootstrapping over its previous history. Startup decisions.

Strict WAL validation scans the active and old files, checks record framing and checksums, reconciles duplicate heights, and validates order. A recovery bridge must begin immediately after the applied height and reach the durable tip without skipping ordinary blocks. Even an empty block matters to that continuity. The only allowed jump is an explicitly flagged full-state snapshot installation. Validation runs before mutating QMDB or its applied marker. Complete bridge check.

The WAL reader supports v1 write-only records, v2 writes plus deletions, and current v3 flags including snapshot installation. This is WAL-format compatibility; it does not establish that any two node releases can safely share consensus or storage. WAL framing.

A snapshot depends on more than its account bytes

An account snapshot carries a metadata record binding height, exact byte length and a BLAKE3 digest of metadata plus account bytes. The loader rejects corrupt metadata, impossible counts, truncation, trailing bytes and duplicate public keys before inserting the accounts. Height-gated recovery also rejects a stale, newer-than-expected or legacy unanchored snapshot.

Here “authenticated snapshot” describes the runtime's validated binding to its recovery contract. A local checksum alone does not prove that an unknown archive was approved by consensus. Establish the trusted chain, source and height separately when choosing a restore point. Snapshot validation, failure tests.

Keep the entire durability set together: QMDB data, active and retained WAL, durable height, applied marker, known keys, account snapshot and metadata, consensus/epoch files, and required large-account blobs. QMDB manifests can reference content-addressed files under large_accounts; a database copy without those files cannot reconstruct the account. The adapter verifies blob length and hash and caps account data at 10 MiB. Large-account storage.

Receipts, replay-protection records, query indexes and HILSA attestations have separate persistence behavior. Do not assume that an account snapshot preserves every historical RPC answer. Recovered signature records above the durable height are discarded so the replay view does not advertise an unrecovered suffix.

Shutdown and restored-state checks complete the recovery boundary

On SIGTERM or Ctrl+C, the node shuts down the drain, attempts an account snapshot, and flushes collected HILSA attestations. If snapshot creation fails, it keeps the previous authenticated snapshot and logs the failure. A graceful stop is a useful barrier only when its outcome is checked. Shutdown sequence.

After recovery, establish the expected release, chain identity, epoch and durable height. Compare a recorded canonical account hash at a common height with an authoritative peer, then let the node catch up before restoring its role. The QMDB root, reporter cumulative root and full account-map hash have different meanings; comparing different root types or heights does not test agreement. The validator lifecycle explains voting and state-sync gates.

The repository has focused tests for off-thread ordered WAL/height publication, writer failure and disconnected acknowledgements, repeated height-zero restarts, interrupted snapshot installs, WAL rotation and atomic snapshot rejection. These provide regression evidence for selected failure boundaries. They were inspected during this documentation run; no crash injection or full Cargo suite was executed. A deployment-specific restore rehearsal remains the way to establish that its captured files, binary and operational procedure work together.