State ownership, roots, and persistent storage

Singularity uses a shared account model with several views and persistence stages. The hot account map is the execution view; a transaction journal is speculative state; the WAL and durable-height record define recoverable local progress; QMDB stores the persistent account representation. These names describe different responsibilities, not interchangeable copies of a single “committed” bit.

This separation permits execution to overlap storage work. It also creates a bounded interval in which a locally executed block is ahead of durable recovery state.

For crash scenarios, continue with recovery barriers; for commands on an affected host, use incident diagnosis.

Read through a view; write through a changeset

VM adapters read through AccountStateView, whose interface contains account lookup and current height, without mutation. The same adapter can therefore read committed state, an in-block overlay, or the transaction's pending writes. Requiring the concrete base store for every nested read would bypass read-your-writes semantics.

Writes leave through StateChangeSet, which represents updates and deletions. The enclosing transaction controls their lifetime; the common state commit controls publication. An adapter must not write directly to the hot map merely because it has finished its own child call. Read boundary, transaction journal.

Deletion is a first-class effect. A missing write and an explicit tombstone cannot mean the same thing: replication, live-key enumeration, snapshots, and root calculation all need to know that an account was removed.

The ordinary commit path

The state layer serializes commit and snapshot transitions with a shared lock. It checks for closed commits, staged-account failures, an incomplete persistence topology, a halted drain, and missing durable bootstrap before accepting an ordinary persistent commit. WAL, data directory, and drain must be configured together. Commit preconditions.

It combines explicit writes with staged transfer state and deletions, sorts keys, and resolves duplicate writes deterministically. Stable sorting matters when equal keys exist because the fold must preserve the original last writer. When no merge is needed, one shared account vector can feed live-map application, WAL, QMDB, and the reporter's diffs without cloning the account data repeatedly.

In the normal persistent runtime, application and enqueueing occur before the separate writer completes disk durability. In configurations without that writer, the WAL/height barrier remains inline. Bootstrap and snapshot installation have explicit paths rather than masquerading as ordinary contiguous blocks. Batch construction and barrier choice.

Interactive persistence flow

Durability has watermarks

The writer's channel has a bound of 16 batches and blocking backpressure. For each ordinary batch it appends the WAL record, publishes the durable-height sidecar, then forwards the batch to the QMDB drain. A write failure latches the shared halt and drops the root acknowledgement; a later commit must refuse to advance. Writer contract.

A crash inside the deferred interval recovers to the durable position. The finalized suffix beyond it is not available as a local log of executable finalized payloads; recovery needs peers/state sync. A simultaneous crash of a quorum within that interval is a residual durability exposure explicitly described by the implementation. “Consensus finalized” therefore must not be documented as “every validator has fsynced this state.” The recovery workflow turns these watermarks into operational decisions.

Roots answer different questions

The reporter's cumulative root sorts diffs by public key, commits full-account bytes and metadata, gives lamport-only updates and deletions distinct domain markers, and folds the diff hash with the previous root and height. A deletion-only block affects this commitment. For a completely empty diff set, the function carries forward the previous root.

That is not a Merkle inclusion proof for an arbitrary account in QMDB. QMDB's root belongs to its persistent data structure and can arrive later. The in-memory fallback root is another bounded implementation path, hashing the supplied account diffs canonically; it must not be relabeled as the cumulative chain root. Cumulative root, fallback hash.

When comparing nodes, record the root type and height together. Comparing a current cumulative root to a lagging persistent root is not a divergence test.

WAL and recovery framing

The block WAL carries full account writes and explicit deletions. Current v3 records add authenticated flags for a complete snapshot installation, which may deliberately jump height. Older v1 write-only and v2 write/deletion records remain readable.

Before mutating recovery files, strict validation scans both authoritative WAL files, checks duplicate-height agreement, and returns records in height order. Ordinary gaps must not be mistaken for an intentional snapshot jump. Rotation keeps the old file until the required durability condition is met. WAL format and strict read.

A backup therefore needs a consistent set of account data, WAL history, height/applied metadata, and required sidecars. Copying an arbitrary live database directory without respecting those relationships is not the same as creating a validated checkpoint.

Large accounts and query material

QMDB's inline account representation has a 53-byte header. Serialized values up to 60 KiB stay inline; larger accounts use a content-addressed blob under large_accounts and a manifest containing its length and BLAKE3 hash. The decoder checks both before accepting the account. Account data is capped at 10 MiB by this storage adapter.

The blob is part of the recoverable state. Keeping a QMDB manifest while omitting its referenced file does not preserve the account. Large-account encoding.

Receipts, transaction indexes, block timestamps, replay-protection records, and HILSA attestations have related but distinct owners. An account snapshot is not evidence that every query index or collected signature has the same retention and durability guarantee. Follow the transaction lifecycle for receipt publication and HILSA for certificate persistence.

Regression evidence

Relevant tests exercise full-field root changes, deletion semantics, WAL rotation and restart, large-programdata sidecar round trips, snapshot authentication, and durable-height consistency. The focused wal_rotation_differential integration test compares rotation/recovery behavior; large-account tests live beside the storage adapter.

These tests protect particular invariants. They do not establish that an arbitrary host disk or a live fleet has survived a crash. Operators should use incident diagnosis and validate the recovered state before restoring traffic.