Observability, incidents and recovery

A node is useful when it accepts valid work, executes it, agrees with its peers and preserves the resulting state. Monitor those properties separately. An HTTP response proves reachability; an advancing height proves progress; neither establishes account-state agreement or that the latest result is on disk.

For an incident, first identify the deployment and capture its evidence. The AWS launch template runs the system eto-node.service with data under /opt/eto/data; legacy single-host scripts use user services such as eto-seq.service and per-node directories below /home/naman/eto/data. Do not apply a restore command from one topology to the other. See the launch unit.

Starting condition: you have an unhealthy endpoint or node and access to its approved topology, service identity and logs. Work through the bounded diagnostics, compare peers only at a common height, then choose containment from the observed failure. A useful result is a timestamped evidence set and a justified recovery decision; these probes do not themselves restore service.

Read the signals together

The RPC health implementation checks the persistent-store drain halt first. It then tracks whether the observed height advances, with a default thirty-second stall threshold. The first probe seeds the observation; a single request cannot establish sustained progress. Setting ETO_HEALTH_STALL_SECS=0 disables the stall check, not the drain-halt check. JSON-RPC failure is an error object despite HTTP 200; the runtime HTTP health route uses an unhealthy HTTP status. See health semantics.

This distinction mattered in the documentation checks: the public endpoint answered reads on 7–8 September 2026 UTC, while getHealth returned -32005 for a halted persistent-store drain and sampled height remained unchanged. That is a dated observation, not a standing description of the service. Check availability before attempting a live workflow.

Capture a bounded local diagnostic set

On a host running the AWS service layout, these are read-only probes. Adapt the unit and ports to the installed configuration:

systemctl status eto-node.service --no-pager
journalctl -u eto-node.service --since '-10 min' --no-pager -n 200
df -h /opt/eto/data
curl --max-time 5 -sS http://127.0.0.1:8899/health
curl --max-time 5 -sS http://127.0.0.1:8899 \
  -H 'content-type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"getHealth","params":[]}'
curl --max-time 5 -sS http://127.0.0.1:8899 \
  -H 'content-type: application/json' \
  --data '{"jsonrpc":"2.0","id":2,"method":"getBlockHeight","params":[]}'

Record the response bodies, status codes, observation times, running version and source revision. Repeat the height probe after a meaningful interval rather than interpreting one sample as progress. A service status may exit nonzero because the service is unhealthy; preserve that output.

The metrics port is configurable. The AWS template sets ETO_METRICS_PORT=19090, while the runtime default is 9090. Read the installed nonsecret effective configuration before assuming either one. The configuration guide explains --print-effective-config and scripts/eto-ops inventory and audit operations. Neither a configuration bundle nor a matching config hash approves a release.

These example host commands were inspected, not executed against a fleet during this run.

Compare state at the same height

eto_canonicalStateHash accepts an optional positional height:

{"jsonrpc":"2.0","id":3,"method":"eto_canonicalStateHash","params":[12345]}

Here 12345 is illustrative: choose a height both nodes have recorded. A result has this shape:

{"height":12345,"hash":"0x<64 hexadecimal characters>"}

Equal hashes at the same height indicate agreement on the committed account-map fields. Unequal latest hashes at different heights are not evidence of divergence. A null result means no recorded sample exists on that node; it does not mean empty state. The endpoint serves stored samples instead of recomputing a sorted full-account hash for every unauthenticated request. See the handler.

Metrics expose eto_canonical_state_hash_lo and eto_canonical_state_hash_height as a lightweight alarm signal. The former is only the leading eight bytes, rendered through monitoring infrastructure; use the full RPC hash for an investigation. Isolate a disagreeing read replica from public reads while establishing the authoritative history. A responsive replica can still serve wrong state.

Monitoring needs installed detectors and delivery

The runtime exports height, recent throughput, ingest accept/reject counts, finality timing, finalized/proposed counters and the eto_merkle_halt_total latch indicator. The most useful first correlation is whether submissions rise, finalized blocks advance and the store remains healthy. A throughput metric by itself cannot explain whether transactions failed or whether a client observed success.

The checked-in Prometheus rules cover stalled height, missing finalization, merkle halt and other conditions. Treat rule thresholds as operational configuration, not measured service guarantees. The sample scrape configuration contains static hosts and roles; it is not an inventory of the current fleet.

Disk rules require node_exporter, and GPU-memory rules require NVIDIA DCGM metrics. The checked-in Prometheus configuration does not add those exporters. Conversely, older rule comments saying finality metrics have not landed are stale: the current renderer emits eto_finality_ms. Inspect the metric renderer and scrape configuration.

Two additional scripts have different effects:

• eto-health-poll.sh samples RPC heights and maintains per-condition state. Its default stall window is thirty seconds and repeat-page interval is 1,200 seconds. If ETO_PAGE_CMD is absent or invalid, it warns that no page will be delivered; a detector in the journal is not an alert received by an operator.
• eto-monitor.sh includes an end-to-end faucet/balance canary and can invoke the pager and cloud log inspection. It performs writes and is not a harmless status command. Its execution canary is useful precisely because consensus height can advance while an application path fails.

Validate detector firing and notification delivery separately in the intended environment. Neither script was run during documentation generation. See the poller and monitor.

Choose containment from the failure

For a multi-node stop, use the coordinated deployment ordering. Stopping the sequencer first can leave validators durably ahead. The coordinator also checks matching epoch files and refuses unproven recovery. A missing epoch file is not a reason to invent one or reset it to zero.

Back up a recoverable state, then rehearse restoration

Preserve the complete node data directory and its associated configuration, release identity and consensus identity reference. The durability set includes the QMDB/WAL state, height and applied markers, key inventory, account snapshot and metadata, consensus files, epoch and large-account blobs. A copied account snapshot without the data required to recover it is not a complete restore point.

The legacy eto-backup.sh archives each node directory, optionally using zstd, and retains hourly and daily sets. It does not stop the node or create a storage snapshot before reading live files. Therefore a successful archive command does not establish a transactionally consistent recovery point. Its textual height manifest is not a proof of the runtime's checksummed durable height. Use a stopped or otherwise consistent capture and an actual restore rehearsal for that assurance.

The legacy eto-restore.sh checks that its user service is inactive, moves the existing directory aside, extracts the chosen archive and leaves startup to the operator. It does not implement the AWS fleet cutover procedure, select a trusted chain history or prove restored account agreement. Its recovery-time target in comments is a target, not a measured result. See backup and restore.

A rehearsal should establish that the selected binary opens the captured storage, recovers the expected height and epoch, agrees with an authoritative peer at a common height, catches up, and executes an appropriate canary. Keep the original data until those checks pass. The next chapter, durability and recovery, explains which writes survive a crash and why a finalized suffix may still need peers.

Confirm the outcome before closing the incident

Retain the failing and recovered observations, the selected authoritative height/hash, the running release/configuration identity and the action taken. Close the incident only after the relevant properties recover: advancing healthy service, same-height state agreement where applicable, and a successful authorized canary for the affected application path. If restoration has not been rehearsed or a required peer cannot be compared, record that gap and keep the recovery decision open. Use the transaction-inspection task to interpret a saved canary signature without resubmitting it.