Run a node · Explanation
Deployments and upgrades
A node upgrade changes both executable behavior and the code that opens existing chain state. Its safety therefore depends on the exact binary/CuEVM pair, fleet configuration and durable recovery point. A build establishes an artifact; a completed rollout must also establish compatible execution, preserved identity and state, healthy peer agreement and advancing public service.
The repository's testnet deployment path defaults to a coordinated cutover. It closes the public proxy, stops validators before the sequencer, verifies their stored state, installs the release, and proves recovery before restoring traffic. It does this even for same-protocol releases. Protocol equality has historically been insufficient evidence that two execution kernels can coexist.
This chapter explains why the checked-in cutover has this ordering and where rollback stops being safe. Use the linked deployment contract and environment-specific release machinery to perform an authorized rollout. It does not claim that a rollout was executed for this documentation. Start with build and configuration for a local binary, or incidents and recovery for a fleet that is already unhealthy.
Release, configuration and chain identity have separate owners
The deployment contract gives each kind of state an owner:
| Object | Owner and release behavior |
|---|---|
svm-vm, lib/libcuevm_state.so, release.json | One release artifact; the binary and shared library travel together |
| Release and rollback directories, prepared/deployed SHA markers | Deployment tooling |
launch.env | Provisioning owns it; deployment may update only its documented runtime library/GPU settings |
bridge.env | An explicit bridge-configuration operation owns it; ordinary rollout verifies the expected content |
node.toml, genesis | Topology and launch ceremony; ordinary binary deployment does not regenerate them |
| Consensus identity key | Operator-managed secret; preserve it across releases |
| Account state, QMDB, epoch and consensus storage | Chain state; preserve and verify it, rather than treating it as an installation cache |
| systemd units and drop-ins | Provisioning; deployment verifies their expected contents |
The release archive has exactly three members. Its S3 location includes the target source SHA and artifact/script hashes; the current-release pointer is the intentionally mutable object. This separates “which release was approved” from “which release a pointer currently selects.”
Genesis files also have distinct roles. The canonical fleet launch genesis is not interchangeable with a historical public genesis hash artifact or a development fixture. Recreating a genesis under an existing node can change its identity or balances without repairing its stored history. The detailed ownership rules are in the deployment contract.
An exact release is the input to fleet mutation
The GitHub deployment workflow resolves a target commit and validates a reusable build artifact. A non-dry-run manual deployment requires an artifact_run_id from a prior successful build of that SHA. The release artifact records enough provenance to bind the source revision, protocol, binary and CuEVM payload.
The workflow additionally gates live deployment with owner authorization: its normal route combines a YubiKey authorization with a one-time password; an explicit emergency bypass retains the one-time-password check. These are operational release controls, separate from validator consensus signatures. No credentials or authorization codes belong in the wiki or repository.
A useful release review records:
- Target commit, artifact run, archive hash and node-script hash.
- Live and target protocol numbers, plus the actual consensus-visible changes between them.
- Expected fleet identities, node roles, genesis and effective configuration.
- Current committed and durable heights, epoch files, and account snapshots.
- Recovery artifact locations and the acceptance checks that will demonstrate advancing service.
Use the workflow's dry-run route to validate its build and preflight path before a live operation. “Dry run” is not a local parser check: fleet preflight can still require cloud credentials and access to the selected environment. The workflow and release script define the actual inputs and behavior.
Why cutover stops and starts nodes in this order
After staging all three nodes, the script suspends stall watchdogs and closes the public RPC proxy. It stops val1 and val2 first. Without their votes, the sequencer cannot finalize new work; it is allowed to quiesce before being stopped.
This ordering matters because a validator durably ahead of the sequencer can reject its next proposals. After stopping, the script requires numeric durable heights, equal validator heights, and validators no further ahead than the sequencer. A sequencer ahead of both validators is a different case: the validators can catch up from it.
The script then requires matching eight-byte qmdb/epoch contents. It also compares canonical account snapshot hashes at comparable heights. Comparing snapshots at different heights would reject legitimate progress, so the sequencer comparison is conditional on equal durable heights.
For a protocol change, the cutover requires explicit authorization and the storage-transition regression gate. It snapshots stopped volumes before installation unless the snapshot-skip override is selected. The checked-in workflow currently defaults skip_cutover_snapshots to true for its internal testnet: explicitly turn that off when a snapshot recovery point is required. Same-protocol coordinated swaps skip that extra snapshot step. Neither case deletes chain data.
Validators start before the sequencer. Each must recover at the expected cutover height; then the sequencer starts and the cluster must advance. The public proxy reopens, public protocol and progress are checked, and the optional post-deploy acceptance script runs. The final success message is therefore stronger than “systemd started.” See the coordinated sequence.
Recovery depends on whether new storage has opened
Before the new binary opens storage, the exit handler can restore the old binary pair, restart the old cluster, prove reconvergence, and reopen the proxy. Once a new validator has opened storage, the failure path is an emergency halt with public RPC closed. It does not assume that reinstalling an older executable can reverse a storage migration.
The script cancels and proves termination of an in-flight SSM command before issuing further recovery mutations. If that cannot be proved, it refuses additional mutations and requires operator intervention. This avoids a recovery action racing an unfinished remote install or start command.
A deprecated rolling path remains behind ETO_ALLOW_ROLLING=1 for same-protocol releases. It is not selected automatically and requires independent evidence that the live and target digests are compatible. Its existence is not a recommendation to use it for routine deployment. The exit handler and selection logic define these boundaries.
Protocol coordination is not an enforced wire upgrade
The inspected source declares protocol 5. Its documented change introduces registry transactions and epoch-based validator membership. Protocol 4 identified the timestamp digest and full-account-map commitment changes. Treat either transition as consensus-visible.
The version-aware control-plane helper accepts a one-version forward window and a sixteen-version backward window. That helper is not a Simplex block-validity gate. The current block wire does not carry and enforce the version in the way required for a seamless breaking upgrade. An upgrade proposal coordinates intent; it does not make mixed incompatible kernels safe. See protocol version semantics.
The native upgrade program stores proposals and stake-weighted votes. Its tally requires approval stake strictly greater than two thirds of active stake before approval, and activation has separate state and height requirements. Keep this governance state distinct from the external operation that installs executable bytes. A proposal's binary hash does not itself distribute or run that binary.
The updater is a separate automation path
eto-updater polls a release manifest and detached signature, selects a platform, downloads an archive and checks its SHA-256. Defaults include a sixty-second poll interval, local RPC at http://127.0.0.1:8899, binary path /opt/eto/svm-vm, and staging directory /opt/eto/staged. Automatic voting and restarting are off by default.
Important implementation boundaries are visible in this path:
- Manifest signature verification is skipped when the compile-time release public key is all zero. A release must actually bake in a valid
ETO_RELEASE_PUBLIC_KEY_HEX; a signature URL alone establishes no signer trust. - The staging decision compares the manifest protocol to the updater's compiled protocol. It is not a general same-protocol patch updater.
- An already-existing staged archive bypasses that iteration's download-and-hash step. Protect the staging directory and verify reused artifacts.
- Automatic voting needs a proposal ID and
ETO_VALIDATOR_VOTE_TOKEN. The helper checks HTTP success but does not inspect a JSON-RPC error body before logging submission. - Automatic swapping begins within 32 blocks of the reported activation height, renames the staged executable onto the configured binary, and attempts a systemd restart. It does not reproduce the fleet cutover's paired-library, snapshot and recovery procedure.
These details come from the updater implementation, rather than its older comments about activation timing. Do not combine this sidecar with the fleet deployment flow without establishing which process owns the installed release and restart sequence.
What was checked
The documentation run inspected the deployment workflow, contract, scripts, protocol and updater implementations. It executed the existing debug node's help, version and configuration-inspection paths, and the seventeen Python fleet-configuration audit tests. It did not build a GPU release, execute cloud preflight, stop services, create snapshots, vote on an upgrade or deploy an artifact.
For a stalled or mismatched fleet, continue with observability, incidents and recovery. For why “finalized” can be ahead of local disk, read durability and recovery.
Build & configure
Choose the correct CPU or CUDA artifact, validate nonsecret startup configuration, and start a disposable testnet with clear role and persistence boundaries.
Observe & recover
Diagnose reachability, execution, finality and durable-state failures with bounded probes, and select recovery evidence without confusing liveness with correctness.