Cluster operations

Day-two operations on a registered data-plane cluster: inspecting health, rotating the agent's join token, and tearing down infrastructure. For first-time provisioning, see Getting started: admin.

Listing and inspecting

$ adj cluster ls
$ adj cluster get prod

adj cluster get shows agent health, active instance count, and when the agent last checked in.

Status: provisioning progress + agent health

$ adj cluster status prod
ID:            1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42
Provisioning:  succeeded
Started:       2026-07-18T09:14:02Z
Log:           /home/dana/.config/adj/provision/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42.log
Name:          prod
Agent health:  connected
Instances:     14
Last seen:     2026-07-19T08:12:41Z

Use this while adj cluster provision is running detached to watch the Provisioning: line move from running to succeeded or failed (a failure also shows the exit code and points at the log file). The Provisioning: / Started: / Log: lines come from local provision state, so they only appear on the machine that ran adj cluster provision.

Once provisioning has succeeded, the command doubles as an ongoing health check: Agent health: is connected, stale, or never (agent has never checked in), and Last seen: is the agent's last check-in time. A stale agent is the first sign of a data-plane connectivity problem (see Troubleshooting's 503/504 entries).

Kubeconfig

$ adj kubeconfig
/home/dana/.config/adj/kubeconfigs/prod.yaml

Prints the path to the kubeconfig adj cluster provision wrote for the active context — resolution order is --kubeconfig, then $ADJ_KUBECONFIG, then that generated file. Use it directly:

$ export KUBECONFIG=$(adj kubeconfig)
$ kubectl get pods -A

Your own ~/.kube/config is never touched by adj cluster provision.

Rotating the join token

$ adj cluster rotate-token prod
⚠  Agent join token (shown once — the prior token is now invalid):
x7Dq2m4T9vKcW1pZa8Rf3sLn0uYbEjH5gOiA6dNqVwM

Invalidates the cluster's previous agent join token immediately. Use this if a token may have leaked, or as routine credential hygiene. The agent running in your data plane keeps its existing connection; the new token only matters the next time an agent needs to (re)join. Note that adj cluster bootstrap and adj cluster provision also rotate the token as a side effect — if you've re-run either of those recently, rotate-token isn't the first thing that changed it.

Deprovision vs. rm — two different operations

These are easy to conflate. They do different things:

Deprovision

$ adj cluster deprovision prod
This DESTROYS the AWS data plane (EKS + FSx + all clones) for cluster prod. Proceed? [y/N]: y

Runs the OpenTofu destroy locally (including the two-pass teardown needed for stuck Kubernetes-provider resources). It does not deregister the cluster record — that's a separate step (cluster rm).

Image defaults and the version gate

Both adj cluster provision and adj cluster deprovision fetch the control plane's client config before showing their confirmation prompt:

• Image defaults. --agent-image/--operator-image/--router-image no longer default to a fixed ref baked into the binary — an unset flag resolves to the image the control plane currently publishes at provision time (a baked-in fallback applies only if the control plane can't be reached). The resolved refs are printed before you're asked to confirm:

  Images: agent=registry.stagdb.com/adjoint-agent:v0.4.0-a8f670b operator=registry.stagdb.com/adjoint-operator:v0.4.0-a8f670b router=registry.stagdb.com/adjoint-router:v0.4.0-55985e0
  Tools: all present
  This provisions real AWS infrastructure for cluster prod — ... Proceed? [y/N]:
  

  The Tools: line lists any managed tools adj will download (pinned + checksum-verified, one-time, to ~/.config/adj/tools) before you confirm.

• Version gate. A release build of adj below the control plane's minimum supported version refuses to run provision or deprovision at all, printing the update one-liner instead of prompting:

  $ adj cluster provision prod
  adj v0.1.0 is below the minimum supported release v0.2.0 for cluster provisioning — update first:
    curl -fsSL https://stagdb.com/adjoint/install.sh | sh
  

  Unreleased/dev builds (anything that isn't a clean vX.Y.Z tag) are exempt from the gate — a notice is printed and the command proceeds. If the control plane can't be reached, the gate fails open: a warning is printed and the command proceeds rather than blocking on a policy it couldn't fetch. Caveat: setting ADJ_CLI_MIN_VERSION above the newest released adj blocks all provision and deprovision operations for release builds with no override flag; keep the floor at or below the latest published release. See Client config for the wire contract and Install for the separate, non-blocking update nag that other commands print.

If you pass a UUID cluster id instead of a name, deprovision needs no login — the only control-plane contact is an unauthenticated GET /v1/clientconfig call for the version gate, which proceeds (fail-open) if the control plane is unreachable — and then goes straight to the local tofu destroy using your local AWS credentials. That's deliberate: a logged-out admin, or a break-glass script, can still tear down infrastructure even if the control plane is unreachable. Passing a cluster name instead requires resolving it via the API first, which does require being logged in.

--aws-profile selects which AWS profile tofu and the AWS CLI use (default: ambient credentials / AWS_PROFILE); --yes skips the confirmation prompt.

By default deprovision (like provision) uses the OpenTofu module embedded in adj — no checkout needed. --module-dir <path> is the developer/ from-source override: point it at a local module checkout instead (needed if you originally provisioned with --module-dir; see "Where provisioning state lives" below). Deprovision needs tofu and aws in both embedded and --module-dir modes — adj manages both the same way it does for provision (PATH first, pinned managed download otherwise; see Install § "Tools adj manages").

Where provisioning state lives

The control plane is the durable home of a cluster's OpenTofu state. Every provision, deprovision, and state push syncs the tofu state (terraform.tfstate) to adjd, where it's stored encrypted at rest (AES-256-GCM) alongside a per-cluster operation lock. That means:

• Any org admin can provision or deprovision a cluster from any machine — the state travels with the cluster record, not with a laptop.
• Embedded-mode provision/deprovision still extract the module to ~/.config/adj/provision/bundles/<cluster-id>/ (honoring $XDG_CONFIG_HOME), and the tofu state still lives inside that directory (.terraform/ too) — but that local copy is now a write-through cache, not the sole record. Every run pulls a fresher remote copy first (if one exists) and pushes the result after.
• Upgrading adj refreshes the module's HCL in place against that existing state the next time you provision or deprovision that cluster — you'll see a one-line notice on stderr when this happens. tofu then reconciles any HCL changes against the state as normal.

See Security model's "What the control plane stores" section for exactly what data that stored state contains (and doesn't).

Operation locking

Only one admin can run tofu against a given cluster at a time. provision, deprovision, and adj cluster state push all acquire a per-cluster lock before touching state and hold it for the duration of the run, heartbeating every 60 seconds. A second admin who tries to start an operation against a locked cluster is refused immediately, before any tofu runs:

$ adj cluster deprovision prod
provision in progress by [email protected] on dana-laptop (started 3m12s ago); try later or run 'adj cluster force-unlock <cluster>'

A lock whose heartbeat is more than 5 minutes old is considered stale — the next admin to acquire it steals it automatically (audited), rather than being blocked forever by a crashed process. If you're certain the holder is actually dead (their machine crashed, the process was killed) and don't want to wait out the 5-minute staleness window, break it explicitly:

$ adj cluster force-unlock prod
You are about to force-release the operation lock on cluster "prod".
Type the cluster name (prod) to confirm: prod
Lock on prod released.

force-unlock is interactive only (no --yes), requires typing the exact cluster name, and is audited server-side. It's for genuinely dead holders, not for wresting control from a still-running apply — breaking the lock does not stop the other admin's tofu process. Its heartbeat will start failing (a non-fatal warning on their side), but the apply keeps running to completion and still pushes its state at the end. Use force-unlock when you're sure the other side is gone, not as a way to interrupt a live run.

Reading and pushing state directly

Two commands bypass the full provision/deprovision flow for direct access to the control-plane copy:

$ adj cluster state pull prod -o ./terraform.tfstate
Wrote tfstate (serial 7) to ./terraform.tfstate (mode 0600).

$ adj cluster state push prod
Push local state (serial 8) for cluster prod to the control plane? [y/N]: y
Pushed local state (serial 8) for cluster prod.

• adj cluster state pull <name|uuid> [-o file] — fetches the control plane's stored copy. Without -o it prints to stdout; with -o it writes the file at mode 0600. This is a read — it takes no lock and is safe to run at any time, including while another operation holds the lock.
• adj cluster state push <name|uuid> [--yes] — uploads this machine's local bundle state byte-for-byte, holding the lock for the duration. Unlike provision/deprovision it does not pull-and-merge first: if the control plane's stored serial is already at or above the local serial, the push is rejected (409) rather than clobbering something newer — run state pull first to inspect it. --yes skips the confirmation prompt for scripting.

adj cluster status gains a State: line summarizing the sync state:

State: synced (serial 7, pushed 2026-08-01T14:02:03Z by [email protected])

The forms you'll see:

Degradation: control plane unreachable or unconfigured

State custody is designed to fail safely in both directions:

Named-path (prod, not a UUID) operations refuse outright when the control plane can't be reached, because a lock-less, sync-less run defeats the whole point of shared custody — the admin can retry once connectivity is back, or deliberately opt into a local-only run with --module-dir. The UUID break-glass path exists specifically so a teardown is never blocked by control-plane connectivity, at the cost of losing the concurrency guarantee for that one run (see the failure-modes table below).

Two admins running break-glass UUID deprovisions against the same cluster at the same time while the control plane is down are not protected against each other — there's no lock authority to arbitrate. This is the same exposure that existed before state custody; it's unchanged, and documented here for completeness.

Failure modes

When a pull replaces your local terraform.tfstate with a fresher remote copy, the previous local file is kept alongside it as terraform.tfstate.pre-pull in the bundle directory — a safety net if you need to recover what was on this machine. The first time a cluster's state reaches the control plane (a cluster provisioned before this feature, or its very first push), the CLI prints state adopted by the control plane (first sync) — no manual migration is needed; existing clusters adopt themselves on the next operation.

Use adj cluster bundle export <dir> to get a look at the module without provisioning anything:

$ adj cluster bundle export ./adjoint-module
exported adj v0.2.0 bundle to ./adjoint-module
module: ./adjoint-module/deploy/tofu/adjoint-cluster

It refuses to write into a non-empty directory. Two uses: read the HCL before you run it, or seed a --module-dir workflow (e.g. to carry local patches across provisions instead of relying on the embedded module).

Deregistering

$ adj cluster rm prod

Deletes the cluster's record from the control plane. A cluster with active instances cannot be removed — you'll get a 409 until every instance against it is destroyed. This is the record-keeping step; it doesn't touch AWS infrastructure at all. The normal teardown order is: destroy instances → adj cluster deprovision → adj cluster rm.

Next

• Troubleshooting — what to check when adj cluster status shows a stale agent.
• Usage and audit — adj org usage's CLUSTERS section is the same health data as adj cluster get, for every cluster at once.