Cluster tfstate & locking

The control plane is the durable home of each cluster's OpenTofu state: adj cluster provision/deprovision/state push sync it here, encrypted at rest, and a per-cluster advisory lock keeps two admins from running tofu against the same cluster at once. See Where provisioning state lives for the CLI-side flow this API supports, and Security model for exactly what data lives in a stored state blob.

Auth: every route below is admin, org-scoped (cross-org access is a 404, same as other cluster routes).

Feature flag: if the control plane has no ADJ_STATE_ENC_KEY configured, every route on this page returns:

{"error": "state sync not configured"}

with status 503. This is the sole degradation signal — the CLI treats it as "operate local-only," not an error to retry.

GET /v1/clusters/{id}/tfstate — fetch the stored state

Returns the decrypted raw tfstate JSON exactly as last pushed. This is a read — it does not require holding the lock.

$ curl -s https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/tfstate \
    -H "Authorization: Bearer $TOKEN" -D -

Response 200 OK — body is the raw tfstate JSON (Content-Type: application/json), plus these headers:

PUT /v1/clusters/{id}/tfstate — push state

Body is the raw tfstate JSON, sent as-is (not wrapped). The caller must already hold the operation lock — acquire it first via POST .../lock.

$ curl -s https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/tfstate \
    -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
    --data-binary @terraform.tfstate

Response 200 OK

{"serial": 8}

Validation, in order:

1. Admin/org check, then the ADJ_STATE_ENC_KEY check (503 if unset).
2. Lock check — the caller must hold this cluster's lock, or the request is rejected outright.
3. Size cap — plaintext over 10 MiB is rejected.
4. The server parses serial/lineage out of the body — malformed input (not a tfstate JSON) is a 400.
5. Conflict rules, evaluated against the currently-stored serial/lineage:
   • Stored lineage differs from the pushed lineage → conflict.
   • Stored serial is higher than the pushed serial → conflict.
   • Equal serial, identical content → no-op, 200 (idempotent retry).
   • Equal serial, different content → conflict.
   • Otherwise (pushed serial higher, same lineage) → accepted, stored, and audited as cluster.state_pushed (payload: serial + size only — the state content itself is never audited).

POST /v1/clusters/{id}/lock — acquire the operation lock

Body: {"operation": "provision"|"deprovision"|"state-push", "host": "<hostname>"} (host is CLI-supplied, display-only — not validated against anything).

$ curl -s https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/lock \
    -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
    -d '{"operation":"provision","host":"dana-laptop"}'

Response 200 OK

{"acquired_at": "2026-08-01T14:02:03Z"}

If the same caller already holds the lock, re-acquiring just refreshes the timestamps (still 200). If another caller's lock is fresh (heartbeated within the last 5 minutes), the request is refused:

Response 409 Conflict

{
  "holder_email": "[email protected]",
  "operation": "provision",
  "host": "dana-laptop",
  "acquired_at": "2026-08-01T13:58:00Z",
  "heartbeat_age_seconds": 42
}

If the existing lock's heartbeat is older than 5 minutes, it's considered stale: the acquire call steals it instead of 409ing — audited as cluster.state_lock_stolen (payload includes the previous holder).

POST /v1/clusters/{id}/lock/heartbeat — extend the lock

Called by the CLI every 60 seconds for the life of an operation. Holder only.

$ curl -s https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/lock/heartbeat \
    -X POST -H "Authorization: Bearer $TOKEN"

Response: 200 OK, empty body.

A heartbeat failure is not fatal on the CLI side — the running operation keeps going and pushes its state at the end regardless; a lost lock just means the server-side row no longer protects that run (see the sharp edge in Operation locking).

DELETE /v1/clusters/{id}/lock — release the lock

Holder-only release; idempotent.

$ curl -s https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/lock \
    -X DELETE -H "Authorization: Bearer $TOKEN"

Response: 200 OK, empty body.

DELETE /v1/clusters/{id}/lock?force=1 — force-unlock

Any org admin may force-release another holder's lock, regardless of staleness. This is what adj cluster force-unlock calls, after its typed cluster-name confirmation. Audited as cluster.state_lock_broken (payload: previous holder).

$ curl -s "https://api.stagdb.com/v1/clusters/1c9c14da-8b0e-4c6e-9f1a-2d7c3a5b6e42/lock?force=1" \
    -X DELETE -H "Authorization: Bearer $TOKEN"

Response: 200 OK, empty body.

Next

• Clusters — registration, health, join tokens, the provisioning bootstrap bundle.
• Cluster operations guide — the CLI-side lock/pull/heartbeat/push flow and its degradation behavior.