Troubleshooting

The hint system

When a command fails against a known API error, the CLI prints the base error followed by an actionable hint: line on stderr:

$ adj clone prod-pg16
Error: instance quota exceeded: per-user cap 3 reached (HTTP 429)
hint: you've hit your instance quota — destroy one ('adj destroy <id>') or ask an admin

The base error always has the shape <message> (HTTP <status>). Not every status has a specific hint — anything without one just prints the base error. -v / --verbose logs the underlying HTTP requests to stderr if you need more than the hint to diagnose something.

Common errors

Your own workload's PVC is stuck Pending

Running your own Postgres, Redis, or any chart on the Adjoint cluster and its PersistentVolumeClaim never binds:

0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims

Two storage rules on an Adjoint cluster differ from a stock EKS cluster:

1. Request exactly 1Gi. The FSx OpenZFS driver takes the PVC size as a sentinel, not a real request — actual capacity comes from the parent filesystem, which every volume shares. Any other value is rejected outright:

failed to provision volume with StorageClass "fsx-openzfs":
rpc error: code = InvalidArgument desc = resourceType Volume expects storage capacity to be 1Gi

So write storage: 1Gi even for a database you expect to grow to hundreds of GiB. To cap real usage, size the filesystem at provision time (adj cluster provision --fsx-storage-gib).

2. fsx-openzfs is the default StorageClass (adj ≥ v0.6.4), so omitting storageClassName works. On a cluster provisioned by an older adj there is no default at all, and a PVC that omits it hangs forever — set it explicitly:

spec:
  storageClassName: fsx-openzfs   # required on clusters from adj < v0.6.4
  resources:
    requests:
      storage: 1Gi                # sentinel — always exactly 1Gi

Do not use the gp2 class EKS creates: it points at the legacy in-tree kubernetes.io/aws-ebs provisioner, which Kubernetes ≥ 1.31 no longer runs, so PVCs bound to it hang with no error. Check what you have with kubectl get sc — the working class is the one whose provisioner is fsx.openzfs.csi.aws.com.

Deprovisioning older clusters

Deprovision can't find OpenTofu state for an older cluster — clusters provisioned by adj ≤ v0.1.0 (or with --module-dir) keep their state in that module checkout directory, not under ~/.config/adj. Re-run with --module-dir pointing at the original checkout.

Deprovision times out on the Kyverno uninstall

adj cluster deprovision (adj ≥ v0.3.1) pre-removes the Kyverno and CloudNativePG helm releases before running the destroy, and prints a ==> kyverno: ... / ==> cloudnative-pg: ... line for each step. If that pre-remove is skipped (the log says why — for example helm/kubectl missing on the machine, or the EKS kubeconfig could not be fetched), the destroy can time out on helm_release.kyverno: Kyverno's admission webhooks keep blocking deletions after its controllers are gone.

If a destroy already timed out (older adj, or a skipped pre-remove), the release is usually wedged in uninstalling and a plain retry fails the same way. Recover manually against the cluster, then re-run the deprovision:

aws eks update-kubeconfig --name <cluster> --region <region> --kubeconfig /tmp/kcfg
KUBECONFIG=/tmp/kcfg kubectl delete validatingwebhookconfiguration,mutatingwebhookconfiguration \
  -l webhook.kyverno.io/managed-by=kyverno
KUBECONFIG=/tmp/kcfg kubectl -n kyverno delete secret -l owner=helm,name=kyverno
adj cluster deprovision <cluster> --yes

The pre-remove needs helm and kubectl on the machine running the deprovision; without them it skips (loudly) and you may need the manual steps above.

Where to look

• adj cluster ls / adj cluster status <name> — agent connectivity and last heartbeat. A disconnected or stale agent explains most 503/504s and slow clones.
• adj org audit --verify — the org's hash-chained history. Useful for reconstructing "who did what, when" after something unexpected happened, and for confirming the audit trail itself hasn't been tampered with.
• adj org usage — one-shot rollup of quota, users, goldens, clusters, pools, and imports; often faster than chasing the same facts across five separate commands.
• adj get <id> — the source of truth for a single instance's current phase. Reach for it any time a command that touches an instance behaves unexpectedly (credentials not ready, still provisioning, etc.).
• -v / --verbose on any command — logs the raw HTTP requests to stderr, useful when reporting an issue.

Login/session issues

If commands start failing with not logged in — run 'adj login' or session expired — run 'adj login' unexpectedly, someone (possibly you, from another machine) may have changed your password or run adj logout --all — both revoke every other session. See Account and sessions.

Still stuck

Check Cluster operations and Goldens and imports for the deeper mechanics behind the commands above, or the generated CLI reference for every flag on every command.