Clone latency on FSx for OpenZFS: from 10 minutes to 16 seconds

A ZFS clone does not copy data. zfs clone writes a bit of metadata pointing a new dataset at an existing snapshot’s blocks, marks them copy-on-write, and returns. The cost is the same whether the snapshot is 10 MB or 10 TB — you only pay for blocks that later diverge. That property is the reason we picked ZFS: it makes it plausible to give every developer their own full-size copy of production data instead of a gutted 500-row fixture set.

AWS offers that property as a managed service. FSx for OpenZFS runs real OpenZFS, and its CSI driver provisions a volume from a snapshot with CopyStrategy=CLONE, sharing blocks rather than copying them. On paper: a developer asks for a clone of a production-size Postgres database and gets one in seconds, no matter how big it is.

We built that, and the block sharing behaved exactly as documented. The latency did not match the underlying primitive, and characterizing the gap is what the rest of this post is about.

Where the latency actually comes from

Our first milestone spike was deliberately small: EKS 1.30, one FSx for OpenZFS SINGLE_AZ_1 filesystem (256 GiB, 128 MB/s), and a ~10 GB golden snapshot — pgbench at scale 700, 10.98 GB logical. We cloned it into three developer namespaces at once, booted a CloudNativePG Postgres on each, and measured clone-to-Ready.

Three concurrent clones came back at 110 s, 177 s, and 238 s — a p50 of 177 s against a gate of 120 s, and a p95 of 238 s.

The storage half passed convincingly. Amplification for the golden plus three clones was 1.107× — 12.16 GB used where four independent full copies would have been about 44 GB. The blocks were genuinely shared, the cloned data directories byte-identical to the golden, and they replayed WAL cleanly after we killed the pods. Copy-on-write worked as advertised.

So the latency was not data movement, and we confirmed it was size-independent — nothing was being read or written. What we were waiting on was the managed control plane in front of ZFS. FSx’s CreateVolume API takes roughly 75 s for a plain provisioned volume and about 2 minutes when creating from a snapshot. That is the floor, and it is independent of the data involved.

The shape of those three numbers is the more consequential detail. 110, 177, 238 — rising with concurrency, spaced about a minute apart. That is not three requests each taking two minutes in parallel; it is the signature of a queue. We ran the test again at ten concurrent clones to characterize how the queue behaves at higher fan-out:

The inter-completion gap is a near-constant ~60 s. FSx CreateVolume serializes at roughly one clone per minute, so the Nth developer to ask for a database waits about N minutes. At K=10 the p50 was 381 s and the p95 was 611 s.

That characteristic is a poor match for the demand pattern a development-environment tool sees. Developer demand for clones is not uniform; it arrives in a burst when a team starts work in the morning, which is precisely the regime in which a serialized API performs worst. Ten concurrent developers is a small team rather than a stress test, and under this API the last request in line waits about ten minutes for an operation the storage layer itself performs in milliseconds.

An unplanned natural control

We built the fix (below), redeployed, and ran a time-to-first-clone benchmark on a separate live cluster, against a smaller golden — prod-pg16-v3, 86 MB and 750k rows — measuring each Instance’s creationTimestamp to its Ready condition. The size difference does not matter here, and that is the point: the claim path moves no data at all, so its latency is independent of golden size, which is what makes these numbers comparable to the 10 GB measurements above.

Nine of the ten claims were fast, and one was not: a single claim raced a momentarily-empty pool, found nothing warm, and fell through to the on-demand path — a real CreateVolume against FSx. It returned in 600.3 s.

We could not have designed a cleaner control. Same cluster, same golden snapshot, same operator build, same measurement — one request took the old path and every other took the new one, roughly 37× apart, with no confounding variable except which code path it took.

It also reproduced the spike’s finding on a live cluster, months later, without our intending it: 600.3 s is what “serializes at ~1/min” predicts for a request landing behind a queue of pool refills. We excluded it from the percentiles — it has no pool-claim timestamp pair, so it is not a sample of what we were measuring — but it is the most informative number in the run.

The fix: moving the API call out of the hot path

If the bottleneck is a control-plane API you do not control, you cannot make it faster. What you can do is arrange not to be waiting on it while the user is.

So we pre-pay it. A pool controller keeps N clones per golden snapshot already provisioned and idle in a holding namespace, each created ahead of demand by the slow serialized path, on our schedule rather than a developer’s. When a developer runs adj clone, they do not create a clone — they claim one.

The claim path makes zero CreateVolume calls. We verified that directly in a separate spike, counting FSx volumes and CSI calls across a claim: the FSx volume count went 3 → 3, with 0 CSI CreateVolume calls, and the resulting Postgres came up Ready with the golden’s data intact.

What a claim does is move an existing volume between namespaces and convince Postgres to adopt it:

1. Patch the pre-warmed PV’s reclaim policy to Retain, so releasing it does not destroy the underlying FSx volume.
2. Delete the holding-namespace PVC. The PV goes to Released; the FSx volume survives.
3. Strip the PV’s claimRef, returning it to Available.
4. Create a PVC named <cluster>-1 in the developer’s namespace, bound by volumeName directly to that PV, carrying CloudNativePG’s data-PVC identity metadata — cnpg.io/pvcRole: PG_DATA, cnpg.io/nodeSerial: "1", and critically cnpg.io/pvcStatus: ready.
5. Create a CloudNativePG Cluster with no bootstrap stanza.

That last pair is the essential part. cnpg.io/pvcStatus: ready is what distinguishes an “adopt me” volume from one still initializing. With it present and no bootstrap stanza, CNPG runs neither initdb nor pg_basebackup — it logs Creating new Pod to reattach a PVC and starts Postgres directly on the existing data directory. No initdb pod is ever created.

We expected to need a specially prepared “cold” golden, since ours is taken as an online volume-snapshot backup and could in principle carry in-progress-backup state. It did not. pg_controldata on the clone showed Backup start location: 0/0, End-of-backup record required: no, Database cluster state: in production. The cloned directory is just a crash-consistent copy of a live PGDATA; Postgres ran ordinary automatic recovery, hit an end-of-recovery checkpoint, and came up. Our existing goldens were directly adoptable.

Measured over the 9-sample benchmark, the claim path gives:

Operator time is the Instance CR’s creationTimestamp to its Ready condition; the CLI wall-clock a developer experiences tracks it about 1 s higher. The remaining ~16 s is not FSx at all — it is the PV rebind (dominated by waiting on the PVC delete) plus CNPG starting a pod and running crash recovery. We moved the two-minute managed-API call out of the hot path, and what is left is Kubernetes and Postgres doing their normal work.

The constraint we did not remove

The pool does not make FSx faster. It moves the wait.

Refill runs at the same serialized rate we measured in the first place: filling a cold pool from 0 to 10 warm clones takes about 580 s, and refilling from 4 to 10 after a benchmark drained it takes about 348 s — both the same ~1 clone/min cadence.

So the honest description of what we built is a buffer. It absorbs a burst up to its warm size at seconds-latency, and beyond that it is bounded by the same API as before. The 600.3 s outlier is not an anomaly; it is what a claim looks like when demand outruns refill — the system’s designed failure mode, observed in practice.

That makes pool size the real operational knob, and it has to be set against expected concurrent-launch rate rather than headcount. A ten-person team that all starts at 9 a.m. needs a bigger warm pool than a thirty-person team spread across time zones. Sizing it from usage telemetry rather than a guess is work we still owe.

Takeaways

1. A managed control plane can dominate the cost of an operation that is nearly free at the storage layer. zfs clone is a metadata write; wrapped in CreateVolume it becomes a two-minute operation. When evaluating a managed version of a primitive you already understand, it is worth benchmarking the API rather than the primitive — the API is the part being purchased.
2. Concurrency behavior is more informative than single-shot latency. One clone at 110 s looked like a tuning problem. The same operation at K=10 was a 611 s p95 and an architectural constraint. Only the fan-out showed whether the design worked, and the indicator was the near-constant ~60 s gap between completions.
3. When an API outside your control is the bottleneck, the remaining lever is removing it from the hot path. We could not make CreateVolume faster, so we arranged for it to happen before anyone was waiting: pre-provision, then claim, with the claim itself kept genuinely cheap. Ours makes zero calls to the slow API, which is why it lands at ~16 s rather than at ~16 s plus an occasional two-minute provisioning call.

Caveats

• n=9, and it is a burst benchmark, not sustained load. A real p95 for a burst. Sustained concurrency — many developers launching over a long window, contending on refill — is future work, and we expect it to be governed by the refill rate above rather than by the 16 s claim number.
• The 600.3 s on-demand figure is a single observation. It is a clean natural experiment and it agrees with the independently measured ~1 clone/min serialization, but it is one sample, not a distribution.
• Filesystem sizing: all numbers come from SINGLE_AZ_1 at 128 MB/s and 1000 IOPS. That matters less than it sounds for the headline result — the claim path moves no data, so the warm-claim numbers are throughput- and IOPS-independent and should hold on larger filesystems. It matters a great deal for what happens after a clone boots: in a separate test, one clone running pgbench alone hit 166 TPS, while ten clones writing concurrently got ~13–19 TPS each, because that budget is a single shared pool split across every clone on the filesystem.
• This is FSx behavior as we measured it, June–July 2026, in us-east-2. AWS may have changed CreateVolume since. Anyone evaluating FSx for OpenZFS would be well served by re-running the fan-out test directly — it takes an afternoon, and it is the number that decides the design.

This pooled claim path is now the default way clones are created in Adjoint.