To ship correct software, developers need data that behaves like production: real volumes, real distributions, real edge cases. The reason almost nobody gets that is arithmetic. A full copy of a multi-TB database costs real time and real money, multiplied by every developer who needs one. The naive approach is O(size × developers), and both terms grow.
So teams generally pick from four options, each of which is a version of the same concession:
| What teams do instead | What it costs |
|---|---|
| Tiny seed or fixture data | The query that returns in 3 ms on 500 rows times out on 50 million. Cardinality-dependent bugs reach production because nothing before production had the cardinality. |
| One shared staging database | Contention and pollution. No isolation. One destructive migration or one long-running ALTER TABLE blocks everyone, so risky work gets scheduled instead of done. |
| Manual production dumps | Hours per copy, perpetually stale by the time it lands, full storage cost per copy, and unmanaged production data sitting on laptops. |
| Per-developer cloud copies (e.g. one RDS instance each) | Honest but expensive: at multi-TB × headcount the bill is the point of the exercise, and because refreshing costs the same again, nobody refreshes. |
Each of these is a rational response to the multiplication. If you can remove the multiplication, all four choices stop being necessary.
What copy-on-write actually does
ZFS stores a dataset as a tree of blocks. A snapshot does not copy anything — it pins the current block tree so those blocks cannot be freed or overwritten in place. A clone is a new, writable dataset whose root points at that same pinned tree.
From there the accounting is simple. Reads on the clone follow shared blocks. A write on the clone allocates a new block and repoints the clone’s tree at it; the snapshot’s block is untouched, because the snapshot still needs it. Nobody copies the unchanged 99%.
So N clones of a base dataset cost:
base + Σ(divergence per clone)
not N × base. Two consequences fall out of that, and they are the whole argument. First,
creating a clone is a metadata operation, so it does not get slower as the dataset gets
bigger. Second, storage is charged for what each developer changes, not for what they
have. A developer who clones a 2 TB database and runs a migration against three tables pays
for three tables’ worth of rewritten blocks.
The question is whether that survives a managed cloud filesystem and a real Postgres on top of it. We measured it on AWS FSx for OpenZFS.
Measured
Our first substrate spike ran on EKS 1.30 with a single FSx for OpenZFS SINGLE_AZ_1
filesystem (256 GiB, 128 MB/s) in us-east-2. The base dataset was a ~10 GB golden — pgbench
at scale 700, 10.98 GB logical — snapshotted and cloned into separate developer namespaces,
with a CloudNativePG Postgres booted on each clone.
Golden + 3 clones: 1.107×. Total storage consumed was 12.16 GB, against roughly 44 GB for four independent full copies of the same data. That is the golden plus three clones, on the ~10 GB dataset described above.
Golden + 10 clones, before divergence: 1.115×. A second run on the same substrate and the same ~10 GB golden cloned ten times: 12.24 GB total, where eleven full copies would be about 121 GB. Per-clone overhead at rest was around 126 MB. These are two separate measurements — 1.107× is the 3-clone run, 1.115× the 10-clone run — and going from 3 clones to 10 barely moved the multiplier, because the multiplier is not a function of clone count.
With writes, amplification tracks divergence, as it must. After driving 120 s of heavy pgbench writes into each of the ten clones, total storage rose to 1.366× — about +2.76 GB, or roughly 276 MB of divergence per clone for that write burst. This is not a defect in the model; it is the model. You pay for changed blocks. Ten clones each carrying 276 MB of changes still cost about 8× less than eleven full copies.
The clones are real copies, not approximations. Data digests on the cloned Postgres data
directories matched the golden’s byte for byte (be80ca2f…), and after killing the pods the
clones replayed WAL cleanly and came back up. There is no consistency tax being hidden inside
the storage number.
Amplification is divergence-driven and base-size-independent. Nothing in the mechanism reads the size of the base dataset. A 1 TB golden shares blocks exactly the way a 10 GB one does; what changes is the absolute size of the base you pay for once, not the multiplier.
What that buys as a workflow
Cheap clones only help if the lifecycle around them is cheap too, and the lifecycle is where a storage property turns into a way of working.
An admin curates a golden snapshot from a real source of truth — an RDS Postgres, say —
and versions it. That golden is immutable, so every clone taken from v3 is the same v3,
which is what makes a test run reproducible weeks later. Then each developer works against
their own clone of it, in their own namespace, with four operations:
- clone — take a private, full-size database from a golden version.
- reset — throw the current one away and re-clone the same version. This is what makes destructive work safe: run the migration, drop the table, rewrite the index, and if it goes badly, discard the evidence in seconds instead of asking who else is on staging.
- refresh — re-clone from a newer golden version when the base data has moved on.
- destroy — reclaim it.
What makes this different from a dump is that reset and destroy are cheap enough to be reflexes. A copy that took two hours to produce becomes something you protect; a copy you can replace in seconds becomes something you use hard and throw away. Disposability is a behavior change, and it only exists if both the storage cost and the time cost are near zero.
The time cost is a separate engineering problem, and on FSx it was the harder one — the managed
CreateVolume API serializes at roughly one clone per minute, which put the tenth concurrent
clone about ten minutes out. With a pre-warmed pool in front of it, the measured p50 for a
developer claiming a clone is 16.0 s. That work is written up separately in
Clone latency on FSx for OpenZFS: from 10 minutes to 16 seconds.
The honest bill
Copy-on-write removes one specific cost — the per-copy multiplication of storage. It does not make infrastructure free, and it is worth being precise about what is left.
You still pay for the AWS infrastructure you run: the EKS cluster and the FSx for OpenZFS filesystem, including its provisioned throughput and IOPS. That is a real, recurring bill, and it has to be sized for the aggregate concurrent activity of your developers rather than per-developer — an FSx filesystem’s throughput and IOPS budget is a single shared pool.
On top of that you pay for one base dataset per golden version you keep, plus each developer’s divergence. Ten developers on one golden version is one base plus ten small deltas. Keeping four golden versions live is four bases — versions are not free, and retention policy is a real decision.
What you do not pay is a full copy per developer, and you do not ship production data to a third party to get any of it: the clones live on infrastructure in your own account.
Who this is not for
Two cases where this is the wrong tool, stated plainly.
Sub-GB databases. If a production database is a few hundred megabytes, pg_dump and
restore takes a minute and costs nothing worth measuring. The argument above is about
collapsing a multiplication; when the quantity being multiplied is small, there is nothing to
collapse, and a plain copy is the better answer.
Regulated PII or PHI, until masking exists. Cloning gives every developer a byte-identical copy of production — which is the point, and also the problem if that data is regulated. Masking is not built yet, so clones today are raw. Where a compliance posture requires that developers never touch unmasked production data, namespace isolation does not address that requirement. This approach currently suits data that is acceptable to clone in full.
Caveats
- The amplification numbers come from a ~10 GB golden. 1.107× is the golden + 3 clones run; 1.115× is the separate golden + 10 clones run. We have not measured the multiplier directly at 1 TB. The mechanism is size-independent and we expect it to hold, but that is an argument from how ZFS works, not a measurement at that scale.
- The 1.12× in this post’s title is an illustrative model, not a measurement. It is a 500 GB golden plus ten developers each changing 6 GB — 560 GB, or about 1.12× — and it assumes pessimistically that every changed byte costs its full size. The measured anchors are the two above: 1.107× at 3 clones and 1.115× at 10, both on a ~10 GB golden. Your ratio depends entirely on how much your developers actually change.
- Divergence dominates over time, and our divergence figures are short-window. The 276 MB per clone came from a 120 s pgbench write burst — a deliberately heavy synthetic write load over two minutes, not a measurement of what a developer accumulates over a working week. We do not have a measured sustained-use divergence number yet. A clone that lives for months under heavy writes converges toward the cost of a full copy. Copy-on-write defers storage cost; it is disposability — reset and refresh being cheap enough to use — that keeps the deferred cost from accumulating. TTLs and per-namespace quotas are load-bearing, not polish.
- Storage efficiency is not performance. These clones share a filesystem’s provisioned throughput and IOPS. In a separate test, one clone running pgbench alone reached 166 TPS while ten writing concurrently got roughly 13–19 TPS each. Cheap storage does not mean free I/O, and the sizing question is independent.
- FSx for OpenZFS, us-east-2, June–July 2026. Managed-service behavior changes. The ZFS property underneath is stable; the numbers around it are a snapshot of one provider at one point in time.
- Postgres only, at the time of writing. The mechanic is substrate-agnostic in principle — it is a filesystem property — but everything above was measured with CloudNativePG.
Adjoint packages exactly this: clone, reset, refresh, and destroy across versioned golden snapshots.