Benchmarking FSx for OpenZFS for Postgres: throughput vs. provisioned IOPS

FSx for OpenZFS is sized along two independent dimensions, and it bills them separately. Provisioned throughput capacity is a fixed ladder of tiers — 64, 128, 256, 512, 1024 MB/s and up — and it is the expensive lever. Provisioned SSD IOPS is a much cheaper add-on above the baseline that comes with your storage. Writing a Terraform module for a Postgres workload requires picking both, and the console does not indicate which one a given database will be sensitive to.

We run Postgres clones for development environments on FSx for OpenZFS, so we had to answer this for our own default sizing. The question was narrow: at a fixed filesystem, which knob buys Postgres TPS — throughput or IOPS — and is the relation linear?

We measured it. The answer is lopsided enough that it changed how we size filesystems.

Method

One EKS cluster, one FSx for OpenZFS filesystem: 256 GiB, SINGLE_AZ_1, in us-east-2. One Postgres 16 database under CloudNativePG, its data directory on a volume from that filesystem, loaded with pgbench at scale 50 — 5,000,000 accounts, about 0.7 GiB logical.

The important detail is that we did not reprovision anything between tiers. FSx accepts throughput and IOPS changes in place via aws fsx update-file-system; each step applies non-disruptively over roughly 2–10 minutes while the filesystem stays mounted and the database stays up. So every row below is the same database, on the same volume, with the same cache state and the same pod — only the filesystem’s provisioned numbers moved. That removes the usual confound where each configuration gets a freshly loaded database with different physical layout.

At each tier we ran in-pod pgbench twice: select-only (-S) and the default TPC-B-like read/write script, each with 4 clients, 2 threads, 60 s.

Two honesty constraints, up front: one run per tier, 60 s each. These are not distributions. The effects below are large enough to survive that — a 2.5× gap does not come from run-to-run noise — but nothing here supports reasoning about a 5% difference.

Results

The first four rows walk throughput across three doublings at fixed IOPS. The fifth changes only IOPS. The sixth is the interesting one: cheapest throughput tier, expensive IOPS.

Findings

1. Throughput scaling is not linear, and its effect is small.

8× the provisioned throughput, 128 → 1024 MB/s, bought +29% TPC-B TPS: 645 → 834. Three doublings of the expensive lever produced less than a third more throughput. The curve rises roughly logarithmically, consistent with marginally lower per-operation queueing rather than any bandwidth constraint being lifted.

That is the expected result once you account for what the workload actually moves. A pgbench TPC-B transaction touches a handful of pages and forces a WAL flush; a 0.7 GiB working set streams nowhere near 128 MB/s, let alone 1024. The read/write workload is bound by sync-write latency per operation, not by bandwidth, so additional bandwidth addresses a constraint that is not binding.

2. Provisioned SSD IOPS is the knob that matters.

3× the IOPS, 1000 → 3000, at fixed 1024 MB/s throughput took TPC-B from 834 → 2,089 TPS — 2.5× — and cut average latency from 4.80 ms to 1.92 ms. That is near-linear in IOPS, which is what a WAL-fsync-bound workload should look like: each commit needs a durable write to land, the rate of durable writes is what IOPS provisions, so TPS tracks it.

Compare the two levers directly. 8× on throughput: +29%. 3× on IOPS: +150%.

3. The cost-optimal shape is cheap throughput plus provisioned IOPS.

The last row is the most useful one for a sizing discussion. 128 MB/s — the lowest tier we tested, the default sizing — combined with 3000 IOPS delivered 1,925 TPS: 92% of the best result we measured, which needed 8× the throughput to add the remaining 8%.

On FSx for OpenZFS pricing, stepping throughput from 128 to 1024 MB/s is the expensive lever; 2,000 extra provisioned IOPS costs a small fraction of that. We are deliberately not printing dollar figures — AWS pricing varies by region and changes — but the ratio is substantial, and it points in the same direction as the performance data. The configuration that performs nearly best is also the less expensive one, which makes the trade-off unusually straightforward.

4. Read-heavy work is insensitive to both knobs.

Select-only sat at roughly 15–16k TPS across every tier, with average latency in the 0.24–0.28 ms range. The spread has no trend in it — the highest select-only number, 16,361 TPS, is at 256 MB/s, in the middle of the ladder, and the lowest is at the tier with the most IOPS.

The reason is that the hot set lives in Postgres shared buffers and the page cache, so the reads mostly never reach FSx. The pod’s CPU is the ceiling. For a read-heavy workload that fits in RAM, neither FSx knob is the bottleneck, and the choice between them has little effect either way.

What this means for sizing

For a write-oriented workload, the data supports scaling provisioned IOPS with expected concurrent write load while holding throughput at a low tier. That runs against the intuition that a “bigger” filesystem means more MB/s, and against what our own default sizing did before this run — we had been scaling throughput with the number of developers and pinning IOPS at 1000, which is the less effective allocation for anything TPC-B-shaped.

Throughput still matters, just not for OLTP. It is the right lever for streaming reads and bulk sequential work: restoring a dump, an initial data load, a large sequential scan that misses cache, a backup drain. Those genuinely move bytes at a rate a 128 MB/s tier will cap. A filesystem serving both — an OLTP database plus a nightly bulk load — is best sized with throughput chosen for the bulk window and IOPS for the transactional steady state; the two are independent enough to reason about separately.

Two practical notes. First, SINGLE_AZ_1 throughput is a fixed list of tiers, not a continuous dial: 64, 128, 256, 512, 1024, 2048, 3072, 4096 MB/s — a request for 200 is not possible, so sizing math needs to be rounded onto that ladder before any decision is made. Second, because updates apply in place and non-disruptively, this is an inexpensive experiment to repeat against a specific workload: a single afternoon of aws fsx update-file-system and pgbench produces a table like the one above for the database actually in use, which is more informative than our numbers.

One operational caution for anyone repeating it: an in-place FSx update puts the filesystem in an optimization window for several minutes, and we saw volume-creation calls against that filesystem stall for the duration. It is non-disruptive to running I/O, but resizing a filesystem is best avoided while anything depends on provisioning new volumes from it on a schedule.

Caveats

• One 60 s run per tier. No variance estimate, no repeats, no warm-up discipline beyond the fact that the same database stayed hot across all six tiers. The 2.5× IOPS effect is far too large to be noise; the +29% throughput effect is directionally solid but its exact magnitude is not something we would defend to two significant figures.
• Small hot set. At scale 50, roughly 0.7 GiB, the data fits comfortably in cache. That is the entire explanation for the flat read-side numbers, and a working set larger than RAM would change them substantially. It would not change the write-side finding: WAL fsync is a durability cost that a bigger cache does not remove, so the IOPS-over-throughput conclusion should hold — and get stronger — as the database grows.
• One Postgres instance, no concurrent cluster load. The filesystem’s provisioned budget is shared across everything on it. These numbers describe one database with the whole budget to itself; a filesystem hosting many active databases splits it, and per-database TPS falls accordingly.
• Modest client concurrency. 4 clients / 2 threads is a light load. Higher write concurrency will move the absolute TPS numbers and may change where the IOPS curve stops being near-linear; we did not find that ceiling.
• SINGLE_AZ_1, us-east-2, July 2026. Deployment type, region, pricing, and FSx behavior can all change. MULTI_AZ_1 has different write-path characteristics that we did not measure at all.