Measured Against the NVMe It Runs On
One instance, one drive: a raw partition of the local NVMe against a flexBlock device whose cache is a sibling partition of the same drive. With the working set in cache, flexBlock keeps pace with the NVMe.
of raw NVMe sequential read and write throughput
warm random 4 KiB read IOPS, four jobs
cold sequential read from S3, prefetch 64
of raw NVMe on MariaDB write-only OLTP
Raw NVMe Versus flexBlock, Warm
fio over io_uring with O_DIRECT on a 20 GiB region already resident in the cache and flushed: the steady state of a working set that fits.
| Test | Raw NVMe | flexBlock | Ratio |
|---|---|---|---|
| Sequential read, 1 MiB, QD32 | 4,069 | 4,069 | 100% |
| Sequential write, 1 MiB, QD32 | 3,212 | 3,213 | 100% |
| Random read, 128 KiB, QD16 | 4,117 | 4,069 | 99% |
| Random 4 KiB read, QD1 | 13,331 | 11,805 | 89% |
| Random 4 KiB read, QD32, one job | 336,876 | 242,017 | 72% |
| Random 4 KiB read, QD32, four jobs | 584,602 | 604,909 | 103% |
| Random 4 KiB write, QD1 | 56,929 | 48,074 | 84% |
| Random 4 KiB write, QD32, one job | 330,675 | 298,443 | 90% |
| Random 4 KiB write, QD32, four jobs | 330,678 | 330,035 | 100% |
| O_SYNC 4 KiB write, QD1 | 55,646 | 41,074 | 74% |
| Mixed 70/30, 16 KiB, QD16, four jobs | 219,633 | 182,212 | 83% |
PostgreSQL 18 with pgbench
XFS, scale 500, 32 clients, an fsync on every commit. TPC-B mean latency 0.58 ms raw, 0.69 ms on flexBlock.
| Test | Raw NVMe | flexBlock | Ratio |
|---|---|---|---|
| Load 50 M accounts | 20.4 s | 23.1 s | 88% |
| TPC-B read/write, tps | 55,276 | 46,045 | 83% |
| Select-only, tps | 424,940 | 417,717 | 98% |
MariaDB 11.8 with sysbench
XFS, OLTP on 16 tables of 1 M rows, 32 threads, redo flushed on every commit. Read/write p95 latency 3.89 ms raw, 3.49 ms on flexBlock.
| Test | Raw NVMe | flexBlock | Ratio |
|---|---|---|---|
| Load 16 M rows | 13.8 s | 13.6 s | 101% |
| OLTP read/write, tps | 13,648 | 14,077 | 103% |
| OLTP read-only, tps | 20,788 | 19,963 | 96% |
| OLTP write-only, tps | 49,542 | 57,912 | 117% |
Cold Reads, Straight From the Bucket
After a detach and reattach the cache is empty, so every read is a miss served from Amazon S3. Sequential reads trigger prefetch, and a random miss brings its whole chunk into the cache, so the region warms within the run.
The 2.6 GiB/s figure uses a deeper prefetch and more concurrent downloads than the defaults, which favor mixed workloads. Compression helps cold reads as far as the data compresses.
What the Numbers Mean
Every write is acknowledged from NVMe and every warm read is served from it, so bandwidth-bound and multi-queue workloads see the drive, not the daemon. Sequential throughput and multi-job random reads and writes match the raw partition within noise, the mixed run reaches 83%, and read-heavy database runs stay within a few percent of raw.
The cost of the extra hop through the daemon, a few microseconds per request, is most visible at queue depth one or on a single submitting thread, and write-heavy mixed and TPC-B runs reach 83%. More queues close the gap. Cold reads are bounded by the object store and the prefetch depth.
Fast, and Correct
The same build passed a checksum-verified test suite on the S3-backed volume before a single number was recorded.
Every block verified
Random sizes and offsets, mixed read/write, trims and O_SYNC writes, all checked by fio against a checksum on every block, with zero errors.
Killed mid-write
The daemon was killed with SIGKILL during a random write run. The kernel held the queued I/O, the restarted daemon recovered the device, and fio verified every block.
Cold from the bucket
After detaching and reattaching an empty cache, every region was read back from S3 and verified.
Databases intact
PostgreSQL passed pg_amcheck over every table and index, and MariaDB passed CHECK TABLE EXTENDED on all 16 tables, both reopened cold from the bucket.
Method
AWS i7i.4xlarge, 16 vCPUs, 128 GiB, Linux 7.0, with one local NVMe drive holding both the raw test partition and the flexBlock cache, and Amazon S3 in the same region. fio 3.41 with io_uring and O_DIRECT on a 20 GiB region, a 5 s ramp, then 30 s per test. pgbench from PostgreSQL 18 and sysbench 1.0.20 against MariaDB 11.8, both on XFS. FlexBlock 1.0.0 with default settings (2 MiB chunks, zstd), a 200 GiB volume on a 250 GiB cache partition. Ratio is flexBlock over raw for throughput and raw over flexBlock for load times, so higher is better.Full details, including latency.
Benchmark It on Your Workload
Apart from the deep-prefetch cold read, every result here used default settings. Talk to us about running flexBlock on your own volumes.