Bare-Metal Linux NVMe Storage Architecture: OpenZFS Mirror vs. mdadm RAID-10 + XFS/ext4

1. The Architectural Trade-Off: Copy-on-Write Integrity (OpenZFS) vs. Raw IOPS (mdadm + XFS)

When provisioning a dedicated bare-metal server equipped with two or four enterprise PCIe Gen4/Gen5 NVMe drives (such as Micron 7450 PRO, Samsung PM9A3, or Solidigm P5520), one of the first decisions an infrastructure engineer faces is choosing the storage stack:
1. Linux Kernel Software RAID (mdadm RAID-1 or RAID-10) formatted with XFS or ext4, or
2. OpenZFS (zpool Mirror or Striped Mirrors / RAID-10 equivalent).

Both stacks are battle-tested in production, but they optimize for fundamentally different goals:

| Storage Property | mdadm RAID-10 + XFS / ext4 | OpenZFS Striped Mirrors (RAID-10 Equivalent) |
| :--- | :--- | :--- |
| Write Model | Overwrite-in-Place (lowest CPU & write-amplification overhead) | Copy-on-Write (CoW) (never overwrites live blocks; immune to write-hole corruption) |
| End-to-End Checksumming | None at filesystem block data level (relies on NVMe hardware ECC) | 256-bit Fletcher4 / SHA-256 / BLAKE3 checksums on every block with automatic self-healing from mirror |
| Inline Transparent Compression | Not native in XFS/ext4 | Native lz4 / zstd block compression (typically 1.8x–3.2x compression on text/JSON/SQL data) |
| Instant Atomic Snapshots & Replication | Requires LVM thin provisioning (lvcreate -s has heavy CoW penalty) | Zero-cost instant snapshots (zfs snapshot) + incremental block-level zfs send | zfs receive |
| RAM Consumption | Uses standard Linux page cache (minimal overhead) | Uses dedicated Adaptive Replacement Cache (ARC) (defaults up to 50% of RAM unless capped) |
| Raw Random 4K NVMe IOPS | 90% – 98% of raw hardware NVMe ceiling | 55% – 75% of raw hardware ceiling (due to CoW, checksums, and ARC pipeline) |

2. Configuring mdadm RAID-10 + XFS for Maximum Raw NVMe Throughput

If your bare-metal server runs a single high-concurrency workload that already implements its own application-level checksums and replication—such as ClickHouse, Cassandra, ScyllaDB, or a dedicated PostgreSQL node backed by pgBackRest—mdadm RAID-10 with XFS delivers the lowest latency and highest raw IOPS:

# 1. Create a 4-drive Linux mdadm RAID-10 array with a 64K chunk size
sudo mdadm --create /dev/md0 \
  --level=10 \
  --raid-devices=4 \
  --chunk=64 \
  /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
# 2. Format with XFS aligning stripe unit (su=64k) and stripe width (sw=2 data disks)
sudo mkfs.xfs -f -d su=64k,sw=2 -L prod_data /dev/md0
# 3. Mount with noatime and discard (or fstrim.timer) in /etc/fstab
# LABEL=prod_data  /var/lib/postgresql  xfs  defaults,noatime,nodiratime  0  2

*Why enable weekly fstrim.timer instead of continuous discard mount options?* Continuous inline TRIM (discard on every file deletion) can stall synchronous NVMe command queues on certain controller firmwares; running systemctl enable --now fstrim.timer executes TRIM asynchronously during off-peak maintenance windows.

3. Tuning OpenZFS for Virtualization (Proxmox VE) & Databases (recordsize Alignment)

When using OpenZFS, virtually all "ZFS is slow for databases" complaints trace back to a single misconfiguration: leaving the dataset recordsize at its default 128K while running a database that reads and writes 8K or 16K pages!

If PostgreSQL modifies a single 8 KB page inside a 128 KB ZFS record, ZFS must read all 128 KB from disk, modify 8 KB in memory, recalculate the checksum, and write a new 128 KB block (16x Read-Modify-Write amplification!).

Essential OpenZFS Pool & Dataset Rules

1. Always Set ashift=12 (4K sectors) or ashift=13 (8K sectors): Never allow ZFS to auto-detect ashift=9 (512-byte sectors) on an NVMe drive that lies about its physical page size. 2. Match recordsize to the Application's Native Page Size: - PostgreSQL Data Directory: zfs set recordsize=8K tank/postgres/data ( keep recordsize=128K on a separate dataset tank/postgres/wal for sequential Write-Ahead Logs!). - MySQL / InnoDB Data Directory: zfs set recordsize=16K tank/mysql/data. - KVM Virtual Machine ZVOLs (volblocksize): volblocksize=16K or 64K. 3. Enable compression=lz4 or zstd-3 and atime=off: On modern CPUs, lz4 compresses and decompresses at multiple gigabytes per second per core, reducing actual bytes written to flash and increasing effective IOPS:

# Create a Striped Mirror (RAID-10 equivalent) ZFS Pool with 4K sector alignment
sudo zpool create -f -o ashift=12 \
  -O acltype=posixacl \
  -O xattr=sa \
  -O dnodesize=auto \
  -O compression=lz4 \
  -O atime=off \
  -O relatime=off \
  tank \
  mirror /dev/disk/by-id/nvme-eui.0001 /dev/disk/by-id/nvme-eui.0002 \
  mirror /dev/disk/by-id/nvme-eui.0003 /dev/disk/by-id/nvme-eui.0004
# Create dedicated datasets tuned specifically for PostgreSQL 17 Data (8K) and WAL (128K)
sudo zfs create -o recordsize=8K -o logbias=throughput tank/pgdata
sudo zfs create -o recordsize=128K tank/pgwal
# Cap ZFS ARC memory to 16 GB on a 64 GB host so PostgreSQL shared_buffers has guaranteed RAM
echo "options zfs zfs_arc_max=17179869184" | sudo tee /etc/modprobe.d/zfs.conf

4. Critical NVMe Pre-Deployment Step: Formatting Logical Sector Size to 4096 Bytes (4Kn)

Enterprise datacenter NVMe drives (Samsung PM9A3, Solidigm D7-P5520, Micron 7450 PRO) often ship from the factory formatted with legacy 512-byte logical sectors (512e) for backward compatibility. Formatting NVMe namespaces to native 4096-byte sectors (4Kn) before creating your mdadm array or OpenZFS pool eliminates read-modify-write sector translation overhead and improves sustained random 4K write IOPS by 8% to 15%:

# 1. Inspect supported LBA formats on your NVMe drive (look for 'lbads:12' = 4096 bytes)
sudo nvme id-ns -H /dev/nvme0n1 | grep -E "LBA Format|in use"
# 2. Format the NVMe namespace to the 4096-byte LBA format index (e.g., -l 1)
# WARNING: This cryptographically erases all existing data on /dev/nvme0n1
sudo nvme format /dev/nvme0n1 -l 1 --ses=0

5. Self-Verification Storage Benchmark (fio) & Decision Matrix

After provisioning either mdadm RAID-10 + XFS or OpenZFS Striped Mirrors, validate sustained random 4K fsync latency and sequential bandwidth using fio:

# Benchmark synchronous database-like 4K random writes (O_DIRECT + fsync)
fio --name=nvme-db-sync-test --filename=/mnt/data/fio-test.bin \
  --ioengine=libaio --direct=1 --bs=4k --iodepth=1 --rw=randwrite \
  --fsync=1 --size=4G --runtime=60 --time_based --group_reporting

| Storage Requirement | Recommended Linux Stack | Key Engineering Reason |
| :--- | :--- | :--- |
| Maximum Raw IOPS & Lowest P99 Latency (PostgreSQL, Valkey, ClickHouse, Kafka) | mdadm RAID-10 + XFS (noatime) | Zero Copy-on-Write fragmentation, minimal RAM overhead, and direct NVMe queue passthrough. |
| Hypervisor Host (Proxmox VE), Instant Rollback Snapshots & Bit-Rot Healing | OpenZFS Striped Mirrors (ashift=12, lz4) | Atomic ZFS snapshots, fast incremental replication (zfs send/receive), and end-to-end Fletcher4/BLAKE3 checksums. |