ZFS Pool Recovery and Troubleshooting Guide

• 1.zpool export poolname flushes all pending transaction groups to disk and marks the pool as exported. This is the cleanest way to take a pool offline. Only works if the pool is in ONLINE or DEGRADED state.
• 2.zpool import -o readonly=on poolname imports the pool in read-only mode, preventing the OS from writing metadata updates. However, NEVER run this on drives suspected of physical failure. Even a read-only import forces massive random read activity across the array to traverse the ZFS metadata tree, which will destroy a drive with failing read heads. Always perform a sector-by-sector clone of every member drive using dedicated imaging hardware first, and only attempt to import the cloned images.
• 3.zpool import -f poolname force-imports a pool that was not cleanly exported. ZFS replays the intent log (ZIL) to recover acknowledged synchronous writes not yet committed via a transaction group sync. This writes to the drives and may advance the on-disk state past a recoverable point.
• 4.If the pool will not import at all, do not use -f repeatedly. Image the drives and work from copies.

• 1.ZFS flushes a new TXG to disk every 5 seconds (default) or when the write buffer fills.
• 2.The uberblock at the top of the pool metadata tree points to the most recent valid TXG. ZFS stores a ring buffer of historical uberblocks, providing a history of recent TXGs.
• 3.zpool import -T <txg> -o readonly=on poolname imports the pool using a specific historical TXG instead of the most recent. This effectively rolls back the pool to an earlier consistent state.
• 4.TXG rollback only works if the on-disk blocks for the older TXG have not been overwritten by subsequent writes. Copy-on-write preserves old blocks until the space is needed, so recently written pools with available free space have better rollback success rates.
• 5.Use zdb -e -u poolname to list available uberblocks and their TXG numbers before attempting a rollback. The -e flag lets zdb read an exported or non-imported pool directly from raw devices.

• 1.Check physical connections first. UNAVAIL often means ZFS cannot find the device path. A reseated SATA cable or a different controller port may resolve it.
• 2.Check SMART data. If the drive reports reallocated sectors, pending sectors, or UNC errors, the hardware is failing.
• 3.If the pool is DEGRADED (still has margin), zpool replace poolname old-dev new-dev initiates a resilver (ZFS term for rebuild). This reads all surviving vdev members to reconstruct the replacement.
• 4.If the pool is FAULTED (no remaining margin), do not attempt to replace. The pool cannot guarantee data integrity. Image every drive and attempt offline reconstruction.

For enterprise server data recovery, FAULTED ZFS pools on production systems require imaging before any repair attempt. ZFS's copy-on-write architecture means the historical TXG data is still on-disk; any write operation (including a replace or scrub) can overwrite those blocks.

The counterintuitive part comes first, because it is what sends people down the wrong path. ZFS native encryption operates per dataset, not per pool and not at the block device layer. The pool imports normally with no key present. Topology reads, vdev labels verify, unencrypted datasets mount, and zpool status reports a healthy pool. The encrypted datasets sit unmounted. So "the pool imported" tells an owner nothing about whether their files are reachable.

zfs list -o name,keystatus,encryptionroot

That listing is the real status check. Each independently encrypted dataset, or a hierarchy of datasets inheriting one key configuration, carries an encryptionroot. A locked dataset shows keystatus of unavailable. Only after zfs load-key succeeds does it flip to available and allow a mount.

Where the key lives

• 1.A randomly generated master key encrypts the file data. It is created when the dataset is created and it is stored on disk in wrapped form. It never leaves the pool.
• 2.The user-supplied passphrase, raw key file, or hex key is the wrapping key, and it is held by the operator rather than by the pool. zfs load-key uses it to unwrap the master key into memory.
• 3.With encryption=on the default suite is aes-256-gcm; older releases defaulted to aes-256-ccm. Passphrases derive the wrapping key through PBKDF2, and the pbkdf2iters property defaults to 350,000 iterations and cannot be set below 100,000.
• 4.zfs change-key is not a way in. It re-wraps the existing master key with a new wrapping key, and it requires the dataset to already be unlocked with the current key loaded.
• 5.While a dataset is unlocked, the key material resides in system memory. On a live machine that is still unlocked, dumping kernel memory to extract key material is theoretically possible. Once the machine crashes, reboots, or loses power, that material is gone and no imaging of the drives brings it back. This is not a service we offer; it is stated here so nobody powers off a still-unlocked box expecting to pick up where they left off.

What a lab can still report

ZFS native encryption protects the data payloads and the file names, not the enclosing filesystem structure. With no key at all, an evaluation can honestly tell an owner what existed: dataset names and the filesystem hierarchy, used capacity along with quotas and reservations, snapshot names and their creation timestamps, and dataset properties such as compression algorithm, record size, cipher, and PBKDF2 iteration count.

File names and file contents inside an encrypted dataset stay obscured. That inventory is often enough to tell somebody which of several pools held the data they care about, or that the snapshot they were counting on was never taken.

On TrueNAS, which uses OpenZFS native encryption rather than anything proprietary, a key configured for automatic unlock at boot is stored in the TrueNAS configuration database on the unencrypted boot pool. Losing the boot device without a backup of that configuration or a manually exported key file loses the wrapping key, and the data pool stays sealed even though every data drive is healthy.

We run this evaluation in-house at the Austin, TX lab with no diagnostic fee, and if the key is gone we say so and charge nothing. No data, no recovery fee applies here in its most literal form. Where a key does exist and the obstacle is pool structure rather than cryptography, the work returns to imaging every member and reconstructing the pool offline from the images.