ZFS Pool Recovery and Troubleshooting Guide

• 1.zpool export poolname flushes all pending transaction groups to disk and marks the pool as exported. This is the cleanest way to take a pool offline.
• 2.zpool import -o readonly=on poolname imports the pool in read-only mode, preventing the OS from writing metadata updates. Don't run it on drives you suspect are physically failing. Clone every member drive sector by sector on dedicated imaging hardware first, and import only the clones.
• 3.zpool import -f poolname forces the import of a pool that appears to be potentially active, such as one that was never exported. That's a read-write import. It writes to the drives and can push the on-disk state past a point you could have recovered from.
• 4.If the pool will not import at all, do not use -f repeatedly. Image the drives and work from copies.

• 1.ZFS flushes a new TXG to disk every 5 seconds (default) or when the write buffer fills.
• 2.The uberblock at the top of the pool metadata tree points to the most recent valid TXG. ZFS stores a ring buffer of historical uberblocks, providing a history of recent TXGs.
• 3.zpool import -T <txg> -o readonly=on poolname imports the pool using a specific historical TXG instead of the most recent. This effectively rolls back the pool to an earlier consistent state.
• 4.TXG rollback only works if the on-disk blocks for the older TXG have not been overwritten by subsequent writes. Copy-on-write preserves old blocks until the space is needed.
• 5.The uberblock ring lives in the vdev labels on each member device, so read it off a device rather than off the pool name. Use zdb -l -u /dev/sdX to list the uberblocks held in that device's labels and their TXG numbers before attempting a rollback. zdb -e -u poolname displays the current uberblock. The -e flag tells zdb to work on an exported pool, one that isn't in /etc/zfs/zpool.cache.

• 1.Check physical connections first. UNAVAIL means ZFS couldn't open the device, and a changed device path is one cause. Reseat the SATA cable or try another controller port before you assume the drive failed.
• 2.Check SMART data. If the drive reports reallocated sectors, pending sectors, or UNC errors, the hardware is failing.
• 3.If the pool is DEGRADED (still has margin), zpool replace poolname old-dev new-dev initiates a resilver (ZFS term for rebuild). That resilver reads the surviving vdev members to reconstruct the replacement.
• 4.If the pool is FAULTED (no remaining margin), do not attempt to replace. The pool cannot guarantee data integrity. Image every drive and attempt offline reconstruction.

For enterprise server data recovery, FAULTED ZFS pools on production systems require imaging before any repair attempt. ZFS's copy-on-write architecture means the historical TXG data is still on-disk; any write operation (including a replace or scrub) can overwrite those blocks.

ZFS native encryption operates per dataset, not per pool and not at the block device layer. The pool imports normally with no key present. Topology reads, vdev labels verify, unencrypted datasets mount, and zpool status reports a healthy pool. The encrypted datasets sit unmounted. So "the pool imported" tells an owner nothing about whether their files are reachable.

zfs list -o name,keystatus,encryptionroot

That listing is the real status check. Each independently encrypted dataset, or a hierarchy of datasets inheriting one key configuration, carries an encryptionroot. A locked dataset shows keystatus of unavailable. Only after zfs load-key succeeds does it flip to available and allow a mount.

Where the key lives

• 1.A randomly generated master key encrypts the file data. It is created when the dataset is created and it is stored on disk in wrapped form. It never leaves the pool.
• 2.The user-supplied passphrase, raw key file, or hex key is the wrapping key, and it is held by the operator rather than by the pool. zfs load-key uses it to unwrap the master key into memory.
• 3.With encryption=on the default suite is aes-256-gcm. Passphrases derive the wrapping key through PBKDF2, and the pbkdf2iters property defaults to 350,000 iterations and cannot be set below 100,000.
• 4.zfs change-key is not a way in. It re-wraps the existing master key with a new wrapping key, and it requires the dataset to already be unlocked with the current key loaded.
• 5.While a dataset is unlocked, the key material resides in system memory. On a live machine that is still unlocked, dumping kernel memory to extract key material is theoretically possible. Once the machine crashes, reboots, or loses power, that material is gone and no imaging of the drives brings it back. This is not a service we offer; it is stated here so nobody powers off a still-unlocked box expecting to pick up where they left off.

What a lab can still report

ZFS native encryption protects the data payloads and the file names, not the enclosing filesystem structure. With no key at all, an evaluation can honestly tell an owner what existed: dataset names and the filesystem hierarchy, used capacity along with quotas and reservations, snapshot names and their creation timestamps, and dataset properties such as compression algorithm, record size, cipher, and PBKDF2 iteration count.

File names and file contents inside an encrypted dataset stay obscured. That inventory can tell somebody which of several pools held the data they care about, or that the snapshot they were counting on was never taken.

TrueNAS uses OpenZFS native encryption, not anything proprietary. If a key is set to unlock automatically at boot, it lives in the TrueNAS configuration database on the boot pool. Losing the boot device without a backup of that configuration or a manually exported key file loses the wrapping key, and the data pool stays sealed even though every data drive is healthy.

If you still have the passphrase, the key file, or an exported key, an encrypted pool falls under the TrueNAS and FreeNAS data recovery we do in-house at the Austin, TX lab.

We run this evaluation in-house at the Austin, TX lab with no diagnostic fee, and if the key is gone we say so and charge nothing. No data, no recovery fee applies here in its most literal form. Where a key does exist and the obstacle is pool structure rather than cryptography, the work returns to imaging every member and reconstructing the pool offline from the images.