NAS and Server Recovery
TrueNAS / FreeNAS Data Recovery
We recover ZFS pools from degraded and faulted TrueNAS and FreeNAS systems. Vdev reconstruction, dataset extraction, GELI decryption (with your key), and iXsystems hardware failures. Free evaluation. No data = no charge.
How TrueNAS ZFS Pools Fail and How We Recover Them
TrueNAS (and its predecessor FreeNAS) uses ZFS as its sole filesystem. ZFS pools consist of one or more vdevs (mirror, RAIDZ1, RAIDZ2, or RAIDZ3), each containing multiple physical drives.
When enough drives in a vdev fail that ZFS can no longer reconstruct missing data from parity, the pool status changes to FAULTED and ZFS refuses to import it. Recovery requires imaging every drive in the pool, reconstructing the vdev geometry from ZFS label data, and force-importing the pool from images to extract datasets and zvols.
ZFS checksums every block of data and metadata using fletcher4 by default (SHA-256 is the default only when deduplication is enabled). This means ZFS can detect silent corruption that traditional RAID controllers miss.
The downside: when ZFS detects a checksum mismatch and cannot correct it from parity, it returns an I/O error rather than silently serving corrupt data. This is the correct behavior, but it means scrub errors on a degraded pool are a warning sign that more data may become inaccessible if another drive fails.
Common TrueNAS Failure Scenarios
zpool import, or HBA and backplane faults that disconnect healthy drives. Each member is imaged first, then reconstructed from the drive images.RAIDZ1 Double Drive Failure
RAIDZ1 tolerates one drive failure. A second failure in the same vdev causes the pool to fault. If the second drive developed bad sectors gradually (common with same-batch drives of the same age), the pool may have been serving data with correctable errors before the complete failure.
Failed Resilver
Resilvering (ZFS rebuild) writes to all surviving drives in the vdev. If a surviving drive has bad sectors in the areas being resilvered, ZFS cannot complete the rebuild and may fault the pool. A failed resilver is the most common TrueNAS failure scenario we see.
Controller or Backplane Failure
iXsystems TrueNAS appliances and custom builds use SAS/SATA HBAs or RAID controllers in IT mode (passthrough). If the HBA or backplane fails, all drives disconnect simultaneously. The drives are healthy; only the connectivity is lost. Recovery involves imaging the drives through a known-good HBA.
Accidental Pool Destruction
Running zpool destroy or zpool labelclear wipes the ZFS labels from each drive. If no new data has been written, the uberblocks and metadata trees are still on disk. We scan for historical uberblocks and reconstruct the pool from the most recent valid transaction group.
Scrub Errors Accumulating
ZFS scrubs verify every block checksum on every drive. If scrub reports uncorrectable errors (cksum column in zpool status), data blocks are failing and ZFS cannot reconstruct them from parity. This is a precursor to pool failure if another drive drops.
Boot Drive Failure
TrueNAS CORE and SCALE install the OS on a separate boot pool (typically a USB drive or small SSD mirror). If only the boot drive fails, the data pool is unaffected. Reinstalling TrueNAS on a new boot drive and reimporting the pool restores access. We see cases where admins inadvertently destroy pool metadata during the reinstall.
DDT RAM Exhaustion During Pool Import
Enabling ZFS deduplication creates a Deduplication Table (DDT) that maps every unique data block to its physical location. The DDT must fit in RAM for the pool to import. The rule is approximately 5 GB of ECC RAM per 1 TB of deduplicated data. When the DDT exceeds physical RAM, `zpool import` attempts to read the table from disk into memory, causing the host to kernel-panic with `VERIFY(ddt_object_remove(...) == 0) failed` or `fork_trampoline()` exceptions. Disabling deduplication later does not shrink the existing DDT; the pool remains unimportable on that hardware until RAM is upgraded or the drives are moved to a recovery workstation with sufficient memory.
ZFS Pool Import I/O Errors
`zpool import` I/O errors indicate ZFS detected checksum mismatches or unreadable sectors it cannot reconstruct from parity. This happens during resilver when a surviving drive has bad sectors in the area being rebuilt, or after power loss corrupts the active uberblock. ZFS maintains 128 uberblocks in a ring buffer on each drive. We locate historical uberblocks and attempt import from an earlier Transaction Group using `zpool import -T <txg_id>`, rolling the pool back to the last consistent state. Any data written after that TXG is lost, but the pool itself mounts. We image all drives before attempting any import command so the originals are never modified.
When SLOG or L2ARC Cache Drives Fail
A failed L2ARC (read cache) SSD doesn't cause data loss. ZFS treats L2ARC as a disposable performance layer; removing a dead cache drive & reimporting the pool restores full access with only a temporary read performance drop.
SLOG (ZFS Intent Log) failures are different. The SLOG is never read from during normal operations; it acts purely as a persistent backup for in-memory synchronous transactions. If the SLOG device dies with uncommitted synchronous writes, ZFS refuses to import the pool because the intent log is incomplete.
Consumer SSDs used as SLOGs are the most common culprit. They lack Power Loss Protection (PLP), meaning they have no supercapacitor to flush volatile DRAM cache to NAND during a sudden outage. These drives often lie to ZFS about write commitment, reporting data as safely stored while it still sits in volatile cache.
Drives with Phison S11 controllers (like the Kingston A400) can enter a firmware fault state where the SSD identifies as "SATAFIRM S11" with an incorrect capacity instead of presenting normally.
We bypass the missing log device with zpool import -m -f, which sacrifices the last pending transaction group to mount the pool from the healthy data drives. The lost writes are limited to whatever synchronous I/O was in flight at the moment of failure.
If your pool won't import after a cache or log drive failure, NAS recovery follows the same imaging-first approach we use across all ZFS systems. We image all pool members through PC-3000 with write-blocking before attempting any import commands, so the originals are never modified. No data recovered means no charge.
OS-Layer Failure vs Data-Pool Failure Triage
The single most common misdiagnosis we see on TrueNAS is treating a non-booting appliance as a lost array. The WebUI is the admin's primary window into the system, so when the middleware fails to start and shares drop offline, the box looks dead. In these three scenarios the ZFS data pool is untouched.
Before running any pool command, image every member so the originals are never modified, then confirm which layer actually failed. See our ZFS pool recovery guide for the data-layer path.
Bad Update: Boot Environment Rollback
TrueNAS stores ZFS boot environments (BEs) on the boot-pool, separate from the data pool. An update creates a NEW boot environment and leaves the previous one intact, so a failed upgrade is reversible without touching data. TrueNAS CORE (FreeBSD) manages BEs with the beadm / bectl CLI and the System > Boot web UI; TrueNAS SCALE (Debian Linux) manages them through the System Settings > Boot web UI. Activate the prior BE, reboot, and the working OS returns with the data pool never modified.
config.db Corruption
TrueNAS stores SYSTEM configuration (users, shares, network, services) in a SQLite database at /data/freenas-v1.db. A corrupt config.db stops the middleware and WebUI from starting, so shares go offline and the box looks dead, but the ZFS data pool is untouched and importable on any current-OpenZFS system. Restore a saved config backup or reset the config, then re-import the existing pool through the TrueNAS WebUI or API. config.db holds configuration, not file data.
Feature-Flag Mismatch After Upgrade
A TrueNAS upgrade can enable newer OpenZFS pool feature flags, for example block_cloning, introduced in OpenZFS 2.2.0. Once a pool has a newer read-write feature active, an older OpenZFS release cannot import it read-write, including the release inside a previous boot environment. This presents as "pool won't import," but the data is intact. The fix is a version match: import under an OpenZFS at least as new as the pool's flags. This is not pool reconstruction.
The Feature-Flag Rollback Trap After zpool upgrade
Boot environment rollback is the correct fix only when the OS broke but the pool was never upgraded. It is the wrong move once zpool upgrade has run on the data pool. OpenZFS feature flags cannot be disabled once they are enabled. If you already ran zpool upgrade and then roll BACK to an older boot environment, that older OpenZFS release can no longer import the pool read-write, and the pool becomes unimportable on that environment.
The resolution for an already-upgraded pool is to run an OpenZFS version at least as new as the enabled flags, not to roll the BE back. The data is present on disk the whole time; the barrier is purely a version match at the import layer.
When the pool will not mount and the version history is unclear, we image every member and reconstruct the import path on a workstation with a current OpenZFS build. That is server recovery work, imaged first through PC-3000 with write-blocking so the originals are never modified. No data recovered means no charge.
Why Does the TrueNAS SCALE Apps Layer Complicate ZFS Pool Recovery?
TrueNAS SCALE embeds a container-apps framework and creates a dedicated system dataset on the data pool you select for Apps. When Apps share the pool that holds your files, heavy container lifecycle activity leaves its footprint in the same on-disk object tree a recovery has to walk; Apps can also be assigned to a separate dedicated pool, which keeps that footprint off your data pool.
This is a SCALE-only concern. TrueNAS CORE runs on FreeBSD and has no Apps engine at all; it uses iocage jails and plugins, and TrueNAS MINI units running CORE have no Apps layer either. Get the version boundary right before you touch the pool.
K3s Era: SCALE 24.04 and Earlier
Through TrueNAS SCALE 24.04 (Dragonfish) and earlier, the Apps engine was K3s, an embedded Kubernetes distribution. It created the ix-applications dataset on the chosen data pool. K3s also provisions child ZFS datasets for persistent volumes through the OpenEBS ZFS LocalPV provisioner (zfs-localpv), so each PVC becomes another dataset under that tree.
Docker Era: SCALE 24.10 and Later
TrueNAS SCALE 24.10 (Electric Eel) replaced K3s with Docker and migrated to a new ix-apps dataset. A box on 24.10 or later runs Docker with ix-apps, not K3s with ix-applications. Identify which engine and dataset name applied to the box before reconstruction so the orphaned app datasets are matched to the right layout.
TrueNAS CORE: No Apps Layer
TrueNAS CORE (FreeBSD) has no K3s, no Docker, and no ix-applications or ix-apps dataset. Containerized workloads on CORE run in iocage jails and plugins. If the box in front of you is CORE, this failure path does not apply and the pool has no Apps system dataset to audit.
How Container Churn Complicates an Offline Import
When Apps are assigned to the pool that holds your files, the apps dataset and its container and PVC child datasets share that pool, so container lifecycle activity runs alongside your files. Catalog syncs, Helm and chart installs, and app upgrades generate heavy small-write activity and frequent dataset create and destroy operations. That raises dirty-data and transaction group pressure and leaves behind orphaned child datasets and snapshots in the pool.
After a power loss or an Apps-heavy workload, a pool that will not import has to have the apps dataset and its orphaned child datasets and snapshots audited during forensic import. They can complicate or slow an offline import, but they are an interference context, not a destroyer of user datasets. Your user data datasets are intact and separable from the ix-applications or ix-apps hierarchy.
On a live TrueNAS appliance, re-import the pool through the TrueNAS WebUI or middleware API, not a bare CLI zpool import. A bare CLI import on the running appliance bypasses the middleware and can leave the appliance's view of the pool inconsistent with what the WebUI expects. See our ZFS pool recovery guide for the data-layer path, and our Proxmox VE recovery page for another host that layers container storage on ZFS.
In the lab we image every member first with write-blocking, then reconstruct the pool from the images. Orphaned ix-applications or ix-apps child datasets and snapshots are skipped or exported separately so they do not block extraction of the user datasets. All work is performed in-house at our Austin, TX lab, imaged through PC-3000 with write-blocking so the originals are never modified. No data recovered means no charge.
ZFS On-Disk Structures Relevant to Recovery
zpool import -T <txg_id> when the active uberblock is corrupt.Understanding ZFS internals is required for targeted recovery. For a deeper technical treatment, see our ZFS pool recovery guide.
ZFS Labels and Uberblocks
- Each drive has four ZFS label copies: two at the beginning (L0, L1) and two at the end (L2, L3) of the device
- Labels contain the pool name, GUID, vdev tree, and an array of 128 uberblocks (root pointers to the ZFS object tree)
- The uberblock with the highest transaction group (txg) number is the most recent consistent state of the pool
- If labels are corrupted, we scan for uberblocks at known offsets across the drive to find a valid root pointer
MOS, DSL, and Dataset Objects
- The Meta Object Set (MOS) is the root of the ZFS object tree; the uberblock points to it
- The Dataset and Snapshot Layer (DSL) tracks all datasets, zvols, and snapshots in the pool
- Each dataset has its own object set containing dnodes (ZFS inodes) that map files to their on-disk block pointers
- If the MOS is damaged, we traverse the block pointer tree manually using known dnode sizes and indirect block structures
Deduplication Tables, TXG Rollback, and Import Mechanics
When ZFS deduplication is enabled, every unique data block gets an entry in the Deduplication Table (DDT). The DDT is stored on disk as a ZFS internal object but must be loaded into RAM for the pool to import. On a highly deduplicated pool, the DDT can grow to tens of gigabytes.
If the DDT exceeds physical RAM, `zpool import` attempts to swap the table to disk, cratering performance and eventually kernel-panicking the host. The only recovery paths are to install enough RAM to hold the DDT or to move the drives to a workstation with sufficient memory.
ZFS pools are transactional; every change is batched into a Transaction Group (TXG). The uberblock points to the latest valid TXG. ZFS maintains 128 uberblocks in a ring buffer on each drive. If the active uberblock or the Meta Object Set it points to is corrupted, the pool refuses to import.
In recovery, we scan the uberblock ring for the most recent valid root pointer and force import from an earlier TXG using zpool import -T <txg_id>. This rolls the pool back to a consistent state, sacrificing any writes committed after that TXG. For pools with corrupted spacemaps, we traverse the block pointer tree manually using zdb to locate dataset objects and extract data without a full pool import.
dRAID Vdevs: Distributed Parity, Fixed Stripe Width, and Sequential Resilver
dRAID is an OpenZFS vdev topology that distributes parity and spare capacity across every child drive instead of reserving one idle physical hot spare. It shipped in OpenZFS 2.1.0, so TrueNAS builds running OpenZFS 2.1 or newer can create dRAID vdevs. Recovery starts the same way every other pool does, with sector-by-sector imaging of every member.
The geometry is written with suffix tags, not bare positional numbers: draid[parity]:[data]d:[children]c:[spares]s. A vdev created as draid2:4d:11c:1s carries parity level 2, four data devices per redundancy group, eleven total children, and one distributed spare. That configuration string is parsed at pool-create time; the resulting geometry is recorded in each member's vdev label and read back with zdb -l as discrete geometry fields, not as the string you typed.
dRAID uses a fixed stripe width and pads with zeros as necessary, so the minimum allocation size rises with the data-device count. With eight data devices and 4 KiB sectors, the minimum allocation is 32 KiB. That padding is what makes a fully sequential resilver possible: the rebuild reads across the children and completes faster than a RAIDZ healing resilver, at the cost of higher I/O load on the entire array. OpenZFS exposes zfs_rebuild_vdev_limit for the maximum bytes in flight per leaf vdev during a sequential resilver, and zfs_rebuild_scrub_enabled to scrub automatically once the sequential resilver finishes. The scrub is the pass that verifies the blocks after the rebuild.
Fault tolerance equals the parity level P. Losing more than P children before the rebuild finishes exceeds the vdev's fault tolerance and the data in that vdev is gone. Distributed spares shorten the rebuild window; they do not make the pool immune to simultaneous hardware failure, and they do not change the rule that imaging with write-blocking comes before any zpool import or forced resilver on a pool you cannot afford to lose.
A declustered layout is more work to reconstruct offline, not less. The fixed stripe width removes the variable-width problem that makes RAIDZ block mapping awkward, but the parity groups and the spare capacity are spread across all of the children, so knowing the member order is not enough to locate a block. The geometry has to be recovered from the labels before anything can be mapped. dRAID is a vdev-level topology, so a dRAID pool carries the same deduplication table RAM overhead and follows the same zpool import -T <txg_id> rollback path as a RAIDZ or mirror pool. Related failure detail lives on our ZFS pool import I/O error page.
- Image every member of the dRAID vdev sector-by-sector with write-blocking on PC-3000 hardware, including the members that have already faulted, before any import or resilver attempt.
- Read the geometry fields back from each member's vdev label with
zdb -lagainst the image files: parity level, data devices per redundancy group, child count, and distributed spare count. - Reconstruct the vdev in software against the images, never against the live members, so the original drives are read-only for the entire job.
- Locate a valid uberblock and import read-only from the images, rolling back with
zpool import -T <txg_id>when the active uberblock is corrupt. - Extract datasets and zvols to customer-supplied destination media. All of this runs in-house at the Austin, TX lab, and no data recovered means no charge.
GELI Encryption and TrueNAS SCALE Native Encryption
TrueNAS CORE uses GELI (FreeBSD disk encryption) for encrypted pools. GELI operates below ZFS: it encrypts entire disk partitions using AES-XTS-256 before ZFS accesses them. Without the GELI master key (or the passphrase/keyfile used to derive it), the raw disk data is indistinguishable from random bytes.
- GELI key storage: TrueNAS CORE stores the GELI recovery key in the system dataset (on the boot pool). If the boot pool is lost and you did not export a backup of the key, the encrypted pool is unrecoverable. Always export and store the GELI recovery key offsite.
- TrueNAS SCALE encryption: SCALE uses ZFS native encryption, which encrypts at the dataset or zvol level. The encryption key is stored in the ZFS metadata itself (protected by a user passphrase or keyfile). This is a different mechanism than GELI: we do not need FreeBSD GELI tools, but we do need the ZFS encryption passphrase or keyfile.
- Recovery with key material: If you provide the correct key (GELI master key, GELI recovery key, or ZFS encryption passphrase), we attach the encryption layer to the drive images and import the pool normally. The decryption happens on our workstation; your key material is used in memory and not written to our storage.
Recovery Methodology for IT Administrators
1. Drive Imaging
Every drive in the TrueNAS system is imaged through PC-3000 with write-blocking. SAS drives (common in iXsystems rackmount appliances) are imaged via SAS HBAs.
For drives with bad sectors, we capture healthy sectors first using head maps, then retry damaged areas. We also image the boot drive(s) to extract TrueNAS configuration, GELI keys, and pool history.
2. ZFS Pool Reconstruction
We read ZFS labels from each drive image to determine pool geometry: vdev type (mirror, RAIDZ1/2/3), member ordering, and ashift (sector size alignment). The pool is imported read-only from the images.
If the pool refuses to import, we locate historical uberblocks and attempt import from an earlier transaction group. For pools with corrupted spacemaps, we traverse the block pointer tree manually to locate dataset objects.
3. Dataset and Zvol Extraction
Individual datasets (file-level shares), zvols (block-level iSCSI targets or VM storage), and snapshots are extracted from the reconstructed pool. Each dataset is verified by checking file counts and sizes against what ZFS metadata reports. For zvols used as VM storage, we also verify the guest filesystem integrity (NTFS, ext4, XFS).
How Much Does TrueNAS Recovery Cost?
Same transparent model: per-drive imaging based on each drive's condition, plus a$400-$800 ZFS pool reconstruction fee that includes dataset extraction. No data recovered means no charge.
| Service Tier | Price Range (Per Drive) | Description |
|---|---|---|
| Logical / Firmware Imaging | $250-$900 | Firmware module damage, SMART threshold failures, or filesystem corruption on individual pool members. |
| Mechanical (Head Swap / Motor) | $1,200-$1,50050% deposit | Donor parts consumed during transplant. SAS drives (common in iXsystems appliances) require SAS-specific donors. |
| ZFS Pool Reconstruction | $400-$800per pool | Vdev reconstruction, pool import, dataset/zvol extraction. Includes GELI or ZFS native decryption if key provided. |
No Data = No Charge: If we recover nothing from your TrueNAS system, you owe $0. Free evaluation, no obligation.
Before sending drives: export your GELI recovery key (TrueNAS CORE) or ZFS encryption passphrase (TrueNAS SCALE) if your pool is encrypted. Without the key, encrypted data is unrecoverable.
TrueNAS Recovery; Common Questions
My TrueNAS pool shows FAULTED and will not import. Can you recover the data?
Should I try to replace a failed drive and resilver before contacting you?
My TrueNAS pool uses GELI encryption. Can you still recover the data?
Does TrueNAS CORE vs SCALE matter for recovery?
Why did my TrueNAS pool fault during a RAIDZ expansion with a new drive?
Why won't my TrueNAS SCALE pool import after using Apps (K3s/Docker)?
Can you recover VMware or Proxmox virtual machines from a failed TrueNAS iSCSI Zvol?
Why does my TrueNAS system kernel-panic when I try to import the pool?
Can a failed SLOG SSD destroy data that was already written to the pool?
What do ZFS pool import I/O errors mean, and can they be fixed?
My TrueNAS SCALE dataset is locked with ZFS native encryption. Can you unlock it?
My zfs send | zfs receive replication died mid-stream. Did I lose data?
My TrueNAS pool faulted when the SAS HBA failed, but the drives are fine. Can you recover it?
My TrueNAS won't boot or the WebUI is down after an update. Is my data gone?
Why won't my TrueNAS pool import after I upgraded TrueNAS?
Can you recover a TrueNAS pool built on dRAID vdevs?
Related services
Need Recovery for Other Devices?
Data Recovery Standards & Verification
Our Austin lab operates on a transparency-first model. We use industry-standard recovery tools, including PC-3000 and DeepSpar, combined with strict environmental controls to maintain drive integrity. This approach allows us to serve clients nationwide with consistent technical standards.
Open-drive work is performed in a ULPA-filtered laminar-flow bench, validated to 0.02 µm particle count, verified using TSI P-Trak instrumentation.
Transparent History
Serving clients nationwide via mail-in service since 2008. Our lead engineer holds PC-3000 and HEX Akademia certifications for hard drive firmware repair and mechanical recovery.
Media Coverage
Our repair work has been covered by The Wall Street Journal and Business Insider, with CBC News reporting on our pricing transparency. Louis Rossmann has testified in Right to Repair hearings in multiple states and founded the Repair Preservation Group.
Aligned Incentives
Our "No Data, No Charge" policy means we assume the risk of the recovery attempt, not the client.
Technical Oversight
Louis Rossmann
Our engineers review all lab protocols to maintain technical accuracy and honest service. Since 2008, his focus has been on clear technical communication and accurate diagnostics rather than sales-driven explanations.
We believe in proving standards rather than just stating them. We use TSI P-Trak instrumentation to verify that clean-air benchmarks are met before any drive is opened.
See our clean bench validation data and particle test videoReady to recover your TrueNAS system?
Free evaluation. No data = no charge. Mail-in from anywhere in the U.S.