QNAP Red Light Error Data Recovery

When the partition-1 md9 config database is corrupted after a firmware update, QTS reports the pools as empty. The recovery path bypasses QTS entirely. We never boot QTS. We image every member drive write-blocked first, then work only on the clones, so the originals are never touched.

Assembling the data array directly is the safe move. Rather than a blanket mdadm --assemble --scan (which also pulls in the md9 and md13 system partitions), we target partition 3 specifically:

mdadm -AfR /dev/md1 /dev/sda3 /dev/sdb3 ...

That forces a read-running assembly of the user-data partition only. QNAP QTS arrays carry standard mdadm 1.x superblocks, so they assemble on a vanilla Linux workstation. This is the same standard-Linux-utilities pattern as Synology SHR, not a proprietary controller, and nothing like Drobo's closed BeyondRAID format.

Next, activate the LVM2 volume group with vgchange -ay vg1000 (or vg1 on older models). Here is the real-world caveat competitors omit: vanilla Linux often errors with Unrecognised segment type thick, because QNAP layers a proprietary thin and thick provisioning implementation over LVM2.

Standard Linux thin-provisioning-tools and the stock dm_thin_pool module cannot parse that custom metadata, so a vanilla workstation refuses to activate the volume group.

Reading the logical volume takes recovery software that understands QNAP's LVM2 structures, or moving the array onto a compatible QNAP host.

Finally, mount the ext4 logical volume read-only:

mount -t ext4 -o ro,noload /dev/mapper/vg1000-... /mnt/recover

The noload flag is mandatory alongside read-only. Mounting a previously-crashed QTS ext4 volume on vanilla Linux without it triggers an automatic journal replay, even under -o ro, and that replay permanently alters filesystem metadata.

-o ro,noload skips the journal entirely. QTS never boots in this path; the files come straight off the reconstructed partition-3 array.

For QuTS hero (ZFS) units, this mdadm, LVM2, and ext4 path does not apply. ZFS pools need uberblock location and transaction-group (TXG) reconstruction, a separate toolchain. The filesystem determines the tooling: QTS units take the mdadm path above, QuTS hero units take the ZFS path, and the two are never mixed.

QuTS hero is the other QNAP operating system, & its data partition is a ZFS pool rather than the mdadm, LVM2, & ext4 stack QTS uses. When a QuTS hero unit shows a solid red light because the pool will not import, none of the QTS commands above apply. You do not run mdadm or vgchange on a ZFS member.

The filesystem decides the tooling: QTS takes the mdadm path, QuTS hero takes the ZFS path below, & the two are never crossed.

ZFS keeps a ring of 128 uberblocks in each vdev label. Every uberblock stamps the transaction-group (TXG) number it was written for, & zpool import selects the newest uberblock that still validates.

When the transaction group it points to is corrupted, the import fails, often with no error text at all: the pool simply refuses to come online. The earlier uberblocks pointing at older, consistent TXGs are still sitting in the label ring, intact.

The recovery runs on cloned member images only. We image every drive write-blocked first, then work on the clones, so the original disks are never written to & never see a new-pool, Migrate, or Initialize prompt. On the images, we enumerate the vdev labels & their valid uberblocks with zdb -lu to identify the newest intact TXG:

zdb -lu /dev/sdX3

With the newest consistent transaction group identified, we force the pool to import from that TXG instead of the corrupted head, read-only, against the cloned set:

zpool import -o readonly=on -T <txg> <pool>

One constraint decides whether the pool will import on a given machine at all: deduplication. If dedup was switched on, the Deduplication Table (DDT) has to fit in RAM before zpool import can finish, & the rule of thumb is roughly 5 GB of RAM for every 1 TB of deduplicated data.

When the DDT is larger than physical RAM, the import hangs & can kernel-panic the host before the red light ever clears. The pool stays unimportable on that hardware until the RAM is increased.

ZFS dedup is not free: it is paid for in RAM at import time, which is exactly when you can least afford a host that cannot mount the pool.

The ZIL Power-Loss Window

The transaction-group rollback above answers a metadata-corruption question: the head TXG no longer validates, so we import read-only from an older, intact one. A power cut raises a second, narrower question about the writes that were in flight at the instant the unit lost power. That is where the ZIL comes in.

The ZIL, or ZFS Intent Log, is how QuTS hero honors synchronous writes. When an application asks ZFS to confirm a write is durable before it continues, ZFS records that write to the ZIL & only then acknowledges it, rather than waiting for the next full transaction group to commit to the pool.

By default on QuTS hero the ZIL lives inside the pool itself, on the same vdevs as the data, not on a separate device. A dedicated log device, or SLOG, is an optional add-on for ZFS in general & is not the QuTS hero default; the in-pool ZIL is what you are working with on a standard unit.

In the ordinary case there is nothing to lose here. When the members are clean & the head TXG still validates, the next zpool import replays the ZIL automatically, & the acknowledged synchronous writes are folded into the pool.

The loss window only opens on the rollback path described above: when the head TXG is corrupted & we import from a prior intact TXG, every transaction newer than the one we choose is discarded by design, including any synchronous writes the ZIL had staged after it. That is the trade for a pool that imports at all.

The label dump from zdb -lu gives us the uberblock ring & TXG status, but it does not contain the ZIL header. The ZIL log-chain header lives in the dataset objset, not the vdev label, so ZIL state is read in a separate dataset-level query rather than from the label dump. We settle the uberblock & TXG question first, then evaluate the ZIL log-chain on the same cloned images when deciding which transaction group to import from.

The practical takeaway is that the damage is bounded. Files that were fully committed to a transaction group before the power event are intact & come back with the pool.

Only the synchronous writes that were acknowledged in the ZIL but not yet folded into a committed TXG sit inside the discard window, & that window is the writes from the power event itself, not the contents of the pool. A QuTS hero unit that lost power is, in the overwhelming majority of cases, a pool we import from a prior good transaction group with your data behind it.