NVMe SSD Cache Failure Recovery for NAS Arrays

A small M.2 drive fails in a caching slot and a volume built on healthy mechanical disks stops mounting. That is not a coincidence and it is not a parity problem. A read-write cache is the only place the newest writes and the newest filesystem metadata live until they are flushed down to the array, so when it dies the layer above the disks goes with it. We image every member write-blocked, reconstruct the mdadm and LVM mapping from the clones, and extract from the newest state the backing disks actually received. No data = no charge.

START A CASE Mail-In Service

Author01/10

Written by

Louis Rossmann

Founder & Chief Technician

Updated August 2026

9 min read

Looking for your specific platform? This page covers the caching mechanism itself, brand-agnostic. For DSM behaviour and SHR layout see Synology recovery, for QTS and QuTS hero see QNAP recovery, and for pool-level ZFS work see TrueNAS pool import recovery.

Bluf02/10

What happens when the NVMe cache drive in a NAS fails?

An NVMe drive running as a read-write cache holds the newest writes and filesystem metadata before they reach the mechanical array. When it dies, that state is gone, the backing volume is left consistent only with an older point in time, and DSM reports the volume as crashed.

Synology documents this failure on its prosumer models. The DS920+, DS1520+, and DS1621+ all accept M.2 NVMe drives configured as read/write caches.

When one of those drives drops off the PCIe bus while it is holding dirty data, often because a consumer-grade controller panics under sustained write load, the uncommitted writes are permanently lost. What is left underneath is a mechanical HDD array carrying an incomplete Btrfs filesystem, and DSM raises Volume Crashed immediately.

What catches people out is that the array below is fine. In this failure the mechanical members are the healthy part of the system, and the redundancy across them did nothing, because the loss happened one layer above them. That is the clearest demonstration you will get that array redundancy buys uptime rather than data protection.

A read-only cache is a different situation entirely. Losing one of those costs speed and nothing else, since every block it held already existed on the mechanical members.

Cache Modes03/10

Read-only cache loss and read-write cache loss are not the same failure

Before anything else, find out which mode the cache was in. It is the single fact that decides whether you are looking at a performance regression or a recovery case.

Cache mode	What the device holds	What its loss costs	Does the volume still mount?
Read-only cache	Copies of blocks that were already committed to the mechanical members	Speed. Every cached block has an authoritative copy on the array below it	Yes. Remove the device and the volume comes up
Read-write (writeback) cache	Dirty blocks and filesystem metadata that have not been flushed down to the array	The only copy of the newest writes and the newest metadata pointers	No. The volume comes up crashed, unmountable, or missing

A single-device read-write cache has no redundancy of its own. That is the design: a writeback tier exists to acknowledge a write once it lands on the fast device, well before it reaches the array, which is also why one small SSD can take an entire volume offline.

Metadata Gap04/10

Why doesn't replacing the failed cache drive bring the volume back?

A replacement NVMe drive arrives blank. It carries none of the dirty blocks, Btrfs tree roots, or dm-cache mapping metadata that lived on the failed device, so the volume manager still finds a filesystem that references a newer state than anything present on the mechanical members.

The customer assumption runs the other way round. A cache is a copy of something, so swapping the broken copy should restore service.

That reasoning holds for a read-only cache and fails for a writeback cache, because a writeback tier is a staging area the array has not caught up with rather than a copy of it. These are the objects that can exist only on the cache device at the instant it dies:

Btrfs tree roots and generation numbers: Btrfs never overwrites a block in place. Each write produces a new tree generation and a new root pointer, so the superblock on the mechanical disks points at whatever generation last reached them. When the newest roots die with the cache, the disks still hold older, self-consistent trees, and locating the newest surviving one is the whole job.
ext4 journal blocks: On a QNAP QTS volume the filesystem is ext4, and the journal records metadata changes before they are committed to the main filesystem. Journal blocks staged on a writeback cache and never flushed leave the on-disk filesystem describing a directory tree that was already superseded.
dm-cache mapping metadata: The cache pool keeps its own map of which backing blocks are currently held on the fast device and which of them are dirty. With the fast device gone, pvck and lvs report a missing physical volume belonging to the cache pool, and the logical volume will not activate cleanly without it.

A cleaner-policy drain is the normal administrative way to flush dirty cache blocks down to the backing disks before removing a fast device. It cannot run here. The policy needs to read the dirty blocks off the fast device in order to write them down, and the fast device is dead or unreadable, which is why the documented graceful path does not apply to an unplanned failure.

The recoverable data is on the mechanical members, in an older but internally coherent state. Getting to it means reading those disks forensically rather than asking the NAS to reconcile them against metadata that no longer exists.

Stack05/10

What sits under the cache tier on each NAS platform?

Recovery work depends on the stack below the cache, and the stack is not the same across product lines even inside a single brand.

mdadm, LVM, and Btrfs or ext4

Synology Hybrid RAID is not proprietary silicon. It is mdadm aggregating mismatched drive partitions into standard RAID sets, those sets governed by LVM into one volume group, and the result formatted Btrfs or ext4. QNAP QTS uses the same three layers with ext4 on top.

Any Linux workstation reads that stack. The recovery question is not whether the format can be read, it is which generation of the Btrfs metadata on the backing disks is the newest one that still verifies.

Native OpenZFS

QNAP QuTS hero runs native OpenZFS, which replaces the md and LVM stack rather than sitting on top of it. TrueNAS and FreeNAS are OpenZFS exclusively. On these systems the phrase "the cache drive failed" is ambiguous until you know which kind of device it was.

Get that identification wrong and you will either panic over a harmless loss or underestimate a fatal one.

L2ARC: A pure read cache. Every block it holds also exists in the pool, so losing the device costs read performance and nothing else.
SLOG (separate ZFS Intent Log): Safely removable during normal operation. The damage case is narrow and specific: if the SLOG device fails and a sudden power loss catches synchronous writes in flight, those in-flight writes are permanently lost.
Special allocation-class vdev: Not a cache at all, despite living in the same fast-device conversation. It holds filesystem metadata and the deduplication table, so losing it faults the pool. If deduplication was enabled, remember that the deduplication table has to fit in RAM at roughly 5 GB per 1 TB of deduplicated data, or the import hangs and kernel-panics on that hardware.

A pool that will not import at all is a different job from a block-cache case: it turns on uberblock and transaction-group selection rather than mdadm geometry, and that route runs through TrueNAS pool import recovery.

Flash Failure06/10

Why did the cache SSD die in the first place?

Flash devices do not fail the way mechanical members fail, and the difference matters for what can be pulled off the failed device afterwards. An NVMe SSD maps the logical block addresses the host asks for onto physical NAND pages through an internal flash translation layer.

When the controller or the translation-layer metadata corrupts, the host loses that map. The NAND cells can be intact and still holding charge while the drive presents as unreadable, because nothing left on board knows where any given block went.

A small M.2 device in a caching slot also carries the entire write stream of the array, because every write to the volume lands there before it reaches the disks. These are the failure modes that follow:

Sustained-write wear. A writeback tier absorbs the entire write stream of the array, which is a duty cycle consumer M.2 parts are not built around.
Thermal throttling. Poor chassis airflow leaves an M.2 caching device running hot under sustained write load.
Power management IC burnout. The device stops responding to the host entirely.
Controller failure. The drive drops off the PCIe bus mid-operation, which is the specific event that strands dirty blocks.
Self-protective read-only lock. The controller decides its NAND is no longer safe to write and refuses further writes, leaving the array with a cache device it cannot flush or clear.

None of that applies to the hard disks in the bays. Those are mechanical devices with heads and platters, and in a cache-failure case they are typically the healthy part of the system. Keep the two failure vocabularies apart when you are reading forum advice, because procedures written for a dead mechanical member will not help a stranded cache tier.

First Actions07/10

What should you do before anything else?

Power the NAS down. Every minute it stays up is another chance for the operating system, a scheduled scrub, or a helpful wizard to write to the members.
Label the bay order. Write the bay number on each disk as you pull it. Member order is recoverable from the metadata, but having it confirmed removes a variable.
Keep the failed cache device. Even a drive that will not enumerate on your desktop may still image on dedicated hardware, and any dirty blocks recovered from it are blocks that do not have to be written off.
Image the backing members as they are. Sector-by-sector, write-blocked, before any assembly attempt. Every later decision gets made against the clones.

Actions that destroy recoverable state

Do not re-initialise the cache pool. Removing and recreating the cache in the NAS interface tells the volume manager to accept a new mapping, and the older state it would have been reconciled against stops being addressable.

Automated repair wizards. A repair action in DSM, QTS, or any other NAS interface exists to make a volume mountable again, not to preserve what was on it.
Forcing a read-write remount. Mounting the filesystem writable lets it replay and rewrite metadata against the incomplete state, which is the fastest way to lose the older tree roots that recovery depends on.
Running btrfs check --repair or mounting with the recovery option. Both write to disk. Both overwrite the historical copy-on-write generation roots that read-only forensic tools walk. Treat both as destructive commands, because on a cache-desynchronised volume that is what they are.
Letting the unit run a parity rebuild across the mechanical members. A full rebuild reads every surviving member end to end. Consumer NAS drives are rated at one unrecoverable read error per 10¹⁴ bits read, roughly 12.5 TB, so a multi-terabyte rebuild carries a real probability of surfacing a latent unrecoverable sector. If one surfaces, the controller cannot recalculate the missing parity block and halts.
Moving the disks into another chassis and selecting Migrate or Repair. That instructs the new operating system to rewrite the system partitions and force an import, and when the import fails it commonly offers a fresh install that overwrites the data partitions.

Process08/10

How does the lab recover a volume after the cache device fails?

The work splits into two stages that never overlap: hardware images the members, then software reconstructs the array and parses the filesystem. Nothing in the second stage touches your original disks.

Write-blocked imaging of every member

Mechanical members are imaged sector by sector on PC-3000 Portable III, PC-3000 Express, or DeepSpar Disk Imager, which handles bad-sector regions on the member drives. The failed M.2 cache device goes on PC-3000 SSD, which images NVMe and SATA SSD members including NAS cache devices. If a mechanical member turns out to have a head or platter fault it goes through the 0.02 micron ULPA-filtered clean bench first, though in cache-failure cases the mechanical members are usually intact.

Virtual reconstruction from the images

Reassembly happens against the clones. We read the mdadm superblocks to establish chunk size, member order, and role, activate the LVM volume group, and use Data Extractor Express RAID Edition where an array needs virtual reconstruction from member images.

On Btrfs volumes, btrfs-find-root walks the historical generations to find the newest tree root that verifies against the data actually present, and btrfs restore extracts files from that generation. On ext4 volumes the equivalent step works from a valid superblock copy and whatever journal state reached the disks.

Service is nationwide by mail-in, and every stage runs in-house at our Austin, TX lab. One location, and nothing is handed to a franchise or an outside partner. Rossmann Repair Group has been in business since 2008.

The same imaging-first discipline governs every NAS recovery case that comes through the lab, whether the array underneath is mdadm-based or a native ZFS pool.

Pricing09/10

What does cache-failure recovery cost?

Two components make up the total. Each drive that has to be evaluated and imaged carries its own line item at the standard drive recovery tiers, and the array-level work is quoted once. A configuration with a failed NVMe cache device plus a set of mechanical members generates an individual line item for each evaluated drive, plus one array reconstruction fee.

Per-member logical work

Imaging and filesystem-level extraction on a member that reads without physical fault

From $250

Array reconstruction

mdadm, LVM, XFS, EXT4, Btrfs, or ZFS virtual reconstruction from cloned members

$400-$800

Member with physical fault

A mechanical member needing head work before it can be imaged is priced at its own tier

$1,200–$1,500

+$100 rush fee to move to the front of the queue. Donor drives are matching drives used for parts. Typical donor cost: $50–$150 for common drives, $200–$400 for rare or high-capacity models. We source the cheapest compatible donor available.

No Data = No Charge. There is no diagnostic fee, and if we cannot recover usable data you pay nothing.

Faq10/10

NAS cache failure recovery FAQ

My NVMe cache drive died. Can I just put a new one in and rebuild?

A new cache device comes up blank. If the failed drive was configured as a read-only cache, adding a replacement is fine because every cached block still had an authoritative copy on the mechanical members. If it was a read-write cache, the newest dirty blocks and the newest filesystem metadata existed only on the device that died, and no replacement reconstructs them. The volume manager still sees a filesystem pointing at a state the backing disks never received.

DSM says Volume Crashed after the cache dropped out. Is the data on the hard drives gone?

Usually not. Synology DSM raises Volume Crashed when the Btrfs filesystem on the mechanical array cannot be brought up consistently, which is exactly what an NVMe write cache dropping off the PCIe bus produces. The mechanical members are physically healthy and still hold every older Btrfs generation tree. Recovery means reading the newest tree root that actually survived on those disks rather than forcing the current one to mount.

Does losing an L2ARC device on TrueNAS destroy the pool?

No. L2ARC is a read cache, so its contents always exist elsewhere in the pool and losing the device costs performance only. A SLOG device is different: it is safely removable during normal operation, but if it fails and an unclean shutdown catches synchronous writes in flight, those writes are permanently lost. A special allocation-class vdev is different again, because it holds filesystem metadata and the deduplication table, so losing it faults the pool.

Can I pull the hard drives out of the NAS and read them on a Linux PC?

The array itself is standard Linux. Synology SHR is mdadm aggregating mismatched drive partitions into normal RAID sets, governed by LVM, formatted Btrfs or ext4, and QNAP QTS is mdadm plus LVM plus ext4. Nothing about that needs the original chassis. The hazard is what happens next: if you put those disks into another NAS and select Migrate or Repair, the operating system rewrites the system partitions and can end up offering a fresh install over your data partitions.

Should I run the DSM repair wizard or btrfs check --repair first?

Neither. Btrfs never overwrites a block in place; it writes a new version elsewhere and updates a pointer, which is the only reason older, intact generation tree roots still exist on your disks after a cache failure. Running btrfs check --repair or mounting with the recovery option writes to disk and overwrites those historical roots. The read-only forensic path uses btrfs-find-root to locate the newest surviving root and btrfs restore to extract from it.

Is a read-only SSD cache safer than a read-write cache?

For data survival, yes. A read-only cache stores copies of blocks that were already committed to the mechanical array, so the device can be removed at any time without losing anything but speed. A read-write cache is the only home for dirty blocks and metadata that have not reached the backing disks yet, which is why a single-device read-write cache turns one small SSD failure into a volume that will not mount.

The cache SSD is detected but reports zero capacity or refuses writes. Can you image it?

Sometimes. An NVMe controller that has locked itself into a self-protective read-only state, or that has lost its flash translation layer metadata, still enumerates while returning nothing useful to the host. If the device responds at all, it goes on PC-3000 SSD for imaging so whatever dirty blocks remain readable are captured before anything else happens. If it does not respond, the recovery runs entirely from the last self-consistent state on the mechanical members.

How much of my data comes back after a writeback cache failure?

The floor is the last state the mechanical array was internally consistent with, and everything written between that point and the moment the cache died is at risk because it never reached the backing disks. Files that were sitting untouched for months are unaffected. Work in progress during the failure window is the part that may be missing. We give you a file listing before anything ships so you can see exactly what came back.

Do I ship the whole NAS or just the drives?

Ship the drives, labelled with their bay order, plus the failed M.2 cache device if you can reach it. The chassis is not needed because the array metadata lives on the member disks, not in the enclosure. Everything is handled at our Austin, TX lab. There is no diagnostic fee, and if we cannot recover usable data there is no recovery fee.

Data Recovery Standards & Verification

Our Austin lab operates on a transparency-first model. We use industry-standard recovery tools, including PC-3000 and DeepSpar, combined with strict environmental controls to maintain drive integrity. This approach allows us to serve clients nationwide with consistent technical standards.

Validated Clean Zone

Open-drive work is performed in a ULPA-filtered laminar-flow bench, validated to 0.02 µm particle count, verified using TSI P-Trak instrumentation.

Transparent History

Serving clients nationwide via mail-in service since 2008. Our lead engineer holds PC-3000 and HEX Akademia certifications for hard drive firmware repair and mechanical recovery.

Media Coverage

Our repair work has been covered by The Wall Street Journal and Business Insider, with CBC News reporting on our pricing transparency. Louis Rossmann has testified in Right to Repair hearings in multiple states and founded the Repair Preservation Group.

Aligned Incentives

Our "No Data, No Charge" policy means we assume the risk of the recovery attempt, not the client.

Technical Oversight

Louis Rossmann

Our engineers review all lab protocols to maintain technical accuracy and honest service. Since 2008, his focus has been on clear technical communication and accurate diagnostics rather than sales-driven explanations.

We believe in proving standards rather than just stating them. We use TSI P-Trak instrumentation to verify that clean-air benchmarks are met before any drive is opened.

See our clean bench validation data and particle test video

No Data, No Fee

Guarantee

2.49M+

Subscribers

4.9

1,837+ Google Reviews

Since 2008

Established

Repairs on Video

Full Transparency

As Featured In