LVM Recovery

Written by

Louis Rossmann

Founder & Chief Technician

Updated 2026-06-04

Send your drives to our lab and we recover the data in-house. All LVM work is performed at 2410 San Antonio Street, Austin, TX 78705. There is no diagnostic fee, and under no data, no recovery fee, you pay nothing if we cannot extract your volume group. The Rossmann Group has been recovering data since 2008, and LVM cases never leave the building or get outsourced.

What LVM actually is, and why corruption looks like drive failure

The Logical Volume Manager sits between your physical block devices and your filesystem. It groups one or more Physical Volumes (PVs) into a single Volume Group (VG), then carves Logical Volumes (LVs) out of that pooled space. The filesystem, ext4 or Btrfs or XFS, lives on the LV.

None of this is a single contiguous reservation on a disk. It is an abstraction held together entirely by on-disk metadata.

That abstraction is the weak point. When LVM cannot read its own metadata, the kernel reports Couldn't find device with uuid or PV unknown device, and most people assume the platters died. They have not.

The error fires because the LVM module scanned the first sectors of the device and could not locate the LABELONE magic string or the matching PV UUID. A single bad sector at LBA 1, a transient write error, or a dropped mdadm superblock that shifts the partition offset produces the exact same symptom as a dead drive, while the read/write heads, the spindle motor, and the user data area remain healthy.

Misreading that error is how data gets destroyed. People initialize the disk, accept a manager prompt to repair it, or run pvcreate to fix the missing UUID. Every one of those actions overwrites the only surviving map of where the data lives.

How is LVM metadata laid out on disk?

Recovery is only possible because LVM metadata has a predictable structure. Knowing where each piece sits is what separates a clean reconstruction from a guess.

LABELONE label

The entry point to the entire LVM tree, written near the start of the device, typically in the second sector. It begins with the ASCII magic string LABELONE and carries the 32-character PV UUID, the device size, and pointers to the data and metadata areas. If this label is overwritten, the volume becomes inaccessible even though the data extents are untouched.

Volume Group Descriptor Area (VGDA)

The metadata area the label points to. For traditional thick LVM, it is stored as plain, human-readable ASCII text, not a binary superblock. A volume group descriptor for a typical NAS is only a few kilobytes of text.

Metadata ring buffer

The VGDA is a circular ring buffer at the start of each PV. Every configuration change, extending an LV or adding a snapshot, writes a new copy sequentially and wraps around at the end. Several historical copies of the extent map usually coexist on the disk, which is what makes carving an older intact copy possible.

Physical and logical extents

LVM manages fixed-size blocks called extents, not files. Physical Extents (PEs) on the drives map to Logical Extents (LEs) presented to the filesystem. If the metadata that records this mapping is lost, the filesystem above becomes a scattered set of blocks with no order.

What is the difference between thick and thin (dm-thin) LVM recovery?

The recovery path forks completely depending on whether the volume uses traditional thick provisioning or modern thin provisioning. Getting this wrong wastes time and risks the data.

Thick LVM keeps a 1:1 extent map as readable ASCII in the VGDA, so it is highly amenable to vgcfgrestore and manual carving. Thin provisioning (dm-thin), standard on QNAP QTS, does not. A thin pool consists of two hidden devices: a large data device (_tdata) and a small metadata device (_tmeta). The extent map is no longer text. It is a binary B-tree stored inside _tmeta.

When a power loss desynchronizes the transaction ID in _tmeta from the actual data blocks in _tdata, the pool becomes unmountable and standard vgcfgrestore is useless against it. The correct path is to export the binary B-tree to XML with thin_dump, prune the broken transaction IDs or orphaned block references, and rebuild the metadata with thin_repair and thin_restore written to the clone.

Why is most NAS storage actually nested LVM?

The majority of LVM recovery cases do not come from bare-metal servers. They come from NAS appliances, where LVM is one layer in a stack. The vendors do not use a proprietary format. They use mainline Linux mdadm, open-source LVM2, and standard dm-thin. The complexity is the nesting, not secrecy.

Synology SHR slices mixed-capacity drives into uniform partitions, builds multiple mdadm arrays across those slices, and pools every md device into one LVM volume group. The consequence for recovery is strict ordering: the mdadm layer must be forensically reassembled before the LVM metadata above it can even be parsed. Trying to fix LVM headers while the underlying software RAID is still degraded is mathematically futile, because the md device that holds the PV does not yet exist.

How do professional engineers recover an LVM volume?

Our workflow is strictly offline and additive. Every command runs against forensic clones, never the original drives.

1. Write-blocked imaging: Every PV, or every member of the underlying mdadm array, is cloned sector-by-sector with ddrescue or the PC-3000 Portable III. On drives with bad sectors in the metadata region, PC-3000 adaptive read parameters force-read the critical LABELONE label and ring buffer.
2. Read-only inspection: We run pvscan, vgscan, and lvs against the clones to enumerate surviving physical volumes, volume groups, and logical volumes without activating anything.
3. Underlying mdadm assembly: For Synology, QNAP, and Netgear images, we reassemble the md array with mdadm --assemble --readonly first, because LVM sits on the md device and cannot be read until the RAID is presented.
4. Metadata restoration or carving: When an /etc/lvm/archive text file survives, we restore with vgcfgrestore --file <archive> <vgname> against the clone. When the archive is gone, we carve the ASCII VGDA out of the ring buffer by locating the LABELONE string and the VG name in the raw image, then save that text block as a custom restore file.
5. Thin-pool repair when present: For QNAP dm-thin pools, we export the binary B-tree with thin_dump, prune broken transaction IDs, and rewrite it with thin_repair and thin_restore.
6. Activate, mount read-only, export: We activate the VG against the clones, mount the ext4 or Btrfs filesystem read-only, and copy the data to a separate array.

All work is performed in-house in Austin, TX. We use named equipment: PC-3000 Portable III, PC-3000 Express, DeepSpar Disk Imager, and a 0.02 micron ULPA-filtered clean bench. Founded in 2008.

What Should You Avoid Before Sending the Drives?

Most permanent LVM data loss comes from forum advice that treats a logical error as a hardware problem. The following are documented, dangerous patterns we routinely undo, or fail to undo when it is too late.

Running pvcreate to fix a missing PV

Advice to run pvcreate --uuid --restorefile on the original disk is the most destructive of all. If the offset is wrong or the restore file is stale, you overwrite the sector 1 label and the ring buffer, destroying the last entry point to the volume.

Calling the Linux stack proprietary

Labs that claim they reverse-engineer a proprietary QNAP or Synology LVM format are describing a problem that does not exist. The stack is standard mdadm and LVM2. The skill is Linux storage architecture, not a closed-format secret.

Moving the drives into a new NAS chassis

Booting the drives in a replacement NAS can prompt the new firmware to auto-initialize or rebuild the array from corrupted state data, overwriting the user data area. Image first, migrate never.

Running vgcfgrestore on the live, degraded array

A failing device can map writes to the wrong physical sectors or degrade further mid-write. Metadata restoration belongs on a clone, not the original media.

Pricing

LVM recovery is priced per physical volume, multiplied by the number of drives that require imaging and analysis. Standard consumer NAS and server drives use our HDD pricing tiers:

• Logical volume-group and filesystem recovery: From $250
• Firmware repair (drive unrecognized or wrong size): $600–$900
• Head swap (clicking or not spinning): $1,200–$1,500

Helium-filled enterprise drives (8TB and larger Toshiba MG, WD Ultrastar, Seagate Exos) use helium-specific pricing: From $200 through $3,000–$4,500. A multi-bay NAS is priced as the sum of the applicable per-drive tiers.

Rush service adds 100. +$100 rush fee to move to the front of the queue Donor drives are matching drives used for parts. Typical donor cost: $50–$150 for common drives, $200–$400 for rare or high-capacity models. We source the cheapest compatible donor available.

No diagnostic fees. No data, no recovery fee. If we cannot reassemble your volume group and extract the filesystem, you pay nothing for the recovery attempt.

How long does LVM recovery take?

Imaging takes roughly 6 to 10 hours per 4TB drive. A single-drive volume with intact metadata is usually recovered in 1 to 2 business days. A multi-drive NAS stack that needs mdadm reassembly, dm-thin repair, and hex carving of a lost VGDA typically takes 2 to 4 business days. Drives with bad sectors or head degradation require slower bitwise imaging with PC-3000, which adds time.

Frequently asked questions

What does "Couldn't find device with uuid" mean?

The LVM kernel module could not match the LABELONE header or PV UUID in the first sectors of the device. It signals logical metadata corruption or a shifted partition offset, not necessarily a mechanical failure. The data area is usually intact.

Does pvcreate destroy data on a missing physical volume?

It destroys the logical topology. pvcreate overwrites the sector 1 label and writes a blank VGDA into the ring buffer. The data extents may survive, but the map needed to reassemble them is gone. Never run it on original media to fix a missing PV.

How do I recover LVM metadata when /etc/lvm/archive is missing?

Through hex carving. Because the VGDA is plain ASCII text in a ring buffer at the start of the PV, engineers scan a clone for the LABELONE string and the VG name, extract the most recent intact configuration, and use it as a custom restore file.

Why does my QNAP show "Storage Pool Degraded" with healthy disks?

QNAP QTS stacks thin-provisioned LVM over mdadm. A degraded pool with healthy drives usually means a transaction ID mismatch in the dm-thin binary B-tree inside _tmeta, which needs thin_dump and XML repair rather than standard LVM commands.

Can vgcfgrestore fix a Volume Crashed error on Synology?

No. That error means the underlying mdadm RAID has failed, and vgcfgrestore only touches the LVM layer above it. The mdadm array must be imaged and reassembled offline before the LVM metadata can be read.

Is Synology SHR or QNAP LVM a proprietary format?

No. Both use standard Linux mdadm, open-source LVM2, and standard dm-thin. They reassemble on any Linux workstation. The work is Linux storage expertise, not reverse-engineering a closed vendor format.