Enterprise Virtualization Recovery
Proxmox VE Data Recovery
We recover KVM virtual machines and LXC containers from failed ZFS pools, degraded Ceph clusters, and corrupted Proxmox storage backends. qcow2, raw, and zvol extraction. Free evaluation. No data = no charge.
How Proxmox VE Storage Fails and How We Recover It
Proxmox VE stores virtual machines and containers on pluggable storage backends: local ZFS, Ceph (distributed), LVM-thin, NFS, or directory-based storage. When the underlying disks fail, Proxmox loses access to the storage backend and all VMs/containers on it go offline. Recovery requires imaging the physical drives, reconstructing the storage layer (ZFS pool, Ceph object store, or LVM thin pool), and extracting each VM's disk image individually.
Proxmox is increasingly popular for homelab, SMB, and enterprise deployments because it provides KVM virtualization and LXC containers on a Debian Linux base with a web GUI and no license fees. The storage flexibility is a strength for deployment but adds complexity to recovery: a Proxmox cluster might use ZFS on one node, Ceph across the cluster, and NFS for backups. Each backend has different on-disk structures and failure modes.
Proxmox recovery is one branch of our broader enterprise server recovery work, where the storage layer is reconstructed from drive images before any guest data is extracted.
ZFS Pool Failures on Proxmox
ZFS is the default recommended storage backend for Proxmox local storage. Proxmox creates ZFS pools during installation and stores VM disk images as zvols (block devices) and LXC containers as ZFS datasets. For detailed ZFS pool recovery procedures, see our ZFS pool recovery guide.
RAIDZ1/RAIDZ2 Vdev Failures
- RAIDZ1 tolerates one drive failure per vdev; a second failure causes pool FAULTED state and ZFS refuses to import
- RAIDZ2 tolerates two failures per vdev but a third renders the vdev unrecoverable through normal ZFS tools
- ZFS stores metadata (uberblock, spacemap, dnode) redundantly by default; data blocks follow the vdev redundancy level
- We image all drives including failed ones, reconstruct the vdev geometry from ZFS labels (at sectors 0 and end-of-disk), and force-import the pool from images
ZFS Mirror Failures
- Proxmox mirrors store identical copies on two (or more) drives; losing all mirrors in a vdev causes pool failure
- Mirror vdevs are simpler to reconstruct: each drive is a standalone copy of the data; we image the healthiest mirror member first
- If one mirror has bad sectors, we combine data from both mirrors at the block level to produce a complete image
- Boot drives (Proxmox OS) are typically on a separate ZFS mirror; if only the boot mirror fails, VM data on the storage pool is unaffected
If your Proxmox node shows pool imported with errors or refuses to import entirely with I/O errors, see our ZFS pool import I/O error page for the specific failure pattern and recovery approach.
Ceph Cluster Recovery on Proxmox
Proxmox integrates Ceph for distributed storage across cluster nodes. Ceph splits VM disk images (RBD) into 4MB objects and distributes them across OSDs using the CRUSH placement algorithm. When enough OSDs fail that placement groups (PGs) lose all replicas, the affected RBD images become inaccessible.
OSD Failure and PG Recovery
Each OSD manages objects on a local disk (typically a dedicated SSD or HDD per OSD). Ceph uses BlueStore as its default backend on Proxmox 5.x and later, storing data directly on the block device and managing its RocksDB metadata database internally via BlueFS.
When an OSD disk fails, Ceph marks its PGs as degraded and begins replicating data to other OSDs. If the cluster does not have enough surviving replicas to recover, PGs are marked "incomplete" or "unfound."
We image the failed OSD drives, parse the BlueStore on-disk format (or FileStore for older clusters) including the RocksDB metadata, and reconstruct the object-to-PG mapping. Combined with the CRUSH map (stored in the Ceph monitor database on the mon nodes), we can determine which objects belong to which RBD image and reassemble the virtual disks.
Monitor Database Corruption
Ceph monitors (mon) maintain the cluster map, including the CRUSH map, OSD map, and PG map. Proxmox runs monitors on each cluster node by default. If a majority of monitors lose their database (stored as a LevelDB or RocksDB instance), the cluster cannot form a quorum and all storage access stops. We extract the monitor database from each node's mon data directory and reconstruct the cluster map from the most recent consistent copy.
How Does CRUSH Map Reconstruction Recover Ceph OSDs?
CRUSH is deterministic: it maps each object to a placement group (PG) and each PG to a set of OSDs using the CRUSH map. The CRUSH map and the OSDmap live in the Ceph monitor (mon) store, backed by RocksDB. At the storage layer, BlueStore manages the raw block device directly and keeps its own metadata in RocksDB through BlueFS.
Reconstructing a Ceph pool means recovering both halves: the mon-side map that says where objects belong, and the OSD-side BlueStore that holds the object data itself. This extends the high-level Ceph cluster recovery flow above with the object-level mechanics.
OSD-Level Extraction
Low-level OSD recovery runs through ceph-objectstore-tool, and the OSD must be stopped while it runs. Against an image it can export and import a placement group, list the objects in a PG, and dump the op log. Listing the objects in a placement group is the starting point for mapping a PG back to an RBD image:
ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID --op listWhat Incomplete and Unfound Mean
A PG marked incomplete means the cluster has lost all complete replicas or the authoritative logs it needs to service I/O for that PG. A PG marked unfound means the cluster knows the objects exist but cannot locate an authoritative copy. Both states tell us which placement groups need object-level reconstruction from the OSD images rather than a normal Ceph backfill.
If the CRUSH map is lost from the mon store because the RocksDB instance is corrupt, the deterministic object-to-OSD mapping breaks and the objects have to be extracted directly from BlueStore. The order does not change: we image every OSD drive through PC-3000 Portable III first, then reconstruct the mapping in software from those images. The imaging hardware reads the sectors; the CRUSH and BlueStore parsing runs against the image files, never the live OSDs.
Two patterns from forum threads destroy the remnants we reconstruct from. Telling Ceph to force a failing OSD back into the cluster, or running ceph-objectstore-tool --op fix-lost on a drive that is still failing, can overwrite the BlueStore metadata and PG logs that the authoritative-copy reconstruction depends on. We make those decisions against images, after the drives are preserved.
What Happens When Proxmox Loses Quorum and /etc/pve Goes Read-Only?
A pmxcfs or quorum failure is a configuration-plane outage, not a storage-plane data-loss event. The Proxmox Cluster File System (pmxcfs) is a database-driven filesystem that stores the Proxmox configuration and replicates it in real time to every cluster node through corosync.
It mounts via FUSE at /etc/pve and is backed by a SQLite database at /var/lib/pve-cluster/config.db. A copy of the data also lives in RAM, which caps the store at 128 MiB. The corosync configuration that governs quorum lives at /etc/corosync/corosync.conf and is also presented at /etc/pve/corosync.conf.
Read-Only on Quorum Loss
On loss of corosync quorum, pmxcfs goes read-only, so VMs cannot be started or reconfigured even when the underlying ZFS, Ceph, or LVM-thin storage is intact. To regain local write access on a single surviving node, an admin can lower the expected vote count with pvecm expected 1, or, if corosync itself will not start, run pmxcfs in local mode with pmxcfs -l. The guest data on the storage backend never moved; only the config plane locked.
config.db Corruption
If config.db itself corrupts, the cluster loses its VM and container configuration metadata while the raw guest data stays intact but orphaned. Recovery restores /etc/pve and config.db from backup, or parses the SQLite database forensically from an image with sqlite3 corrupt.db .recover >data.sql, never against the live node.
The takeaway for triage is that the VMs are recoverable once the config plane is rebuilt, because the storage plane was never the failure. That is also why the common forum instruction to force quorum and reboot is a hazard: pushing nodes to write conflicting corosync state can split-brain the cluster and corrupt the very config.db you are trying to preserve. We work from an image of the node before touching the live config plane.
LVM-Thin and qcow2 Disk Image Recovery
Proxmox supports LVM-thin as a storage backend for VMs that do not require ZFS checksumming or Ceph distribution. LVM-thin uses a thin provisioning pool on an LVM logical volume, where each VM gets a thin LV.
When the LVM layer itself is the damaged part (volume-group descriptors or PV metadata rather than the thin pool), the recovery path is the same as our dedicated LVM volume-group recovery work, restoring the metadata from a config archive before the thin LVs can be read. Proxmox 7 and later also supports Btrfs as a storage backend; Btrfs is a separate Copy-on-Write-aware job handled through our Btrfs filesystem recovery workflow, which reads older generation roots forensically rather than overwriting them in place.
LVM-Thin Pool Corruption
If the thin pool metadata LV is corrupted (power loss during metadata commit), the entire thin pool becomes inaccessible. We parse the thin pool superblock and space maps from the raw disk image to locate each thin LV's block mapping.
qcow2 Header Corruption
Proxmox uses qcow2 format on directory-based and NFS storage. qcow2 files have a header, L1/L2 mapping tables, refcount tables, and data clusters. If the header or refcount table is corrupted, qemu-img check may fail to repair it. We rebuild the L1/L2 tables and refcount structures from the data cluster layout.
LXC Container Rootfs
LXC containers store their rootfs as a directory, ZFS dataset, or thin LV depending on the storage backend. Recovery extracts the container rootfs from whichever backend was in use. ZFS datasets are extracted as part of the pool reconstruction; LVM-thin LVs are extracted from the thin pool metadata.
Recovery Methodology for IT Administrators
If you are evaluating our capability to handle Proxmox environments, this is the procedure.
Some Proxmox deployments place storage on Linux mdadm software RAID with ZFS or LVM layered on top; those nodes need the array reassembled first through our Linux software RAID (mdadm) reassembly process before the upper storage layer can be reconstructed.
Drive Imaging
Every drive in the Proxmox node (or cluster, if Ceph) is imaged through PC-3000 with write-blocking. For drives with bad sectors, we use head maps to capture healthy sectors first, then revisit damaged areas with aggressive retry parameters. NVMe drives used as ZFS SLOG or Ceph journal/WAL devices are imaged through PCIe adapters.
Storage Backend Reconstruction
For ZFS: we read ZFS labels from each drive image to determine pool geometry (mirror, RAIDZ1/2/3), reconstruct the vdev layout, and import the pool read-only from the images. For Ceph: we parse BlueStore on-disk structures from each OSD image, extract the CRUSH map from the monitor database, and rebuild the object-to-PG-to-RBD mapping. For LVM-thin: we parse the thin pool metadata device to recover the block allocation map for each thin LV.
VM and Container Extraction
KVM VM disk images (qcow2 or raw) are extracted from the reconstructed storage backend. For qcow2 files with backing files (linked clones), we resolve the backing chain and consolidate into a standalone image. LXC container rootfs directories or datasets are extracted as tar archives. Each recovered VM/container is verified by mounting the guest filesystem read-only and checking integrity.
How Do You Recover a Proxmox Backup Server Datastore?
A Proxmox Backup Server datastore is a content-addressed chunk-store, not array redundancy. PBS deduplicates at the application layer: each chunk is named by its SHA-256 checksum and split into directories under .chunks/ using a 4-hex-digit (2-byte) prefix of that checksum.
Per-snapshot index files, .fidx for fixed-size chunks of VM disk images (around 4 MiB each) and .didx for the dynamically sized chunks of pxar file archives, map a snapshot back to the chunks it references. A backup repository is a discrete store; it does not give you a degraded-array fallback the way pool redundancy does.
Missing or Corrupt Chunk
Each chunk carries a SHA-256 name and a trailing CRC-32, so a missing or corrupt chunk breaks the chain of trust for whatever references it. If the manifest and index files survive, only the specific .fidx blocks or .didx files that point at that chunk are affected, not the whole snapshot. We carve the readable chunks and rebuild the index references around the gap.
Index and Catalog Desync
When the .fidx or .didx files corrupt, the snapshot desyncs from the chunk-store and the deduplicated blobs can no longer be reassembled into contiguous VM images or pxar archives. The per-snapshot catalog index is separate: when it corrupts, browsing a restore fails even though the chunks still exist. We reconstruct the mapping from the surviving index structures rather than re-running a verify on the original media.
PBS does not keep a RAM-resident deduplication table the way ZFS does. The chunk directories are preallocated and the index mappings are on-disk, so a PBS datastore does not need a multi-gigabyte dedup table in memory to be read.
If the backing store underneath PBS is ZFS with ZFS deduplication enabled, the ZFS DDT RAM rule applies to that ZFS layer (roughly 5 GB of RAM per 1 TB of deduplicated data, and an oversized DDT hangs zpool import and kernel-panics the host). Running ZFS dedup beneath PBS is redundant: PBS already deduplicates above it, so the only thing the ZFS DDT buys you is the RAM exhaustion.
Every datastore drive is block-level imaged through PC-3000 Portable III before any reconstruction. We never delete index files, re-run a verify, or repair on the original media; deleting a .fidx or .didx file orphans the chunk-store and turns a recoverable snapshot into a manual chunk-carving job. The imaging hardware reads the sectors; the index-to-chunk reassembly runs in software against the image files.
What Is Inside a Proxmox Backup Server Snapshot Manifest?
A PBS snapshot is laid out at <datastore-root>/<type>/<id>/<time>/, and within it index.json.blob is the manifest. Datastore configuration lives at /etc/proxmox-backup/datastore.cfg. Block-based VM backups use fixed-size chunks, typically 4 MiB, mapped by .fidx indexes; file-based pxar backups use dynamically sized chunks mapped by .didx indexes. This builds on the chunk-store basics in the Proxmox Backup Server section above.
Chunk Layout Under .chunks/
A chunk is data plus a trailing CRC-32 and a leading type marker, identified by the SHA-256 of its content and stored under .chunks/ in directories grouped by the 2-byte (4 hex digit) checksum prefix. Those directories, 0000 through ffff, are preallocated at datastore creation, so the backing filesystem must support at least 65538 subdirectories per directory.
Orphaned but Recoverable Chunks
If index.json.blob or the .fidx / .didx index files corrupt, the raw deduplicated chunks survive but become orphaned, and the datastore is unrestorable until the index-to-chunk mapping is rebuilt. proxmox-backup-manager verify checks chunk checksums; rebuilding broken .fidx maps from the surviving chunks is forensic carving against datastore images.
The destructive forum pattern to avoid is deleting the index and re-running a verify on the original media. That can trigger garbage collection, and garbage collection permanently purges the orphaned-but-recoverable chunks that the rebuild depends on. We image every datastore drive through PC-3000 Portable III first and rebuild the manifest-to-chunk mapping in software against the images, so a corrupt index never becomes a permanent loss.
How Do You Resolve a DRBD Split-Brain Without Losing Data?
DRBD is block-level network replication, and split-brain branches the block-level timeline. DRBD mirrors a block device across two nodes in single-primary or dual-primary resource roles. A split-brain happens when replication breaks and both nodes accept writes independently while disconnected, so each node now holds a different version of the same blocks. The userspace tool drbdadm status is how you identify that the nodes have diverged into a split-brain condition.
Why Reconciliation Is Destructive
Resolving a split-brain is inherently destructive: DRBD forces one node to discard its diverged block history and resync from the chosen authoritative node. Any write made to the discarded node during the split is permanently destroyed once the resync runs. That is why we image both nodes through PC-3000 Portable III first, so the timeline DRBD is about to throw away is preserved off-media and can be carved on its own.
Forum Advice That Destroys Data
Do not force primary on both nodes; that tells DRBD to overwrite the block differences and destroys the divergent timeline outright. The same applies to the PBS pattern of "just re-run the backup" or "delete the index and re-verify," which orphans the chunk-store. None of these run on original media in our workflow. We make the destructive decision against image files, after both sides are preserved.
The recovery order is fixed: image both nodes, identify the diverged condition with drbdadm status, determine which node holds the writes worth keeping, then reconstruct from the images. DRBD replication is uptime, not backup; a split-brain proves it, because both replicas can end up holding partial, conflicting copies of the same production data at the same time.
Proxmox VE Recovery Pricing
Same transparent model as every other service: per-drive imaging based on each drive's condition, plus a $400-$800 reconstruction fee covering ZFS pool import, Ceph object reassembly, or LVM-thin parsing. No data recovered means no charge.
| Service Tier | Price Range (Per Drive) | Description |
|---|---|---|
| Logical / Firmware Imaging | $250-$900 | Firmware module damage, SMART threshold failures, or filesystem corruption on individual drives. |
| Mechanical (Head Swap / Motor) | $1,200-$1,50050% deposit | Donor parts consumed during transplant. SAS drives require SAS-specific donors. |
| Storage Reconstruction + Extraction | $400-$800per storage backend | ZFS pool import, Ceph object reassembly, or LVM-thin parsing. Includes VM/container extraction. |
No Data = No Charge: If we recover nothing from your Proxmox environment, you owe $0. Free evaluation, no obligation.
Every Proxmox case is handled in-house at the Austin, TX lab. Single location, no outsourcing, no diagnostic fees. The Proxmox recovery path differs from VMware VMFS and Hyper-V: ZFS pools, Ceph object stores, LVM-thin pools, PBS chunk-stores, and DRBD replicas each carry their own on-disk structures, so we reconstruct the specific backend rather than running a generic VM-extraction pass.
Proxmox VE Recovery; Common Questions
Can you recover a degraded ZFS pool on Proxmox VE?
How do you recover from a Ceph OSD failure in a Proxmox cluster?
Can you recover LXC containers separately from KVM VMs?
My vzdump backup file is corrupted. Can you extract data from it?
Can you restore from a Proxmox Backup Server datastore with corrupted chunks?
How do you resolve a DRBD split-brain without losing data?
Proxmox says it lost quorum and /etc/pve is read-only. Are my VMs lost?
Can you recover a Ceph cluster after the CRUSH map or monitor database is lost?
Should I enable ZFS deduplication under Proxmox Backup Server?
Related services
Need Recovery for Other Devices?
Dell, HP, IBM enterprise servers
VMFS datastores and vSAN
TrueNAS / FreeNAS ZFS pools
Degraded and faulted ZFS pools
VMDK, VHD/VHDX, QCOW2 extraction
RAID 0, 1, 5, 6, 10 arrays
Complete service catalog
Ready to recover your Proxmox environment?
Free evaluation. No data = no charge. Mail-in from anywhere in the U.S.