VMware and Hyper-V Recovery from Corrupted RAID Arrays

Why is a Failed Array Under a Datastore Two Stacked Recovery Problems?

The nesting-doll structure forces the order of operations. The host has no way to read the .vmdk descriptor, the VMFS file descriptor heap, or the NTFS run list for a .vhdx while the block device underneath is incomplete. Tools that operate at the VM layer assume a stable block device beneath them. On a degraded array that assumption is false, and running them can write damage into the only surviving copies of the container headers.

Why Should You Not Rebuild the Degraded Array First?

RAID 5 tolerates one failed member by recomputing its contents from distributed parity. To rebuild that member the controller must read every sector of every surviving drive. On modern multi-TB members the probability of encountering an unrecoverable read error across a full-surface read is significant, and the chance rises with array capacity.

If the controller hits a URE while the array is already degraded, it can stop recalculating parity for the affected stripe. A legacy or low-end controller treats that as a second member failure and aborts the rebuild; the array crashes. Modern Dell PERC and LSI/Broadcom MegaRAID may puncture the affected stripe and continue, losing only that stripe, but a punctured stripe can land on VMFS metadata or a .vmdk or .vhdx flat extent.

The rebuild read also stresses aging members mechanically. A same-batch survivor that is marginal can fail outright during the sustained read, turning a single-member fault into a double-member fault on a RAID 5 that cannot tolerate it.

For irreplaceable, unbacked data on a degraded array we do not gamble on a live rebuild. Rebuilding a degraded array forces parity recalculation across the surviving members, so each member is imaged read-only first and the array is reconstructed against the copies. RAID is fault tolerance against drive failure; it is not a backup, and a failed array with no separate backup is exactly the case this workflow exists for.

How Does the VMware Leg Differ from the Hyper-V Leg?

This page owns the compound array-failure case. The deep container internals, VMFS metadata structures, VHDX Block Allocation Table layout, snapshot and checkpoint chain consolidation, and vSAN or Storage Spaces Direct specifics live on the two dedicated pages above. Send a case here when the array under the datastore has failed and must be reconstructed before any of that container work can begin.

How Do You Recover a VM Datastore on an iSCSI LUN When the NAS RAID Failed Beneath It?

An iSCSI LUN that an ESXi or Hyper-V host treats as a raw SAN block device is not a physical partition on the NAS. On Synology an Advanced LUN is a large sparse file, named in an EP_DAT_00000 style, inside a hidden @iSCSI directory on the underlying Btrfs or ext4 volume. Those volumes sit on standard Linux storage: Synology Hybrid RAID is mdadm plus LVM plus Btrfs or ext4 and reassembles on a vanilla Linux workstation.

QNAP QTS is mdadm, LVM, and ext4 with its config database on the first partition; a QTS block LUN is a logical volume carved from an LVM thin pool, while a file-based LUN is an image file on the ext4 volume. TrueNAS is the exception: it runs OpenZFS, so a TrueNAS iSCSI LUN is a zvol exposed at the block layer rather than a file inside a dataset.

That structural difference dictates the workflow. The container file cannot be addressed until the host filesystem under it is readable, and the host filesystem cannot be read while the NAS array is failed.

On a Synology unit running Btrfs the underlying filesystem is never repaired in place; btrfs check --repair can permanently destroy the copy-on-write tree, so we traverse generation roots with btrfs-find-root and extract non-destructively with btrfs restore; a QNAP QTS ext4 volume is checked read-only with e2fsck -n and never written to during recovery. The LUN-level @iSCSI container detail is covered on our iSCSI LUN recovery page.

1. Image every NAS member read-only. Each drive is captured through DeepSpar Disk Imager, PC-3000 Portable III, or PC-3000 Express; the imaging hardware reads sectors and does not assemble the array.
2. Reconstruct the NAS array virtually. For mdadm-based Synology or QNAP we assemble with mdadm --assemble --readonly and activate LVM; for TrueNAS we zpool import read-only with txg rollback if needed. Reconstruction is a software stage run against the images.
3. Parse the host filesystem. Btrfs, ext4, or ZFS is read non-destructively from the reconstructed image to reach the LUN container: the hidden @iSCSI directory on Synology, the thin-pool logical volume or image file on QNAP QTS, or the zvol on TrueNAS.
4. Extract the LUN container intact. On Synology the sparse EP_DAT_00000 file is copied out whole; on QNAP QTS the block LUN is read from its LVM thin-pool logical volume or the file-based LUN image is copied out; on TrueNAS the zvol is read at the block layer.
5. Mount the container and parse the inner filesystem. The LUN image is attached as a loopback block device and the VMFS or NTFS the initiator wrote inside it is parsed read-only, after which the .vmdk or .vhdx containers are extracted exactly as in the single-array case.

Why Doesn't SMB3 Continuous Availability Cover a Failed Array Behind the Scale-Out File Server?

Hyper-V doesn't need local storage. VM configuration, the VHD and VHDX files, and snapshots can all sit on an SMB 3.0 file share reached by UNC path, rather than on a local CSV volume or inside an iSCSI block LUN.

In enterprise deployments that share is published by a Scale-Out File Server running on Windows Server Failover Clustering: every node in the cluster accepts SMB connections and routes I/O to shared backend storage, commonly a Storage Spaces pool built from a shared SAS JBOD, or a shared hardware RAID array presented to every node. That backend is the third storage topology on this page, and it is the one that changes which drives we ask you to ship.

Continuous Availability and persistent handles

The Hyper-V host requests a persistent file handle from the file server. If the network connection is lost or a file server node fails, the client reconnects to another node in the cluster and keeps reading and writing without taking the VMs offline. The handle is designed to survive a node. It has no mechanism for surviving the disks.

The SMB Witness service

Witness runs alongside SMB3 and lets the cluster actively notify a client that a node has gone offline, so the client moves its persistent handles to a surviving node instead of sitting through TCP timeouts. It reports the state of the clustered network resource. Nothing in that signal describes the health of the array underneath the node.

SMB Direct and RDMA

SMB Direct bypasses the host networking stack and places data directly into memory, which cuts latency and CPU overhead on the transport. It changes nothing about the dependency on the physical storage. When the backend array degrades, RDMA carries the I/O hang as efficiently as it carried the throughput.

The distinction that decides the case is node failure versus storage stall. When the backend RAID array or the SAS JBOD under the file server degrades, stalls, or fails, the active node hasn't crashed and its network heartbeat is still up, so Witness has nothing to announce and transparent failover never fires. The node's storage path blocks instead, and the pending I/O the Hyper-V host is waiting on never returns.

Hyper-V reacts to that storage timeout by parking the running guests in Paused-Critical rather than crashing them outright, which is why the symptom reads as frozen VMs on a cluster that still looks healthy from the network side. Meanwhile the writes that were in flight to the .vhdx files can tear.

VHDX carries a metadata log region for structural consistency, and an aborted write leaves that region incomplete; Windows will attempt a VHDX metadata log replay automatically the next time the file is mounted. Over data already damaged by a degraded array, that automatic replay is a hazard rather than a fix, so the container is opened from a reconstructed image instead of letting a live host mount it.

Most of the advice that circulates for this failure aims at the wrong layer. Bringing the cluster back online or restarting the witness service treats a block-level storage fault as a quorum or networking problem, and it moves nothing.

Running chkdsk or Repair-Volume against the backend volume forces mountability by truncating metadata the utility can't verify, and that truncation is irreversible. Forcing a controller rebuild pins every surviving member at sustained full-surface read while the card writes reconstructed data to the replacement member, and importing a foreign configuration commits controller metadata changes to the array before anything has been imaged; the geometry is already recorded on the drives, so there is nothing to gain by letting a card rewrite it.

A backend RAID 5 or RAID 6 rebuild has to read every sector of every surviving member, and consumer drives carry a worst-case manufacturer specification of one unrecoverable read error per 10^14 bits read, about 12.5 TB, so the risk of hitting one rises with array capacity and drive age. What a URE does next depends on the controller: a legacy or low-end card aborts the rebuild and drops the logical drive, while modern Dell PERC and LSI/Broadcom MegaRAID puncture the affected stripe and finish the rebuild, losing that stripe rather than the whole array.

The routing conclusion is the part that decides your shipping list. When the .vhdx files live on the SOFS share, the Hyper-V hosts hold no copy of the guest payload; it is all on the file server's backend storage, so the drives that need imaging are the backend members. That is the same member-level array reconstruction as any other failed hardware or software RAID set. A file server chassis that failed at the backplane or the controller instead is handled like any other enterprise server array. Free evaluation, no diagnostic fee, and no charge if nothing comes back.

1. Ship the backend members, not the hypervisors. Power the file server down to stop further writes, label each drive with its bay or slot position so the original member order is documented, and mail the complete set to our Austin, TX lab. No recovery is attempted against the live UNC path or through the cluster nodes.
2. Image every backend member read-only. Each drive is cloned block by block through ddrescue, DeepSpar Disk Imager, PC-3000 Portable III, or PC-3000 Express; the SAS members common in SOFS backends are imaged sector by sector on SAS-capable imaging hardware. The imager reads and captures sectors and does not assemble the array.
3. Reconstruct the backend array virtually in software. On a compound hardware RAID backend, stripe size, member order, and parity rotation are recovered from the on-disk metadata and reassembled with Data Extractor Express RAID Edition, run against the image files rather than the live members. The SMB layer plays no part here; the share, the persistent handles, and the witness exist only while a live client is talking to a live node.
4. Parse the file server volume and pull the containers. NTFS or ReFS is read directly from the reconstructed image, the VM directory is walked, and each .vhdx base disk plus its .avhdx differencing files are extracted intact.
5. Mount the guest read-only from the extracted copy. The VHDX metadata log state is examined on the copy, never replayed against the original image, and the guest filesystem is verified before delivery. Checkpoint chain merges and the deeper container internals are documented on our VHDX and checkpoint recovery page.

How Do You Send a Failed Array In and What Happens Next?