What to Do When a RAID 5 Rebuild Fails

RAID 5 distributes parity across all drives to survive one drive failure. A rebuild failure occurs when the array loses one drive, begins regenerating data onto a replacement, and encounters a second error before regeneration completes. The array is now worse off than after the original failure.

• 1.One drive in the array fails or goes offline. The array enters degraded mode.
• 2.The controller serves data by computing the missing drive's contribution from parity on each read request.
• 3.An administrator inserts a replacement drive (or a hot spare activates). The controller begins rebuild: reading every sector of every surviving drive and XORing the results onto the replacement.
• 4.A second drive reports an Unrecoverable Read Error or fails outright. The controller cannot XOR the stripe where the error occurred.
• 5.The rebuild aborts. The array transitions from degraded to failed.

A stale drive contains data from before it was removed from the array. Forcing it back online causes the controller to recalculate parity using outdated blocks, silently corrupting every stripe that received writes while the drive was absent.

• 1.RAID 5 parity for each stripe is the XOR of all data blocks in that stripe.
• 2.When a drive goes offline, the controller stops including it in parity calculations and continues serving I/O using parity reconstruction.
• 3.Writes that occur while the drive is offline update the remaining drives but leave the offline drive unchanged.
• 4.If the stale drive is forced back in, the controller XORs its outdated blocks with current blocks. The resulting parity is wrong for every modified stripe.
• 5.Reads from affected stripes return silently corrupted data. The corruption is invisible until a parity scrub or until an application encounters garbage output.

The probability of hitting an Unrecoverable Read Error (URE) during a RAID 5 rebuild is a function of drive capacity, member count, and the manufacturer's published URE rate. For modern high-capacity arrays, this probability is not negligible.

• 1.Consumer drives (WD Red, Seagate IronWolf non-Pro) specify a URE rate of 1 in 10¹⁴ bits read. That equals roughly 1 unrecoverable error per 12.5 TB of sequential reads.
• 2.Enterprise drives (Seagate Exos, WD Ultrastar) specify 1 in 10¹⁵ bits, or roughly 1 error per 125 TB.
• 3.A 4-drive RAID 5 with 14 TB consumer drives rebuilds by reading 3 surviving members sequentially: 3 x 14 TB = 42 TB total reads.
• 4.At a 1-in-10¹⁴ URE rate, reading 42 TB means reading 3.36 × 10¹⁴ bits. The expected number of UREs is 3.36 (42 TB / 12.5 TB per expected error). The probability of completing that read with zero UREs is under 5%. In practice, a 4-drive RAID 5 rebuild with 14 TB consumer drives is more likely to hit a URE than not.

These numbers explain why RAID 5 is no longer recommended for drives above 2 TB in production environments. RAID 6 (dual parity) or RAID 10 (mirrored stripes) tolerate a single URE during rebuild without losing the array. For existing RAID 5 deployments, regular mdadm --action=check or controller-level patrol reads surface latent errors before they are discovered during the high-stakes rebuild window.

A degraded array has lost one drive but continues operating with parity intact; all data remains computable. A failed rebuild is a different state: partial parity has been written to the replacement drive, and the pre-rebuild data on surviving drives may be partially overwritten.

RAID 5 distributes parity across all member drives so that no single drive is a parity bottleneck. The pattern by which parity rotates from drive to drive across successive stripes is called the parity rotation algorithm. Using the wrong rotation during virtual reconstruction produces corrupted output.

Left-Asymmetric (Forward Parity)

Parity starts on the last drive for stripe 0 and shifts one position to the left for each subsequent stripe. Data blocks fill the remaining positions left-to-right, skipping the parity position. Used by Linux mdadm (--layout=left-asymmetric) and some older Adaptec controllers.

Left-Symmetric (Default for mdadm)

Parity rotates in the same left-shifting pattern, but data blocks continue from where the previous stripe left off rather than restarting at position 0. This is the default for Linux mdadm and produces better sequential read performance because consecutive data blocks are distributed more evenly across drives.

Right-Asymmetric (Backward Parity)

Parity starts on the first drive and shifts one position to the right for each stripe. Common on Dell PERC (MegaRAID-based), LSI MegaRAID, and HP SmartArray controllers. This is the most frequently encountered rotation in enterprise hardware RAID.

Right-Symmetric (Backward Symmetric)

The right-shifting complement of left-symmetric. Data blocks continue from the previous stripe's ending position. Less common in practice; found on some Areca and 3ware controllers.

When a RAID controller is dead, its metadata wiped, or the array is in a double-fault state after a failed rebuild, reconstruction moves to software. We use the PC-3000 DE RAID module (ACE Lab) to virtually reassemble the array from individual drive images without depending on the original controller.

1. Each member drive is connected individually to a PC-3000 Portable III or PC-3000 Express through a write-blocked hardware channel. Sector-level images are created, including any drives with bad sectors (sectors with read errors are logged and filled with a pattern marker).
2. The technician inputs the detected array geometry: number of member drives, stripe size (identified via hex analysis), parity rotation algorithm, and drive order (determined from bay labels and parity block positions).
3. PC-3000 DE RAID assembles a virtual disk image by reading sectors from each drive image in the correct stripe order. Parity blocks are used to reconstruct data from any single missing or damaged drive.
4. The output is a single contiguous disk image that represents the original logical volume. This image is mounted read-only, and the recovered file system is verified before copying data to the client's destination media.

The entire process operates on images, not live drives. If the first parameter guess is wrong, the geometry is adjusted and reconstruction runs again without risk. For arrays where the original drive order is unknown, PC-3000 provides an auto-detection mode that tests candidate orderings against known file system signatures.