Degraded RAID Array: Safe Troubleshooting Before You Rebuild

1. Auto-rebuild starts immediately, giving the administrator no opportunity to check SMART data on surviving drives.
2. For parity arrays (RAID 5/6), the rebuild reads every sector of every surviving drive under sustained sequential I/O. RAID 1/10 rebuilds read only the mirror partner. On aging consumer 8TB drives in a parity array, a 24TB rebuild places sustained mechanical stress on the remaining members, increasing the risk of a secondary failure or latent sector error.
3. Drives from the same manufacturing batch tend to fail in close succession. If one drive from a batch of 8 has failed, the remaining 7 are statistically more likely to fail under the increased load of a rebuild.
4. Rebuild times on large arrays can exceed 24 hours, during which the array runs with zero fault tolerance (RAID 5) and the drives experience sustained sequential I/O.

The resync window is a bounded interval, not a background condition. It opens the moment synchronization starts, whether that is a reconstruction onto a replacement member or a parity re-sync after an unclean shutdown, and it closes when the operation reports 100 percent. What the array can survive is decided inside it.

RAID 1 with one surviving member, RAID 5, and the pair being rebuilt inside a RAID 10 all carry zero redundancy for the length of that window. Single-degraded RAID 6 is the exception that carries one drive of margin through it, which is the practical reason it absorbs a read error that stops a RAID 5 reconstruction.

Two things set how long the window stays open. The first is the total capacity that has to be read, which on a parity array is every allocated sector of every surviving member. The second is the throttle. Linux md enforces a floor and a ceiling on synchronization throughput through its speed limit parameters, dev.raid.speed_limit_min and dev.raid.speed_limit_max. Hardware controllers expose the same idea as a rebuild rate, a percentage of controller resources allocated to the operation. A low rebuild rate keeps the volume responsive for the host and holds the zero-redundancy window open longer.

None of the four md sync actions is guaranteed read-only

Four Linux md sync actions matter on a degraded array, and the background-operations section above uses two of them. The distinction is worth holding precisely, because all four can write to the member drives:

• resync rebuilds redundancy across the members that are already present, which is what runs after an unclean shutdown or at array creation. It writes parity on RAID 4, 5, and 6, and mirror copies on RAID 1 and RAID 10.
• recovery reconstructs a missing member onto a replacement drive or a hot spare. It writes the reconstructed content to that target member.
• check reads every member, compares the data against the parity, and records mismatches in mismatch_cnt without correcting them. A read error is the exception. The normal md read-error path reconstructs that block from the other members and writes it back to the failing drive.
• repair runs the same read pass and corrects every mismatch it finds, writing new parity on RAID 4, 5, and 6 and overwriting all but one copy on RAID 1 and RAID 10.

check is the only one of the four that leaves a clean member byte-identical, and it stops being read-only at the exact moment it meets the bad sector you were worried about. On a degraded array holding data you cannot lose, all four are write operations against the exact members a recovery would have to read.

Latent sector errors surface on the pass that has no parity left

A latent sector error is a physical defect sitting unread on a member. The controller does not know it is there, because nothing has touched that sector. Normal operation reads only live data, so a defect in a region no host has requested since it was last written stays dormant and stays invisible. The full end-to-end member read that a reconstruction performs is, for a large share of those sectors, the first access since the data was written. That is the trap: the pass that finally touches them is the one pass where a degraded parity array has no redundancy left to reconstruct what they held.

Keep the defect and the event separate. A latent sector error is the dormant physical defect; a URE is the read attempt that fails on it after the drive exhausts its internal retries and ECC. The section below on why a degraded RAID 5 reconstruction hits an unrecoverable read error works through the per-bit rating and the controller-dependent outcomes, and that arithmetic is a probability rising with array size and drive age, not a scheduled event.

Scrubbing on software RAID and patrol read on hardware controllers exist for this specific reason: read every sector while the array still has redundancy, so a defect is found, reallocated, and rewritten from parity long before a reconstruction needs that sector. The background-operations section above covers what those same schedules do to a member that is already marginal, and how to stop them before imaging.

A parity stripe update cannot be made atomic across two disks

Updating a block in a parity array is two writes: the data block to one member and the recalculated parity block to another. They land on independent devices, and there is no mechanism in the array itself that commits both or neither. If power is lost or the controller resets between them, the stripe is left with data and parity that no longer agree, and nothing on disk records which of the two is authoritative. That is the RAID write hole.

On a healthy array the disagreement stays latent. Reads of that stripe are served from the data blocks directly, so a stale parity block sits there doing nothing until something needs it. A degraded array is the case where something needs it. The absent member's blocks are not read at all; they are computed from the surviving data and that parity block. Feed a stale parity block into the calculation and the result is wrong data handed to the host with no error reported at any layer, so the filesystem and anything reading through it accept the block as valid. That is silent corruption, and it is the difference between a degraded array being slower and a degraded array being dangerous.

Which mitigations close the write hole?

Write-intent bitmap

A map of which regions of the array have writes in flight. After an unclean shutdown the resync covers only the regions the bitmap marks dirty instead of the whole array. On Linux md it is stored inside the md metadata with --bitmap=internal or kept as an external bitmap file. It shortens the resync window and it does not close the write hole.

Partial parity log (PPL)

The Linux md consistency policy for RAID 5, selected with --consistency-policy=ppl. It logs enough about a stripe update to recover a stripe whose write was interrupted, which closes the write hole on that array. Its documented costs are a write-performance penalty and incompatibility with a write-intent bitmap; you pick one or the other.

Dedicated write journal

mdadm --write-journal on a separate device for RAID 4, 5, and 6. Data and parity updates are logged to the journal device before they land on the array, so an interrupted stripe update can be resolved from the log rather than guessed at.

Controller-side non-volatile write cache

A hardware controller running write-back holds pending stripe writes in cache with a non-volatile backup behind it, so an interrupted stripe update can be completed once power returns. When that backup module is absent or has failed, the controller drops to write-through and the array is exposed on power loss. Check the module's health before trusting write-back on an array that is already degraded.

ZFS RAID-Z

RAID-Z avoids the write hole by construction. Every write is a variable-width full stripe written copy-on-write, so live data is never overwritten in place and there is no read-modify-write cycle to interrupt. It does not protect against drive failures that exceed the vdev's redundancy level.

Capture the controller configuration and image the ejected member

Two captures belong before any reconstruction, and neither is covered by the imaging section below.

1. Record the controller's own configuration read-only, before anything is inserted, re-seated, or imported. The vendor management tools already listed in the background-operations section do this: OMSA or racadm and iDRAC on Dell, Smart Storage Administrator on HPE, MegaCLI or StorCLI on LSI/Broadcom, arcconf or maxView on Adaptec. Export the full configuration and the controller event log to a file on a different machine and keep it. That capture records the controller's view of state; the array geometry itself is written to the member drives, in the trailing-sector DDF structures on PERC and MegaRAID and in the RAID Information Sector at the head of each drive on HP Smart Array P-series and E-series controllers, so a dead card does not take the layout with it. On software RAID the read-only equivalent is the superblock dump in the first-response section above.
2. Image the ejected member as well, not only the survivors. Controllers eject members for localized bad sectors or for a timeout while most of the surface is still readable, and a partially readable ejected member can supply the stripes where a survivor returns a bad sector. Recovery practice is to image it through the same write-blocked path as the rest, on a DeepSpar Disk Imager or a PC-3000 Portable III when it reads slowly, and to hold that image as a source of last resort during virtual assembly.

Reconstruction then runs in software against those image files, never against the members. The same split applies to Linux software arrays, where mdadm superblock and LVM recovery reads the geometry back out of the member metadata, and the stripe arithmetic a virtual assembly has to reproduce is set out in how parity is calculated across a stripe. Our array recovery service performs both stages in-house at the Austin, TX lab, with no diagnostic fee and no recovery fee if the data does not come back.

The URE specification is a per-bit-read rating and a worst-case manufacturer floor, not a quoted success rate and not a countdown timer. A parity rebuild forces a full-capacity read across every surviving member, which is what drives the risk up.

1. A consumer SATA drive rated at one URE per 10^14 bits will, on average, return one unrecoverable sector for every 12.5 TB it reads.
2. Reconstructing a single failed member in RAID 5 means reading every sector of every surviving drive. On a large array that read total runs on the order of 48 TB, well past the 12.5 TB figure the worst-case rating is built on.
3. Because the array is already degraded it holds zero spare parity. When the rebuild read hits a URE on a surviving member, the missing block in that stripe cannot be recomputed.
4. What happens next is controller-dependent. HP Smart Array P-series and E-series and low-end consumer controllers abort the resync and can drop the virtual disk offline; Dell PERC and LSI/Broadcom MegaRAID puncture that stripe and finish the rebuild; and Linux mdadm records the sector in its Bad Block Log and continues.
5. The same sustained sequential read that triggers the URE also loads aging members from the same manufacturing batch, raising the chance a second drive drops during the rebuild window.

Shingled Magnetic Recording members compound the problem. Under the sustained sequential write of a rebuild the CMR cache zone fills and the drive pauses to reshingle; the controller reads that latency as a dead drive and ejects the healthy SMR member, turning a single-drive rebuild into a multi-drive failure.

URE (Unrecoverable Read Error)

A sector a drive cannot read after exhausting its internal retries and ECC. Rated at roughly one per 10^14 bits, or about 12.5 TB, on consumer SATA drives.

SMR (Shingled Magnetic Recording)

A recording method that overlaps tracks and relies on a CMR cache zone. The background reshingle pause during sustained writes can be misread by a controller as a drive failure and trigger ejection during a rebuild.

DDF (Disk Data Format)

The SNIA on-disk metadata structure that lets enterprise controllers store array geometry on the member drives, so the array can be reconstructed from the disks without the original controller.