The rebuild that finishes
what the failure started.

Nearly every RAID case that arrives at this lab failed twice. The first failure is the one people describe on the phone: a disk died, the controller dropped a member, the volume went offline. The second failure is the one that actually cost the data, and it was carried out deliberately, by someone trying to help.

Arrays do not fail all at once

Redundancy is not durability. A RAID 5 set tolerates one member failing, and a RAID 6 set tolerates two. What redundancy buys you is time to replace the failed member, and nothing else.

The trouble is that losing the first member is silent. The volume stays up, performance barely moves, and unless someone is actually reading the alerts, nothing visibly changes. Arrays routinely run degraded for weeks or months. By the time anyone notices, the noticing event is the second disk going, and at that point the array is offline and the situation is already serious.

Why the second disk goes so soon after the first

It is rarely coincidence. The members were bought together, from the same batch, and have run the same hours under the same thermal and vibration conditions. They age together. On top of that, a rebuild is the heaviest sustained read load an array ever sees: every remaining member is read end to end to recompute the missing one. A marginal disk that coped fine with normal use can fail precisely during that read. Which means the rebuild itself is when the next failure is most likely.

The stale member problem

Here is where cases are lost. Suppose disk 3 dropped out two months ago and nobody noticed. The array kept running on the remaining members, and every write since then has updated them. Disk 3 still spins, and still holds data, but that data is two months out of date. In array terms, it is stale.

Then disk 5 fails, and the volume goes offline. The array is now short two members and will not assemble. Someone looks in the controller UI, sees that disk 3 is present and appears healthy, and forces it back into the array to bring it online.

The controller accepts it and begins rebuilding. It now believes disk 3 holds current data, so it uses disk 3's two-month-old contents, together with the surviving members, to recompute what should be on the replacement for disk 5. Then it writes that computed result across the array.

The output is wrong, because one of the inputs was wrong. And it is not wrong in an obvious, contained way. Stripes are interleaved across all members, so the bad reconstruction is sprayed through the whole volume. File system metadata, database files and virtual machine disks all get corrupted in place, and the correct data that was on the surviving members is overwritten as the rebuild progresses.

Before the rebuild, that array was recoverable. Almost always fully. After a completed rebuild from a stale member, a significant amount of it is simply gone, because the correct version was overwritten by a computed wrong one.

Other ways the recovery attempt causes the loss

• Initialising instead of importing. On an unfamiliar controller UI, the option that creates a fresh array sits next to the one that imports an existing configuration. Choosing wrong writes new metadata over the old.
• Members reinserted in the wrong order. Disk order is part of the array's geometry. If the disks come out of the tray unlabelled and go back in a different order, the assembly is wrong. Order can be recovered from the stripe pattern in the data, but that is analysis time that a photograph of the tray would have saved.
• Running fsck or chkdsk on a badly assembled volume. If the geometry is wrong, the file system looks catastrophically damaged, and the consistency tool will start deleting the inconsistencies to make it mount. The volume was fine. The assembly was not.
• Letting a NAS "repair" its own volume. Consumer NAS boxes offer a repair action that does the same forcing and rebuilding as a controller, behind a friendlier word.
• Restoring a backup on top of the array. Done before imaging, this overwrites the very thing a lab would have recovered, and it is unrecoverable if the backup then turns out to be incomplete.

What to do when an array goes down

1. Power the array down. A degraded array left running keeps writing, which widens the gap with any stale member and reduces your options.
2. Photograph the tray before touching anything. Bays and serial numbers visible. This one photo saves hours of analysis.
3. Do not accept any rebuild, force-online, import or initialise prompt until someone has established which members are current and which are stale.
4. Write down the history. Controller model, RAID level, member count, when each disk failed, what has been attempted and in what order. Different people often did different things, and the sequence matters.
5. Check whether the backup actually restores before planning around it. Silently failing backup jobs are the single most common unpleasant surprise in these cases.

How the lab assembles it instead

Every member is imaged independently, read-only, including the ones the controller had already given up on. Nothing is assembled on the original disks at any point.

The geometry is then derived from the images themselves rather than taken on trust from the controller: disk order, stripe size, parity rotation, and start offset, all determined by looking at where recognisable file system structures land. Staleness is worked out the same way, by comparing what each member holds against the others, which is what tells us which disk must be excluded from the reconstruction.

Only then is the array assembled virtually, from images, and mounted. If an assumption was wrong, the volume does not mount cleanly, the parameters are adjusted and it is tried again. The cost of being wrong is another attempt, not another loss. That is the whole difference between this and letting the controller do it.