How to replace a failing hard drive in a Linux RAID array

When a hard drive starts failing in a Linux RAID configuration , it's easy to panic: the server becomes unresponsive, the system won't boot, or the storage pool goes into an "UNAVAIL" state. The good news is that, if the RAID is properly configured and you handle it carefully, you can replace the faulty drive and recover without data loss.

In this article we will see, in a very detailed and practical way, how to identify a broken disk, how to remove it from a software RAID with mdadm , how to handle more delicate cases such as a ZFS pool that does not want to import due to a lack of replicas, what happens when you try to read disks in Windows with WSL2 and XFS, and how to handle scenarios where the RAID disks have different sizes and you then want to take advantage of the remaining space.

Basic concepts: Software RAID in Linux (mdadm, ZFS, RAID types)

Before we start tinkering with anything, it's important to understand some fundamental concepts about RAID in Linux . This will prevent unpleasant surprises and irreversible errors when it comes time to replace a damaged disk.

Most traditional Linux servers use software RAID based on mdadm . This system creates virtual /dev/mdX devices (for example, /dev/md0, /dev/md1…) that are built from physical partitions such as /dev/sda1, /dev/sdb2, etc. File systems like ext4, XFS, or others are mounted on these /dev/mdX devices.

A different, and increasingly common, case is the use of ZFS as a file system and volume manager . ZFS does not rely on mdadm, but rather manages its own redundancy using vdevs (mirrors, RAIDZ, etc.). When a disk fails in ZFS, the replacement and recovery process is handled with commands like zpool , not mdadm.

It's also important not to confuse RAID with backups: RAID offers redundancy and service continuity , but it doesn't replace a backup. A RAID 1 array can save you if a disk fails, but not if you accidentally delete data or if the entire logical structure of the file system becomes corrupted.

In many providers (such as Hetzner and other dedicated ones), system images already come with a pre-configured software RAID1 on two disks, divided into several partitions: swap, /boot, the root / and sometimes /home, each with its corresponding /dev/mdX.

How to inspect the status of a software RAID with mdadm

When you suspect one of your disks is starting to fail, the first step is to see exactly what's happening in the RAID array . For this, you have several basic tools that you should be familiar with.

A very useful command is `lsblk` , which shows how the disks, partitions, and RAID devices are organized on the system. With a single command, you can see which physical disks are present and how they relate to `/dev/mdX`.

Example output of lsblk (simplified):

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop1 7:1 0 110.6M 1 loop /snap/core/...
sda 8:0 0 3.6T 0 disk
├─sda1 8:1 0 16G 0 part └─md0 9:0 0 16G 0 raid1
├─sda2 8:2 0 512M 0 part └─md1 9:1 0 511M 0 raid1 /boot
├─sda3 8:3 0 2T 0 part └─md2 9:2 0 2T 0 raid1 /
└─sda4 8:4 0 1.7T 0 part └─md3 9:3 0 1.7T 0 raid1 /home
sdb 8:16 0 3.6T 0 disk
├─sdb1 8:17 0 16G 0 part └─md0 9:0 0 16G 0 raid1
├─sdb2 8:18 0 512M 0 part └─md1 9:1 0 511M 0 raid1 /boot
├─sdb3 8:19 0 2T 0 part └─md2 9:2 0 2T 0 raid1 /
└─sdb4 8:20 0 1.7T 0 part └─md3 9:3 0 1.7T 0 raid1 /home

With this structure, you can see that there are two disks, sda and sdb, and that each partition of sda has its mirror in sdb, forming the arrays md0, md1, md2, and md3. In such an environment, if one of the disks fails, the other should continue to contain all the information.

To delve deeper into a specific array, use `mdadm --detail` . For example, to see the state of `/dev/md0`:

mdadm --detail /dev/md0

The output includes data such as RAID level, size, number of active disks, and finally, a table with the devices that make up the array. You'll see something like this:

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1

This table is where you can detect if a partition is marked as faulty , "removed," or if a disk is missing. This information is key to knowing which disk needs to be replaced or which one should be removed without breaking the system.

Removing a healthy disk from a RAID (for example, in dedicated servers)

In some cases, there isn't a physical disk failure; you simply want to free up one of the disks in the RAID array for another use: independent storage, backups, etc. This is common in dedicated servers where the provider installs a RAID 1 configuration by default, but you need the second disk free.

The idea is this: mark the partitions on that disk as defective in each array, remove them from the RAID, and then resize the number of devices so that mdX stops waiting for that second disk. The system will continue to function with a single-disk "RAID," which is really a pseudo-RAID but saves you from having to reinstall.

The process must be repeated for each mdX array involved (swap, /boot, /, /home…). Let's look at it step by step for md0, knowing that you have to do the same for md1, md2, etc.

1. Review the partition involved in the array

First, check which physical partitions make up /dev/md0:

mdadm -D /dev/md0

At the end of the exit you'll see something like:

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1

If your goal is to remove the sdb disk from the RAID, that table shows you that you will have to act on /dev/sdb1 in the case of md0, /dev/sdb2 in md1, and so on.

2. Mark the partition as defective

mdadm doesn't let you simply remove a partition it considers healthy; you first have to mark it as failed. This doesn't physically damage the disk, it just tells the array to stop using that replica.

Example for md0 and sdb1:

mdadm /dev/md0 --fail /dev/sdb1

The typical response will be something like "set /dev/sdb1 faulty in /dev/md0". From that moment on, the array knows that this replica is no longer trusted and puts it into a faulty state.

3. Remove the failed partition from the array

Once marked as faulty, you can remove the partition from the RAID :

mdadm /dev/md0 --remove /dev/sdb1

This will cause mdadm to "unlink" the physical partition of the array. Logically, md0 will still exist, but it will only be backed up by /dev/sda1.

4. Adjust the number of RAID devices

At this point, md0 may remain in a degraded state if it still believes there should be two devices. For the system to consider it stable, you need to change the expected number of disks in the array:

mdadm --grow /dev/md0 --raid-devices=1 --force

Although the word "grow" may seem to indicate expansion, in this context you are actually changing the internal configuration to tell the array that there will now only be one active device.

After repeating this process on all the mdX drives, you can restart the server. The result will be a system that boots using a software RAID with a single disk, and the other disk will be completely free to partition and format as you wish.

Actual replacement of a failing disk in mdadm RAID

When the problem is not that you want to free up a disk, but that one of them has actually died , the procedure is similar but with some key differences: instead of keeping only one disk, the goal is to replace the damaged disk with a new one and rebuild the RAID.

In general terms, the cycle would be:

• Identify which disk is failing (smartctl, kernel messages, mdadm –detail).
• Mark the partitions on that disk as faulty in each mdX.
• Remove them from the RAID with mdadm –remove.
• Physically replace the defective disk with a new one (avoid scams when buying records).
• Partition the new disk so that its partitions match in size and type with those of the healthy disk.
• Add the new partitions to each /dev/mdX with mdadm –add and let it rebuild.

With mdadm –detail /dev/md0 You can check the status of the rebuild and see when all disks are active and synchronized again. You can also monitor the process with watch -n1 cat /proc/mdstatwhich will refresh the resync progress bar every second.

In more advanced configurations (such as RAID 5 or RAID 6), the same principle applies, but the impact of losing more than one disk at a time can be catastrophic. That's why it's critical to quickly replace a disk that starts to fail before another one does.

Create and manage a new RAID array (example RAID6 with mdadm)

Although the goal here is to replace failing disks, it helps to see how a RAID array is built from scratch to understand what is being modified when a disk is replaced. A typical example is creating a RAID 6 array with multiple disks using mdadm.

The first step is to ensure that mdadm is installed on the system. On Debian/Ubuntu, this is done with:

sudo apt-get install mdadm

In Red Hat-type systems, the equivalent command would be:

sudo yum install mdadm

Before creating the array, it's advisable to check that the disks don't already have superblocks from old RAIDs. This is done by examining the potential disks or partitions:

sudo mdadm -E /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

If no previous signature appears, the next step is to partition each disk using fdisk (or parted) by creating a primary RAID partition (code fd). The typical sequence in fdisk would be: create new partition (n), primary (p), choose number, accept default sectors, change type (t), select code fd, and write changes (w).

Once all partitions are prepared, the array is created:

sudo mdadm --create /dev/md0 --level=6 --raid-devices=5 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

This will create /dev/md0 as a RAID6 array using five partitions. The array will begin rebuilding automatically, and you can monitor the progress with:

watch -n1 cat /proc/mdstat

Once the RAID is ready, you need to create the file system , for example ext4:

sudo mkfs.ext4 /dev/md0

Then it is mounted on a directory, for example /mnt/Raid6:

sudo mkdir -p /mnt/Raid6
sudo mount /dev/md0 /mnt/Raid6/

To make it mount automatically at startup, an entry is added to /etc/fstab, something like this:

/dev/md0 /mnt/raid6 ext4 defaults 0 0

And to avoid losing the array configuration, the mdadm configuration file is generated:

sudo mdadm --detail --scan --verbose >> /etc/mdadm.conf

Testing RAID by causing a disk failure

Before entrusting your data to a newly created RAID array, it's advisable to verify that it behaves as expected in the event of a failure . The safest way to do this is to simulate the failure of one of the disks using mdadm, without touching the physical hardware.

Suppose you have a RAID array that includes /dev/sdb1, /dev/sdc1, and a spare disk /dev/sdg1. You can force the logical failure of /dev/sdc1 with:

sudo mdadm --manage --fail /dev/md0 /dev/sdc1

Then check the array details:

sudo mdadm --detail /dev/md0

During the boot process, you should see that the disk marked as faulty has been cleared and replaced by the spare, which will begin synchronizing. This is the best proof that the RAID responds well to a real-world incident.

Special cases: problems with ZFS when a disk in the pool fails

In more modern environments, especially with hypervisors like Proxmox, ZFS is used to manage disks in pools with native redundancy . A typical example: you have a mirrored ZFS pool (RAID 1), one of the two disks fails, and suddenly the server stops booting because the root pool cannot be imported.

In that situation, when trying to manually import the pool, you may encounter messages like:

zpool import -f rpool
cannot import 'rpool': no such pool or dataset
Destroy and recreate the pool from a backup source

Or try other options:

zpool import -N

It returns a UNAVAIL status and indicates "insufficient replicas" and "damaged devices or data". In short, ZFS doesn't see enough valid copies to ensure the integrity of the pool, and therefore refuses to import it in the standard way.

In these cases, it's critical not to improvise too many random commands. Some ideas to keep in mind:

• If the mirror was correctly mounted, a single healthy disk should be sufficient to import the pool in degraded mode. If ZFS reports insufficient mirrors, there may be severe corruption or poor device detection.
• You can't "tell" ZFS to directly replace a dead disk with another if the pool fails to import; it's not like mdadm where you work at the partition level without mounting the file system.
• The prudent thing to do is usually to try an import forced and read-only Using advanced flags (e.g., -F, -o readonly=on), check which datasets are recoverable and copy data out if possible.
• If the pool truly cannot be imported in any way, the next sensible step is to clone or block image the still-working disk, to preserve that "backup" before further experimentation.

The major problem in a scenario like the one described (where the working disk is also the only place where the data is stored) is that there's no additional safety net . That's why it's best to halt any aggressive repair attempts and focus first on saving the contents of the still-responsive disk, even if it's outside of ZFS.

Read a Linux RAID (XFS, ext4) from Windows using WSL2

Another very common scenario is that of someone removing a disk from a Linux server and wanting to read its data from a Windows PC . At first glance, it might seem simple: you connect the disk via USB, use a tool like Linux Reader, and that's it. But when that disk is part of a software RAID, things get complicated.

In the specific case of a disk with a GPT table where all partitions appear as "Linux RAID", Windows and many graphical utilities don't know what to do with them. For example, fdisk might display something like this:

Device Start End Sectors Size Type
/dev/sdd1 1032192 5031935 3999744 1.9G Linux RAID
/dev/sdd2 5031936 9031679 3999744 1.9G Linux RAID
/dev/sdd3 30720 1032191 1001472 489M Linux RAID
/dev/sdd4 9031680 3907028991 3897997312 1.8T Linux RAID

If inside /dev/sdd4 there is, for example, an XFS filesystem belonging to an mdadm array, trying to mount it directly usually results in something like:

mount -t xfs /dev/sdd4 /mnt/tempdisk
mount: wrong fs type, bad superblock on /dev/sdd4...

Or if you try it without specifying the FS type:

mount /dev/sdd4 /mnt/tempdisk
mount: unknown filesystem type 'linux_raid_member'

These messages indicate that /dev/sdd4 is not a filesystem itself , but rather a member of a RAID array. The "real" filesystem resides on /dev/mdX, which doesn't yet exist in this context. To read the contents from WSL2 (Ubuntu, for example), you will need to:

• Load the mdadm modules into the WSL2 kernel if they are not already loaded.
• Use mdadm –examine on RAID partitions to identify the array to which they belong.
• Recreate the mdX array in "assemble" mode (mdadm –assemble) so that /dev/md0 (or the one it's supposed to be) appears.
• Once /dev/md0 exists, mount that device with the appropriate FS type (XFS, ext4, etc.).

Without going through the RAID assembly phase, you won't be able to directly mount partitions marked as Linux RAID , even if you know they contain an XFS file system. And in native Windows, except for very specific tools (usually paid or read-only), it's difficult to handle these cases effectively.

RAID with different sized disks and use of leftover space

A curious, but increasingly common, case is that of someone setting up a RAID 1 array with two drives of very different sizes , for example, a 1 TB NVMe and a 500 GB SATA. In a pinch, this might be acceptable, but the RAID 1 will only utilize the smaller drive's capacity for the mirror.

In that scenario, the typical result will be that the "second half" of the 1 TB disk remains unused, simply because the RAID 1 is limited to the first 500 GB (the size of the smaller disk). That portion will not be part of the array and will remain as unallocated space or on a separate partition.

Let's say you now decide to do things properly and want to replace the small disk with another 1 TB disk identical to the first. The disk replacement procedure in mdadm is similar to what has already been described: the old disk is marked as faulty, removed from the array, the new disk is inserted, the corresponding partitions are created, and it is added to the RAID to be rebuilt.

The next question is: how to take advantage of the additional space you now have on both disks to expand, for example, the root partition (/) by about 500 GB? Broadly speaking, the steps usually followed are:

• Resize the RAID partitions on both disks to occupy the extra space (using tools like parted or fdisk, and always with extreme care).
• Use mdadm –grow about the mdX array to tell it that its effective size is now larger.
• Once the RAID device has more capacity, resize the file system on top (for example, extend ext4 with resize2fs; see How to free up disk space in Linux).

These types of operations, especially those involving the root partition and UEFI/EFI, swap, and other components, require modifying the partition table, RAID array, and file systems. It is highly recommended to have recent backups and a thorough understanding of how the EFI partitions and swap are configured to avoid rendering the system unbootable.

If everything is done correctly, neither the swap nor the EFI partition should become misconfigured, but any errors in sector numbers, partition order, or devices registered in /etc/fstab and the boot loader can leave the system in "rescue" mode asking for manual repairs.

In summary, using different sized disks in RAID1 is viable in the short term but not recommended in the medium to long term, because it greatly complicates future space management when you try to homogenize the hardware.

After all this, the key takeaway is that replacing a failing disk in a Linux RAID isn't magic, but it does require a methodical approach: first, understand the RAID topology (mdadm or ZFS); then, definitively identify which disk or partition needs to be addressed; and finally, execute the appropriate commands (mdadm –fail, –remove, –add, zpool import, etc.) with a cool head. With a well-organized strategy, good backups, and a patient approach, you can successfully navigate most situations, from freeing up a disk on a dedicated server to recovering a rogue ZFS pool or attempting to retrieve data from Windows via WSL2.