Software RAID under Linux

Introduction

This tutorial explains how to install, setup, and manage a mdadm software RAID on Linux systems. The example commands are for a server with two block storage devices. However, you can adjust the commands as you need. It is briefly explained how to format the storage devices and how to create one partition on each storage device. Then, it is shown how to create a new RAID with these two partitions.

Prerequisites

• 1 server
  • A Linux OS installed
  • Root access or a user with sudo permissions
  • At least two free partitions on two different drives

Example terminology

• RAID:

  • md0
  • Device file: /dev/md0
  • Mount point: /mnt/<your-mount-point>

• Drives and partitions

  • sda sda1 sda2 sda3
  • sdb sdb1
  • sdc sdc1

In the example commands below, RAID md0 is created with the partitions sdb1 and sdc1.

Step 1 - Preparations

First you should think about which RAID system you want to run. This depends on the target and how many drives are available on the server itself.

Note: A RAID should not be seen as a data backup as it does not provide protection against data loss. It only increases the availability of the data.

Step 1.1 - RAID level selection

Choosing the right RAID level is not easy and depends on several factors:

• How many drives does the server have?
• What are your goals?
  • More storage space / Less availability
  • Higher availability / Less storage space

Here is a list of the most used RAID levels:

Pros and cons:

A list of additional RAID levels that are used less frequently:

• Linear: Merge multiple partitions
• Multipath: No RAID, but a mapping of a file to two different paths on the same partition (mirroring).
• Faulty: Emulates a faulty RAID system for test cases
• Level 4: Like Level 0, but with an additional device for parity bits (increased reliability).
• Level 6: Like Level 5 but with two independent parity bits per segment (increased reliability).

Step 1.2 - List of drives in the system

For a short and clear list of all available block devices, you can use the command lsblk. Here is an example output:

$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0 19.1G  0 disk
├─sda1    8:1    0 18.8G  0 part /
├─sda2    8:14   0    1M  0 part
└─sda3    8:15   0  256M  0 part /boot/efi
sdb       8:16   0   10G  0 disk
sdc       8:32   0   10G  0 disk

The output shows drive 1 (sda) with three partitions (sda1, sda2, sda3), drive 2 (sdb) without any partitions and drive 3 (sdc) also without any partitions.

Note: For a software RAID, it is not necessary to add the entire drive to the RAID. Single partitions are sufficient.

For a list with more detailed information, you can use fdisk -l.

In the next steps, it is explained how to format sdb and sdc, how to create a partition on each drive (sdb1 and sdc1) and how to mark both partitions as RAID partitions. Next, you will learn how to create a new RAID with the new partitions sdb1 and sdc1.

Step 2 - Create a software RAID

Step 2.1 - Preparing the drives

First you need to format the drives accordingly.

In the example output of lsblk above, the drives sdb and sdc do not have any partitions yet. On both drives, you need to:

• Create a partition table
• Create a partition
• Mark the new partitions as RAID partitions

Note: When you follow the steps below, you will lose all data on your drives. Only run these commands on new, empty drives.

• Create a partition table
  Create a new, empty partition table on both drives:
  • For large 2 TB drives or PCs with UEFI:

    sudo parted /dev/sdb mklabel gpt
    sudo parted /dev/sdc mklabel gpt

  • For drives smaller than 2 TB and BIOS:

    sudo parted /dev/sdb mklabel msdos
    sudo parted /dev/sdc mklabel msdos

• Create a partition
  Create a partition on both drives:

  sudo parted -a optimal -- /dev/sdb mkpart primary 2048s -8192s
  sudo parted -a optimal -- /dev/sdc mkpart primary 2048s -8192s

  If you want to use the whole drive, just enter 0% 100% instead of 2048s -8192s.

  Note: We deliberately leave 8192 sectors at the end of the drive unused to be prepared for failures. It allows you to use drives that have a few sectors less as replacements due to the space left free.

  You can use lsblk to check if the partitions were created successfully:

  NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
  [...]
  sdb       8:16   0   10G  0 disk
  └─sdb1    8:17   0   10G  0 part
  sdc       8:32   0   10G  0 disk
  └─sdc1    8:33   0   10G  0 part

• Mark new partitions as RAID partitions
  Mark the newly created partitions as RAID partitions:

  sudo parted /dev/sdb set 1 raid on
  sudo parted /dev/sdc set 1 raid on

Step 2.2 - Create the software RAID

Under Linux, mdadm is the main tool. It is the interface to the RAID functions of the kernel.

You can create RAID with a single command. In this command you need to specify the RAID level and the partitions:

• RAID 1
  

  RAID: md0
  Devices: sdb1, sdc1

  sudo mdadm --create /dev/md0 --auto md --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

• RAID 5
  

  RAID: md0
  Devices: sdb1, sdc1, sdd1, sde1

  sudo mdadm --create /dev/md0 --auto md --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

The parameters in detail:

You can use the newly created block device mdX immediately and you can also shut down the system or restart it during this time. In this example, the block device is called md0.

$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
[...]
sdb       8:16   0   10G  0 disk
└─sdb1    8:17   0   10G  0 part
  └─md0   9:0    0   10G  0 raid1
sdc       8:32   0   10G  0 disk
└─sdc1    8:33   0   10G  0 part
  └─md0   9:0    0   10G  0 raid1

You can query the current status of the RAID creation with the following command:

watch cat /proc/mdstat

Example output:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc1[1] sdb1[0]
      10471424 blocks super 1.2 [2/2] [UU]
      [===============>.....]  resync = 75.5% (7907008/10471424) finish=0.3min speed=112850K/sec

Now that RAID is created, you can format and mount it.

• Format the newly created RAID:

  sudo mkfs.ext4 /dev/md0

• Mount the RAID:

  sudo mkdir /mnt/<your-mount-point>
  sudo mount /dev/md0 /mnt/<your-mount-point>

  You can replace <your-mount-point> with any directory name. This directory will be the mount point of your RAID. This means that all the data you save in this directory will be saved in your RAID md0.

• Automatically create the RAID:
  To have RAID automatically created after a system reboot, you need to add a new entry in the mdadm.conf file.

  sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf
  sudo update-initramfs -u

• Automatically mount the RAID:
  To have RAID automatically mounted after a system reboot, you need to add a new entry in the /etc/fstab file.

  Names such as mdX can change, so it is recommended to add the UUID in the fstab file instead. The UUID never changes.

  • Find out the UUID
    You can use lsblk to view the name of the RAID block device:

    sdb       8:16   0   10G  0 disk
    └─sdb1    8:17   0   10G  0 part
      └─md0   9:0    0   10G  0 raid1 /mnt/<your-mount-point>
    sdc       8:32   0   10G  0 disk
    └─sdc1    8:33   0   10G  0 part
      └─md0   9:0    0   10G  0 raid1 /mnt/<your-mount-point>

    In this example, the RAID name is md0. In the command below, replace md0 with the name of your own block device.

    sudo blkid | grep md0

    The output should include the UUID. Copy this UUID now.

  • Add an fstab entry

    sudo nano /etc/fstab

    You need to add the following information:

    <device/UUID> <mount-point> <file-system> <mount-options> <dump> <pass>

    The entry should look like this and you can add it at the end of the file:

    UUID=<your-UUID>   /mnt/<your-mount-point>   ext4   defaults   0 2

    Once you added the line, save your changes.

Step 2.3 - Add a hot spare (Optional)

Hot spare drives/partitions in a RAID are normally not used by the RAID. They are only used if one of the active drives/partitions of the RAID system has an error or is defective. If no hot spare drive is defined in a software RAID, you need to manually start the rebuild of a defective RAID yourself. If a hot spare is present, the rebuild will start automatically. You can add a hot spare drive with mdadm --add.

You can view this information with sudo mdadm --detail /dev/md0.

• Set a hot spare
  

  RAID: md0
  Devices: sdb1, sdc1
  Hot spare: sdd1

  This example will use sdd1 as the hot spare. Before you add a new device to the RAID, you need to make sure it is formatted as seen with sdb and sdc in "Step 2.1". Once it is formatted, you can add the partition as hot spare:

  sudo mdadm --add /dev/md0 /dev/sdd1

• Use the hot spare
  If sdc1 was removed from the system, for example, RAID would automatically rebuild. It would then look like this:

  RAID: md0
  Devices: sdb1, sdd1
  Defect: sdc1

• Add a new hot spare
  When sdc1 works again, it is not automatically readded to the RAID. However, you can manually add it as a hot spare.

  sudo mdadm --add /dev/md0 /dev/sdc1

  RAID: md0
  Devices: sdb1, sdd1
  Hot spare: sdc1

Step 3 - Removing a software RAID

To remove a software RAID, you need to perform the following steps:

Example
RAID: md0
Devices: sdb1, sdc1

1. Stop the RAID

   sudo umount /dev/md0
   sudo mdadm --stop /dev/md0

2. Remove automatic mount entries (e.g. /etc/fstab)

   nano /etc/fstab

   You need to remove the mount entry you added previously. In the example from step 2, the entry looked like this:

   UUID=<your-UUID>   /mnt/<your-mount-point>   ext4   defaults   0 2

   After you made your changes, update initramfs:

   sudo update-initramfs -u

3. Delete RAID entries in mdadm.conf

   nano /etc/mdadm/mdadm.conf

   You need to remove your RAID entry. It should look like this:

   ARRAY /dev/md0 metadata=1.2 name=<your-server-name>:1 UUID=<your-UUID>

4. Delete the superblock of the used partitions

   sudo mdadm --zero-superblock /dev/sdb1 /dev/sdc1

5. Disable the RAID flag

   sudo parted /dev/sdb set 1 raid off
   sudo parted /dev/sdc set 1 raid off

Step 4 - Managing a software RAID

Step 4.1 - Get RAID status

You can get a short list of all RAIDs in the system with the output of the file /proc/mdstat.

$ cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sdc1[0]
       8380416 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdb3[1] sdc3[0]
       536739840 blocks super 1.2 [2/2] [UU]
       bitmap: 3/4 pages [12KB], 65536KB chunk

md1 : active raid1 sdb2[1] sdc2[0]
       1047552 blocks super 1.2 [2/2] [UU]

unused devices: <none>

To get more detailed information, you can use this command:

sudo mdadm --detail /dev/md2

/dev/md2:
           Version : 1.2
     Creation Time : Fri Feb 22 17:19:37 2019
        Raid Level : raid1
        Array Size : 536739840 (511.88 GiB 549.62 GB)
     Used Dev Size : 536739840 (511.88 GiB 549.62 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sun May 26 13:49:02 2019
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : rescue:2
              UUID : c76e5446:f3c41fbc:0b62ca86:803a7e88
            Events : 2170

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sdc3
       1       8       19        1      active sync   /dev/sdb3

Step 4.2 - Replace defective drive

The commands below use the following example:

RAID: md0
Devices: sdb1, sdc1
Defect: sdb1
Replacement: sdd1

The following steps explain how to remove the defective partition sdb1 from the RAID and how to add the working partition sdd1 as a replacement.

• Remove the defective drive from the RAID
  First you need to mark the defective drive as "failed" and remove it from the RAID:

  sudo mdadm --manage /dev/md0 --fail /dev/sdb1
  sudo mdadm --manage /dev/md0 --remove /dev/sdb1

  You can now replace the defective drive with a new working drive. When the drive is available on the server, you can partition it and add it to the RAID.

• Partition the new drive
  If no hot spare drive is available, you need to partition a new drive. It is important that the new drive has the same partitioning as the defective drive!

  To partition the new drive, it is sufficient to copy the partition table from an existing drive.

  • For MBR partitioned drives:

    sudo sfdisk --dump /dev/sdc > sdc_parttable_mbr.bak # Creates a backup of the partition table
    sudo sfdisk -d /dev/sdc | sudo sfdisk /dev/sdd      # Copies the partition table from sdc to sdd

  • For GPT partitioned drives:

    sgdisk --backup=sdc_parttable_gpt.bak /dev/sdc      # Creates a backup of the partition table
    sgdisk --load-backup=sdc_parttable_gpt.bak /dev/sdd # Copies the created backup of the partition table to sdd

• Add working device to the RAID system
  When the new drive is partitioned correctly, you can add it to the RAID system:

  sudo mdadm --manage /dev/md0 -a /dev/sdd1

• Rebuild
  It should automatically start to rebuild the RAID. You can view the process with watch cat /proc/mdstat.

  When the rebuild of your RAID is done, you can use sudo mdadm --detail /dev/md0 to check if you have 2 Active Devices again.

Note: If the system is on the RAID itself, it is necessary to install the bootloader on the appropriate drive. You can do this with the following command:

update-grub
grub-install /dev/sda

Step 4.3 - Expand RAID

When you expand your RAID, make sure you know what you are doing. There is always a risk that you might lose your data.

Only RAIDs with levels 1, 4, 5, and 6 can be expanded.

Following steps are necessary:

• Add the additional drive/partition to the RAID
• Change the RAID level
• Change the size of the file system

The example commands below use this example:

You can run sudo mdadm --detail /dev/md0 to get this information about your RAID.

• Add additional drive to the RAID
  You need to add the new partition as a hot spare first:

  sudo mdadm /dev/md0 --add /dev/sdd1

  You can run sudo mdadm --detail /dev/md0 to check if sdd1 is now available as hot spare.

• Change the RAID level
  Now the RAID can be extended with the new drive:

  sudo mdadm --grow --raid-devices=3 --level=5 /dev/md0 --backup-file=/root/md0.bak

  Note: In the file specified by --backup-file critical areas are saved (typically a few MiB). If the system crashes during the extension, the extension can be continued later using the following command:

  sudo mdadm /dev/md0 --continue --backup-file=/tmp/md0.bak

  The backup file must not be located on the RAID to be extended! The use of backup-file is not mandatory, but strongly recommended.

• Adapt the mdadm.conf entry
  If the entry in the mdadm.conf file includes information about the RAID level and the number of devices, you need to update the information accordingly. You can edit the file with this command:

  nano /etc/mdadm/mdadm.conf

  • If the entry does not include any information about the RAID level or the number of devices, you do not need to change anything. Example:

    ARRAY /dev/md0 metadata=1.2 name=<your-server-name>:0 UUID=<your-UUID>

  • If the entry includes information about the RAID level (level=1) and the number of devices (num-devices=2), you need to update the information accordingly. Example:

    ARRAY /dev/md0 level=5 num-devices=3 metadata=1.2 name=<your-server-name>:0 UUID=<your-UUID>

• Change the file system size
  To use the new space, you need to expand the file system.
  • You can view the current size with this command:

    df -h /mnt/<your-mount-point>

  • You can expand the file system with these commands:

    sudo umount /dev/md0 /mnt/<your-mount-point> # Unmount the file system
    sudo fsck.ext4 -f /dev/md0                   # Force the check, even if it has recently been checked
    sudo resize2fs /dev/md0                      # Extend file system to maximum size
    sudo mount /dev/md0 /mnt/<your-mount-point>  # Mount the file system again

    If you get an error message like target is busy., you might still have watch processes running. You can use lsof +D /mnt/<your-mount-point> to list all processes that are using this directory. If the process is not important, you can use kill -9 <PID> to kill the process.

  • If you run the command from before, you can compare the new size to the old size:

    df -h /mnt/<your-mount-point>

Step 4.4 - RAID monitors

To monitor the RAID, this entry can be stored as a crontab (sudo crontab -e):

0 0 * * * /usr/share/mdadm/checkarray --cron --all --quiet >/dev/null 2>&1 # Runs every day at 00:00 AM

Conclusion

This tutorial explained how to select a suitable RAID level for your project and then configure it on Linux systems using mdadm. Furthermore, administrative tasks like expanding a RAID or exchanging defective hard disks were discussed.

License: MIT