Configuring Software RAID on Linux Using MDADM

MDADM is a tool that allows to create and manage software RAIDs on Linux. In this article we’ll show how to use mdadm (multiple disks admin) to create RAID array storage, add and manage disks, add a hot-spare and more.

mdadm: How to Install a Software Raid Management Tool?

To install mdadm, run the installation command:

• For CentOS/Red Hat (yum/dnf is used): yum install mdadm
• For Ubuntu/Debian: apt-get install mdadm

mdadm and the dependent libraries will be installed:

Running transaction
Installing : libreport-filesystem-2.1.11-43.el7.centos.x86_64 1/2
Installing : mdadm-4.1-1.el7.x86_64 2/2
Verifying : mdadm-4.1-1.el7.x86_64 1/2
Verifying : libreport-filesystem-2.1.11-43.el7.centos.x86_64 2/2
Installed:
mdadm.x86_64 0:4.1-1.el7
Dependency Installed:
libreport-filesystem.x86_64 0:2.1.11-43.el7.centos
Complete!

Creating RAID 1 (Mirror) Using 2 Disks on Linux

I have two extra disks installed on my Linux server, and I want to create a software mirror on them (RAID1). The drives are empty. First of all, you have to zero all superblocks on the disks to be added to the RAID:

# mdadm --zero-superblock --force /dev/vd{b,c}

I have two clean disks: vdb and vdc.

mdadm: Unrecognised md component device - /dev/vdb
mdadm: Unrecognised md component device - /dev/vdc

This listing means that neither of the disks have ever been added to an array.

To create a software RAID1 from two disks to the device /dev/md0, use this command:

# mdadm --create --verbose /dev/md0 -l 1 -n 2 /dev/vd{b,c}

Where ‘-l 1’ is the array type (RAID1 in our case,

and ‘-n 2’ is the number of disks added to the array.

After you enter the commands, confirm the actions and the software RAID will be created:

If you list the information about your disks, you will see your RAID md0 drive:

# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 20G 0 disk
├─vda1 253:1 0 512M 0 part /boot
└─vda2 253:2 0 19.5G 0 part /
vdb 253:16 0 20G 0 disk
└─md0 9:0 0 20G 0 raid1
vdc 253:32 0 20G 0 disk
└─md0 9:0 0 20G 0 raid1

In order to create an ext4 file system on your RAID1 drive, run this command:

# mkfs.ext4 /dev/md0

Create a backup directory and mount the RAID device to it:

# mkdir /backup
# mount /dev/md0 /backup/
# df -h

Filesystem Size Used Avail Use% Mounted on
.............
.............
/dev/md0 20G 45M 19G 1% /backup

The array has been mounted without any errors. In order not to mount the device each time manually, make the following changes to fstab:

# nano /etc/fstab

/dev/md0 /backup ext4 defaults 1 2

How to View State or Check the Integrity of a RAID Array?

To check data integrity in the array, use the following command:

#echo 'check' > /sys/block/md0/md/sync_action

Then view the output of the following file:

#cat /sys/block/md0/md/mismatch_cnt

If you get 0, your array is OK:

To stop the check, run the following:

#echo 'idle' > /sys/block/md0/md/sync_action

To check the state of all RAIDs available on the server, use this command:

# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 vdc[1] vdb[0]
20954112 blocks super 1.2 [2/2] [UU]

You can view more detailed information about the specific RAID using this command:

# mdadm -D /dev/md0

Let’s consider the main items in the command listing:

• Version – the metadata version
• Creation Time – the date and time of RAID creation
• Raid Level – the level of a RAID array
• Array Size – the size of the RAID disk space
• Used Dev Size – the space size used by devices
• Raid Device – the number of disks in the RAID
• Total Devices – is the number of disks added to the RAID
• State – is the current state (clean — it is OK)
• Active Devices – number of active disks in the RAID
• Working Devises – number of working disks in the RAID
• Failed Devices – number of failed devices in the RAID
• Spare Devices – number of spare disks in the RAID
• Consistency Policy – is the parameter that sets the synchronization type after a failure, rsync is a full synchronization after RAID array recovery (bitmap, journal, ppl modes are available)
• UUID – raid array identifier

You can view the brief information using fdisk:

# fdisk -l /dev/md0

Disk /dev/md0: 21.5 GB, 21457010688 bytes, 41908224 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Recovering from a Disk Failure in RAID, Disk Replacement

If one of the disks in a RAID failed or damaged, you may replace it with another one. First of all, find out if the disc is damaged and needs to be replaced.

# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 vdb[0]
20954112 blocks super 1.2 [2/1] [U_]

From the previous command you can see that only one disk is active. [U_] also means that a problem exists. When both disks are healthy, the output is [UU].

The detailed information about the RAID also shows that there are some problems:

# mdadm -D /dev/md0

/dev/md0:
Version : 1.2
Creation Time : Tue Dec 31 12:39:22 2020
Raid Level : raid1
Array Size : 20954112 (19.98 GiB 21.46 GB)
Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Dec 31 14:41:13 2020
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
State : clean, degraded

– the last line shows that the one disk in the RAID is damaged.

In our case, /dev/vdc must be replaced. To restore the array, you must remove the damaged disk and add a new one.

Remove the failed drive:

# mdadm /dev/md0 --remove /dev/vdc

Add a new disk to the array:

# mdadm /dev/md0 --add /dev/vdd

Disk recovery will start automatically after you add a new disk:

# mdadm -D /dev/md0

/dev/md0:
Version : 1.2
Creation Time : Tue Dec 31 12:39:22 2020
Raid Level : raid1
Array Size : 20954112 (19.98 GiB 21.46 GB)
Used Dev Size : 20954112 (19.98 GiB 21.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Dec 31 14:50:20 2020
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Consistency Policy : resync
Rebuild Status : 48% complete
Name : host1:0 (local to host host1)
UUID : 9d59b1fb:7b0a7b6d:15a75459:8b1637a2
Events : 42
Number Major Minor RaidDevice State
0 253 16 0 active sync /dev/vdb
2 253 48 1 spare rebuilding /dev/vdd
rebuild Status : 48% complete shows the current array recovery state.
spare rebuilding /dev/vdd shows which disk is being added to the array.
After rebuilding the array, check its state:
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

How to Add or Remove Disks to Software RAID on Linux?

If you need to remove the previously created mdadm RAID device, unmount it:

# umount /backup

Then run this command:

# mdadm -S /dev/md0

mdadm: stopped /dev/md0

After destroying the RAID array, it won’t detected as a separate disk device:

# mdadm -S /dev/md0

mdadm: error opening /dev/md0: No such file or directory

You can scan all connected drives and re-create a previously removed (failed) RAID device according to the metadata on physical drives. Run the following command:

# mdadm --assemble —scan

If you want to remove an operable drive from an array and replace it, first tag the drive as a failed one:

# mdadm /dev/md0 --fail /dev/vdc

Then you can remove it using this command:

# mdadm /dev/md0 --remove /dev/vdc

You can add a new disk, just like in case of a failed drive:

# mdadm /dev/md0 --add /dev/vdd

How to Add a Hot-Spare Drive to an MDADM Array?

You can add an extra hot-spare drive for quickly rebuild the RAID array if one of the active disks fails. Add a free disk to the md device you want:

# mdadm /dev/md0 --add /dev/vdc

When you check the RAID status, we will see the disk as a spare:

To make sure that the hot-swap works, mark any drive as failed and check the RAID status:

# mdadm /dev/md0 --fail /dev/vdb

After checking, you can see that the rebuilding of the array has started.

The /dev/vdb disk is marked as a failed, and the hot-spare disk became one of the active RAID disks. So the rebuild process has started.

To add an additional operable disk to the RAID, you must follow these two steps.

Add an empty drive to the array:

# mdadm /dev/md0 --add /dev/vdb

Now this disk will be displayed as hot-spare. To make it active, expand the md RAID device:

# mdadm -G /dev/md0 —raid-devices=3

Then the array will be rebuilt:

After the rebuild, all the disks become active:

Number Major Minor RaidDevice State
3 253 32 0 active sync /dev/vdc
2 253 48 1 active sync /dev/vdd
4 253 16 2 active sync /dev/vdb

How to Remove an MDADM RAID Array?

If you want to permanently remove your software RAID drive, use the following scheme:

# umount /backup – unmount the array from the directory

# mdadm -S /dev/md0 — stop the RAID device

Then clear all superblocks on the disks it was built of:

# mdadm --zero-superblock /dev/vdb
# mdadm --zero-superblock /dev/vdc

Mdmonitor: RAID State Monitoring & Email Notifications

The mdmonitor daemon can be used to monitor the status of the RAID. First, you must create the /etc/mdadm.conf file containing the current array configuration:

# mdadm –detail –scan > /etc/mdadm.conf

Add to the end of /etc/mdadm.conf the administrator email address to which you want to send notifications in case of any RAID problems:

MAILADDR raidadmin@woshub.com

Then restart mdmonitor service using systemctl:

# systemctl restart mdmonitor

Then the system will notify you by e-mail if there are any mdadm errors or faulty disks.

Inactive MDADM RAID

In case of hardware failures or emergency shutdowns, the software RAID array may become inactive. All drives are marked as inactive, but have no errors.

# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : inactive vdc[1] vdb[0]
20954112 blocks super
unused devices: <none>

In this case, you need to stop the array using this command:

# mdadm --stop /dev/md0

And reassemble it:

# mdadm --assemble --scan –force

If the md device is registered in /etc/fstab, remount it using this command:

# mount -a

mdadm makes it easier to manage software RAIDs on Linux. In this article I have described the main things when working with the tool, and covered the typical questions that arise when working with RAIDs using mdadm.