mdadm RAID 1 configuration various operations in RHEL

In this article I am going to explain the use of soft RAID in linux. I am using RAID-1 in this whole article. Apart from simple configuration I have shown the  couple of scenarios.
Scenario 1 : Increasing the soft RAID.
Scenario 2 : Use of Spare Disk in RAID-1.
Scenario 3 : Increasing the redundancy by adding one more mirror disk.
Scenario 4 : Removing the newly added mirror.

First we will start with basic creation of RAID-1 array and creating one file system on top of it and checking the status of RAID.

Step 1 : Creating RAID 1 device using two disks. Each disk partition is of 200MB.

[root@localhost ~]# mdadm –create –verbose /dev/md1 –level=1 –raid-devices=2 /dev/sdc1 /dev/sdd1

Step 2 : Created one ext4 file system on top it and mounted it.

[root@localhost ~]# mkdir /RAID-1

[root@localhost ~]# mount /dev/md1  /RAID-1/

[root@localhost ~]# df -h /RAID-1/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        190M  1.6M  175M   1% /RAID-1

Step 3 : Checking the status of soft RAID.

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:22:28 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 17

Number   Major   Minor   RaidDevice State
0       8       33        0      active sync   /dev/sdc1
1       8       49        1      active sync   /dev/sdd1

Scenario 1 : If we want to increase the size of mdadm raid1. Current size of file system is 200MB (approx) now if I want to increase the size of softraid. I need to replace the existing partitions with partitions of larger size. In this case I am doing the below.

Replacing /dev/sdc1 of 200MB with /dev/sdc3 of 600MB (approx)
Replacing /dev/sdd1 of 200MB with /dev/sdd3 of 600MB (approx)

Step 1 : I need to mark the existing partition as failed one and check the status.

[root@localhost ~]# mdadm –manage /dev/md1 –fail /dev/sdc1
mdadm: set /dev/sdc1 faulty in /dev/md1

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:27:09 2014
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 19

Number   Major   Minor   RaidDevice State
0       0        0        0      removed
1       8       49        1      active sync   /dev/sdd1

0       8       33        –      faulty   /dev/sdc1

Step 2 : Still the file system is accessible because it was present in RAID.

[root@localhost ~]# cd /RAID-1/

[root@localhost RAID-1]# ll
total 102414
drwx——. 2 root root     12288 Nov 16 08:21 lost+found
-rw-r–r–. 1 root root 104857600 Nov 16 08:21 testfile1

[root@localhost RAID-1]# dd if=/dev/zero of=/RAID-1/testfile2 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.106767 s, 98.2 MB/s

Step 3 : Now lets add the new partition on same disk which is of larger size.

[root@localhost RAID-1]# mdadm –manage /dev/md1 –add /dev/sdc3
mdadm: added /dev/sdc3

[root@localhost RAID-1]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:29:03 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 48

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
1       8       49        1      active sync   /dev/sdd1

0       8       33        –      faulty   /dev/sdc1

Step 4 : Lets do the same thing with another disk partition (/dev/sdd1) as well.

[root@localhost RAID-1]# mdadm –manage /dev/md1 –fail /dev/sdd1
mdadm: set /dev/sdd1 faulty in /dev/md1

[root@localhost RAID-1]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:30:35 2014
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 2
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 50

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
1       0        0        1      removed

0       8       33        –      faulty   /dev/sdc1
1       8       49        –      faulty   /dev/sdd1

[root@localhost RAID-1]# mdadm –manage /dev/md1 –add /dev/sdd3
mdadm: added /dev/sdd3

Step 5 : Check the status now and then removed the old disks.

[root@localhost RAID-1]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:31:18 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 71

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

0       8       33        –      faulty   /dev/sdc1
1       8       49        –      faulty   /dev/sdd1

After verification I have removed the disks from RAID array.

[root@localhost RAID-1]# mdadm –manage /dev/md1 –remove /dev/sdc1
mdadm: hot removed /dev/sdc1 from /dev/md1

[root@localhost RAID-1]# mdadm –manage /dev/md1 –remove /dev/sdd1
mdadm: hot removed /dev/sdd1 from /dev/md1

Now final output becomes.

[root@localhost RAID-1]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 204608 (199.85 MiB 209.52 MB)
Used Dev Size : 204608 (199.85 MiB 209.52 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:31:52 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 73

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3
[root@localhost RAID-1]#

Step 6 : Extend the size of mdadm so that it can reflect the correct size.

[root@localhost ~]# mdadm -G /dev/md1 -z max
mdadm: component size of /dev/md1 has been set to 637824K
[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:34:16 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 86

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

Step 7 : Now we can resize the file system to reflect the correct size.

[root@localhost ~]# resize2fs /dev/md1
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/md1 is mounted on /RAID-1; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 3
The filesystem on /dev/md1 is now 637824 blocks long.

[root@localhost ~]# df -h /RAID-1/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        600M  113M  457M  20% /RAID-1

Scenario 2 : Showing use of spare in mdadm.

Step 1 : I have added the new disk to array configuration of same size of other two disks which were already present thr.

[root@localhost ~]# mdadm –manage /dev/md1 –add /dev/sde3
mdadm: added /dev/sde3

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:46:06 2014
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 147

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

4       8       67        –      spare   /dev/sde3

Step 2 : For showing that spare is getting activated if the active disk is getting failed.

[root@localhost ~]# mdadm –manage /dev/md1 –fail /dev/sdd3
mdadm: set /dev/sdd3 faulty in /dev/md1

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:46:31 2014
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1

Rebuild Status : 62% complete

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 159

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
4       8       67        1      spare rebuilding   /dev/sde3

3       8       51        –      faulty   /dev/sdd3

Scenario 3 : If I want to add one more disk as mirror to configuration which we created initially.

Step 1 : Add the disk to existing RAID-1 configuration. New disk will be shown in spare status.

[root@localhost ~]# mdadm –add /dev/md1 /dev/sde3
mdadm: added /dev/sde3

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:37:52 2014
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 87

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

4       8       67        –      spare   /dev/sde3

Step 2 : I have increased the raid devices to 3 from 2 which we set earlier during intial creation. As soon as I did that I can see that rebuilding has started.

[root@localhost ~]# mdadm –grow /dev/md1 –raid-devices=3
raid_disks for /dev/md1 set to 3

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:38:23 2014
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Rebuild Status : 94% complete

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 106

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3
4       8       67        2      spare rebuilding   /dev/sde3

Scenario 4 : Reducing the mirror which we added in previous step.

Step 1 : Mark the newly added disk as faulty disk.

[root@localhost ~]# mdadm –manage /dev/md1 –fail /dev/sde3

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:42:57 2014
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 140

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3
2       0        0        2      removed

4       8       67        –      faulty   /dev/sde3

Step 2 : Change the soft raid device to 2 from 3 which we did in previous scenario.

[root@localhost ~]# mdadm –grow /dev/md1 –raid-devices=2
raid_disks for /dev/md1 set to 2

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:43:13 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 145

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

4       8       67        –      faulty   /dev/sde3

Step 3 : We can remove the faulty disk from configuration.

[root@localhost ~]# mdadm –manage /dev/md1 –remove /dev/sde3
mdadm: hot removed /dev/sde3 from /dev/md1

[root@localhost ~]# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Sun Nov 16 08:20:31 2014
Raid Level : raid1
Array Size : 637824 (622.98 MiB 653.13 MB)
Used Dev Size : 637824 (622.98 MiB 653.13 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Sun Nov 16 08:43:34 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : localhost.localdomain:1  (local to host localhost.localdomain)
UUID : a3caf018:baf715ab:99714d90:59ec7d26
Events : 146

Number   Major   Minor   RaidDevice State
2       8       35        0      active sync   /dev/sdc3
3       8       51        1      active sync   /dev/sdd3

Tip : Don’t forget to issue mdadm –detail –scan > /etc/mdadm.conf to make this persistent across reboots.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s