Create raid array
Create raid 6 array with 4 disks
$ mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sd[b-e]
Save your raid configuration
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
in
/etc/mdadm/mdadm.conf rename
ARRAY /dev/md0 metadata=1.2 name=ion:1 UUID=aa1f85b0:a2391657:cfd38029:772c560e
to:
ARRAY /dev/md0 UUID=aa1f85b0:a2391657:cfd38029:772c560e
and recreate the
initrd for the kernel and include the configuration files relevant for the MD-RAID configuration
$ sudo update-initramfs -u
Create filesystem
To create our file system for best performance we need to calculate our stride and stripe sizes.
Stride size: divide the array chunk size by the file system block size.
We find the
Array chunk size by looking at
/proc/mdstat or using
mdadm
$ cat /proc/mdstat
... 512k chunk ...
$ mdadm --detail /dev/md{...}
...
Chunk Size : 512K
...
A block size of 4k offers best performance for ext4.
Therefore, in the above example, stride size is 512 / 4 = 128
Stripe size: multiply the stripe by the number of data disks.
In raid-6, 2 disks are used for parity, and in raid-5, 1 disk is used for parity.
In my example I have 4 disks and am using raid-6, therefore I have 2 data disks.
Therefore, I have a stripe size of 128 * 2 = 256.
Create the file system:
$ mkfs.ext4 -b 4096 -E stride=128,stripe-width=256 /dev/md0
Mount filesystem
$ sudo mkdir /mnt/raid
$ sudo chmod 1777 /mnt/raid
$ sudo mount -o noauto,rw,async -t ext4 /dev/md0 /mnt/raid
Make it permanent - add to /etc/fstab
/dev/md0 /mnt/raid ext4 defaults 1 2
Export via NFS
$ sudo apt-get install nfs-kernel-server
Mount the raid drive in the exported tree
$ sudo mkdir /export
$ sudo mkdir /export/raid
$ sudo mount --bind /mnt/raid /export/raid
Make it permanent - add to /etc/fstab
/mnt/raid /export/raid none bind 0 0
Configure /etc/exports
/export 192.168.1.0/24(rw,fsid=0,insecure,no_subtree_check,async)
/export/raid 192.168.1.0/24(rw,nohide,insecure,no_subtree_check,async)
Start the service
$ sudo service nfs-kernel-server restart
smartmontools: monitor S.M.A.R.T. ( (Self-Monitoring, Analysis and Reporting Technology) attributes and run hard drive self-tests.
enable SMART support if it's not already on
$ for i in `ls -1 /dev/sd[a-z]`; do smartctl -s on $i; done
turn on offline data collection
$ for i in `ls -1 /dev/sd[a-z]`; do smartctl -o on $i; done
enable autosave of device vendor-specific attributes
$ for i in `ls -1 /dev/sd[a-z]`; do smartctl -S on $i; done
check the overall health for each drive
$ for i in `ls -1 /dev/sd[a-z]`; do RESULTS=`smartctl -H $i | grep result | cut -f6 -d' '`; echo $i: $RESULTS; done
If any drive doesn't show PASSED, immediately backup all your data as that drive is probably about to fail.
Configure smartd to automatically check drives
$ vim /etc/smartd.conf
DEVICESCAN -H -m root -M exec /usr/libexec/smartmontools/smartdnotify -n standby,10,q
DEVICESCAN means scan for all devices
-H means monitor SMART health status
-m root means mail to root
-M exec /usr/libexec/smartmontools/smartdnotify means run the smartdnotify script to email warnings
-n standby,10,q means don't take the disk out of standby except if it's been in standby 10 times in a row, and don't report the unsuccessful attempts.
pick up changes
service smartd restart
Other options for smartd.conf:
-a \ # Implies all standard testing and reporting.
-n standby,10,q \ # Don't spin up disk if it is currently spun down
\ # unless it is 10th attempt in a row.
\ # Don't report unsuccessful attempts anyway.
-o on \ # Automatic offline tests (usually every 4 hours).
-S on \ # Attribute autosave (I don't really understand
\ # what it is for. If you can explain it to me
\ # please drop me a line.
-R 194 \ # Show real temperature in the logs.
-R 231 \ # The same as above.
-I 194 \ # Ignore temperature attribute changes
-W 3,50,50 \ # Notify if the temperature changes 3 degrees
\ # comparing to the last check or if
\ # the temperature exceeds 50 degrees.
-s (S/../.././02|L/../../1/22) \ # short test: every day 2-3am
\ # long test every Monday 10pm-2am
\ # (Long test takes a lot of time
\ # and it should be finished before
\ # daily short test starts.
\ # At 3am every day this disk will be
\ # used heavily as backup storage)
-m root \ # To whom we should send mails.
-M exec /usr/libexec/smartmontools/smartdnotify
Note: this will email root - if you don't monitor root's mails, then you may want to redirect mails sent to root to another email address
root: user@gmail.com
Now we want to monitor the RAID array.
add the to/from email addresses to mdadm config
$ vim /etc/mdadm.conf
MAILADDR user@gmail.com
MAILFROM user+mdadm@gmail.com
I tested this was working by running
/sbin/mdadm --monitor --scan --test
gmail automatically marked the test mail as spam, so I had to create a filter to explicitly not mark emails sent from user+mdadm@gmail.com as spam (note the +mdadm part of the email address,
neat gmail trick)
replace a failed drive
find the drive's serial no
$ hdparm -i /dev/sdd | grep SerialNo
Model=WDC WD2003FZEX-00Z4SA0, FwRev=01.01A01, SerialNo=WD-WMC130D78F55
fail and remove the drive from the array
$ sudo mdadm --manage /dev/md0 --fail /dev/sdd
$ sudo mdadm --manage /dev/md0 --remove /dev/sdd
remove the old drive (using the above serial no to ensure you remove the correct drive), add the new one and add it to the array
$ sudo mdadm --manage /dev/md0 --add /dev/sdd
the array should start rebuilding
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdd[4] sde[3] sdb[1] sdc[0]
3906765824 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/3] [UU_U]
[>....................] recovery = 0.5% (11702016/1953382912) finish=970.5min speed=33344K/sec
unused devices: <none>
Tools for monitoring:
logwatch – monitors my /var/log/messages for anything out of the ordinary and mails me the output on a daily basis.
mdadm – mdadm will mail me if a disk has completely failed or the raid for some other reason fails. A complete resync is done every week.
smartd – I have smartd running “short” tests every night and long tests every second week. Reports are mailed to me.
munin – graphical and historical monitoring of performance and all stats of the server.
