Mark's Business Consulting, Support & Services Blog
Setting up Mirrored Software (MDADM) RAID1 Array on Ubuntu 14.04 Without Loosing Existing Data On Disks Larger Than 2TB.
Tuesday 17th November 2015
This is a technical article about Ubuntu and setting up RAID arrays. A little background: I'm all for supporting Open Source innitiatives that give people an alternative choice to the big multinationals like MicroSoft and Apple. It's not mainstream knowledge that both Windows and Apple (OS X) are actually based on or ripped off from UNIX. Ubuntu (a flavour of UNIX) is a freely available Operating system which runs on most hardware and is a credible alternative to Windows and OSX. Of course with any OS there are software compatability issues, try running your Windows software on a Mac for example! Having used Apple, Windows and UNIX for over 20 years now I've got a reasonable idea of the strengths and weaknesses of each. I'd say Apple OS X is my favorite OS but... that's because it's basically sitting on Unix. The downside - you have to buy Apple hardware which is overpriced. Ubuntu runs on cheap PC hardware and is pretty sleek so comes in a very close second (or first place when you weigh up the cost benefits and freedom). Windows... well lets just say no other operating system has caused so much downtime, heartache and loss of data/earnings/time/sanity. I'm a geek and pretty experienced so if I struggle then god help the average person out there. It's for this reason that I've decided to start making available my precedures for doing stuff on Ubuntu and other Open Source frameworks. I want people to have the choice and support to make the (informed) move away from having to pay to stay with MicroSoft and/or Apple. Of course I still use Apple and Microsoft OS's too and fully support our clients who are based on those systems. It's about presenting a choice. Thus ends my preamble to what is my first Ubuntu/OpenSource post.
A Guide to Understanding this Document:
While reading this document I have put command line examples within ===== lines. I've also placed a $ sign at the start to indicate it is a command line. I'm assuming you know how to do command line terminal stuff.
The Scenario (aka Why you want to setup a RAID on Ubuntu 14.04)
You have a 3TB disk full of data and you want to add a second disk, which is empty and then create a mirrored RAID (Redundant Array of Independent Disks) with the data in it. Which is another way of saying:
“How to I create a RAID Array without Loosing my Data on One of the Disks?”
Setting the scene:
- /dev/sdc [3TB] => This is the disk with the data which we don't want to loose.
- /dev/sdd [3TB] => This is the empty disk without any data
In this scenario I have two disks in a PC, which used to be running Windows 7. I've moved all the data onto disk sdc leaving sdd empty.
The disks are identical spec, both being 3TB in capacity.
The disks were obviously formatted for Windows 7 before the box was migrated to Ubuntu 14.04 and so they have additional small partitions with Windowsy stuff on them. The main partition on each disk is currently NTFS which we'll want to change.
- Create a mirrored RAID (RAID level 1) using both disks.
- Formatted as EXT4 (for Unix).
- Capable of more than 2TB partition size.
- Without losing any data.
1. Install MDADM on Ubunto 14.04 using Apt
$ sudo apt-get install mdadm
2. Format the empty disk /dev/sdd ready for converting to RAID
I did this using Disks application.
I removed all partitions from the disk and then formatted it as a single partition.
It's worth noting that in Ubuntu 14.04 the system can format partitions beyond 2TB. If you want to you can obviously do the partitioning from the command line too.
3. Setup a single disk RAID1 Partition (to be called /dev/md0) on the newly formatted disk.
First you have to unmount the disk so it is not “busy”. You can do this by clicking on the Files application and clicking the unmount icon next to the drive.
$ sudo mdadm --create --verbose /dev/md0 --force --level=1 --raid-devices=1 /dev/sdd1
4. You now have an unformatted RAID array called /dev/md0
Congratulations you now have a 1 disk RAID. We're not done yet but this is a start! Of course a 1 disk RAID is just a Disk... but we'll add more disks later.
5. Format the Newly Created RAID Array
Because this disk is greater than 2TB I have opted for the option below first time I tried this. You want EXT4. Go ahead and format it! -(Not seeing EXT4 in the pulldown read the "Important Note" below first!)
IMPORTANT NOTE:- Unlike shown above you need to format it to Ext4 so it will mount. If you see the above view in the format Disk pulldown then the system hasn't behaved. Restart the process by stopping the raid, reformatting the original disk partition and doing step 3 again. There's more notes on how to handle problems at the bottom. I'm sharing it at this point as I hit this snag first time round and it took a few hours to figure out what to do. I'd rather you find it easier! Sharing is caring!
6. Copy All Your Data From the Data Disk to This RAID Partition.
It's tempting to use the GUI (Graphical User Interface) to copy the files. But when copying large amounts of data I prefer to use the command line, as that's a little bit more stable. Use cp or rsnyc etc to copy the files over. You could of course just move the files using (mv) but if anything goes wrong you may be left in a mess. Using copy and then delete means you still have the original data files for rollback if there is a problem mid way.
How long will it take? To give you an idea. On this system it took about 2.5 hours for 1.4 TB of data!
Verifying Your Data has Copied Across
Once the copying had finished check the size of the two drives are equal using df (RAID verses Data)
$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 188909348 16316724 162973476 10% /
/dev/sdc2 2930130940 1589123424 1341007516 55% /media/username/DataDisk
/dev/md0 2884024800 1589004620 1148497032 59% /media/username/6d11246b...70cf3f2856
You can see they are both about 1.589TB of data which is almost spot on... good enough.
I also did a quick visual check too.
7. Prepare the Data Drive to Be Added To the Mirrored RAID Array (aka Grow your RAID Array)
Because this was previously a windows disk I'm deleting the old partitions and making it into a single partition disk.
Windows Hard Drive Before Formatting:
Eject the Drive before formatting it!
Footnote at this stage... "You're probably wondering why the hard drive is called MediaRAID when it's not in a RAID yet? Well this was it's name in the Windows 7 Build and there were two disks in a mirrored RAID controlled by Windows. Then one day for no reason, the RAID disappeared and I had two hard disks in the box. Did I mention how Windows has caused a lot of problems?"
Looking at the above image you can see "Data" - which is a 960GB Solid state hard drive. "MediaRAID" which is the drive with all the data and "3.0 TB Volume" which is the newly created RAID Array, note it's Icon is different. Of course your machine will be different. But this image shows you the GUI way to eject a disk. This view is within the "Files" applicaton.
Unix Hard Drive After Formatting:
8. Add the Newly Formatted Drive to the Raid Array
Now prepare the other drive to be included to the RAID by issuing the following command:
$ sudo mdadm --grow /dev/md0 --raid-devices=2 --add /dev/sdc1
mdadm: added /dev/sdc1
raid_disks for /dev/md0 set to 2
This will grow the RAID Array (md0), set the amount of devices on it to 2 (sdd1 + sdc1) by adding the reformatted sdc1 to it.
How To Check the Progress of a RAID in Rebuild/Repair Mode on Ubuntu 14.04?
You would expect the system to take some time to rebuild the RAID (syncing the two disks).
So you are looking for the comment underlined and in bold when you use mdadm --detail:
$ sudo mdadm --detail /dev/md0
Version : 1.2
Creation Time : Tue Nov 17 12:41:01 2015
Raid Level : raid1
Array Size : 2930135296 (2794.39 GiB 3000.46 GB)
Used Dev Size : 2930135296 (2794.39 GiB 3000.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Tue Nov 17 16:50:26 2015
State : clean, degraded, recovering
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
Rebuild Status : 0% complete
Name : Mark-Ubuntu:0 (local to host Mark-Ubuntu)
UUID : 077f37e4:1ed7745d:be0519c6:c274bb7f
Events : 380
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 33 1 spare rebuilding /dev/sdc1
You can also check the progress of the RAID rebuild from the Disks application:
Which shows that the array is degraded and the percentage of recovery.
How Long Will It Take to Rebuild a 3TB RAID 1 (Mirrored) Array?
According to the screen shot above this is going to take some time! To rebuild the 3TB raid with 1.6TB of data is going to take 6 days and 15 hours! And this is a beefy machine!
[18/Nov/2015 - editors note - It actually only took about 10-12 hours in total to rebuild the array. When I checked later on that evening it had jumped to 67% with 6 hours to go.]
Obviously once this is done I'd recommend doing a reboot to make sure the system (Nautilus etc) sees the RAID correctly.
You can also verify the raid is working by literally removing one of the drives while having the RAID unit mounted and still being able to view a video in it. In which case the output of mdadm --detail would be something like:
$ mdadm --detail
Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 65 1 active sync /dev/sdd1
During the process things may not go smoothly but don't panic. Should you find that the RAID /dev/md0 is mounting as /dev/md126 or /dev/md127 or your RAID is not mounting at all and the RAID is not visible to the Files program then read on.
As hinted at earlier the first time through steps 1 to 6 above I was given the option to format to GPT for the raid array. This was a mistake but because no other options presented I went with it. (A mistake in hindsight!)
The second time through (after going through these problem solving steps below) and deciding to start from the beginning, I was given the option to format to Ext4. If you're not given the option to format to Ext4 then go back a few steps and start again. To do this you would stop the RAID which you can do this from Disks app or by issuing the command mdadm –stop /dev/md0 or md127 etc. (read below)
Having RAID problems with Ububtu 14.04 Firstly check your RAID from the Command Line:
$ sudo cat /proc/mdstat
In the above you see the raid is identified as md127! It should have been mounted at md0!
You also note it is (auto-read-only) which means you can't write to it either.
This is a clue that it is not mounted and there is a problem. Reading around it seems the kernal ignores the mdadm.conf and tries to mount the RAID without proper instructions (defaulting to md126, md127 etc.).
The config file /etc/mdadm/mdadm.conf has the definitions for the RAID array:
First I backup the mdadm config file (do this all as root):
$ mv /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_20151117
Next I stop the raid array and do an assemble:
$ sudo mdadm --stop /dev/md0
$ sudo mdadm --assemble --scan
Then I create a new config file:
$ /usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf
I edited the /etc/mdadm/mdadm.conf config down to just the basic definition removing –name etc.:
# definitions of existing MD arrays
ARRAY /dev/md/0 UUID=c6db02cd:132d8584:37326913:8e459533
I then ran a command to update the initialisation of the file system:
$ sudo update-initramfs -u
You may also want to do a check of the disks:
$ fsck /dev/md0
Further Reading and Credits:
Obvioulsy I've put a bit of effort into collating everything here. If there's anything you want to contribute then please contact me and I'll update this. If you've found it helpful then please help me by sharing this article. There's buttons below to do so and of course you can link to it too. Thanks. Mark
Add a comment | Posted by Mark Zaretti at 10:11
The Internet is Awesome: Understand Internet Neutrality
As a professional web developer with clients all over the world it's my job to keep an eye on the "internet". One of the biggest challenges that will impact all of us is the constant attempt by corporations such as Comcast in the US to take over the web. With massive budgets and lobbying they are trying to make the internet a two tier system. This means that services like Vimeo, YouTube and Netflix will go up in price and that many websites will find themselves "slower" than their corporate competition. If you care about your internet and net neutrality then take a moment to find out more and click the button. You need the net, and the net needs you! Thanks, enjoy and share my blog.