LVM Disaster! Recovering Data
How I recovered my data when a Drive died only a few hours after installing and adding it to the logical volume.. And why I have no plans to EVER use LVM again!
‘, ‘
Now.. I know I broke all the rules for setting up a LVM system, but none of that should matter! What happened to me should have been VERY easy to recover from.. and actually wasn\’t all that hard.. but the fact that there is so much information on the internet about how to recover from LVM crashes.. and most of them not working or being out dated, etc.. make me think.. “Gee.. are the benefits of LVM worth the data risk?”.. Uh.. the answer is.. “HELL NO!”..
So.. here is what happened.. Keep in mind that this is my home data server.. I don\’t want to waste the data space to run RAID5. I also don\’t back up my data like I should.. so if I wouldn\’t have been able to recover this 110 GB LVM, I would have been screwed!
I built a new machine with 4 x 30GB Drives in Hot Swap bays on a FastTrack TX2 controller. I created a /boot partition on one drive and then added the rest to a Volume Group. I then partitioned the Volume Group into a swap and / partition. Everything was going great, I was migrating data from 2 other servers (that this one will replace) and I was very happy. I was going to be able to remove a 60 GB drive from one of the other servers and planned on replacing one of the 30 GB with the 60 GB for added space.
But, before I could remove all the data from the 60 GB drive, I ran out of space on the 4 x 30 LV. So, I grabbed another 30 GB drive, ran pvcreate and added it to VolGroup01. Now, The drive was part of the group, but I didn\’t realize you have to unmount the partition to resize it.. So, being that my root was on that partition (yeah.. that is one of the rules I broke), I wasn\’t able to use the added space. Well, not more then 10 hours later, the drive died. I knew it was dead, and I did copy about 11 GB worth of data from the server onto another machine and was going to back it up to DVDs when I decided to reboot the server.. BAD IDEA!
So, now I have a server that is kernel panicing on bootup because there is no root drive. That\’s great! A single dead drive kills the whole LVM array! Woohoo! Now that is just STUPID!
Anyway, so I searched the internet, read a crap load of mostly useless information that did not work and fought with this for 2 or 3 days. Now, I have installed a 250 GB drive and have recovered ALL of my data from the 4 x 30 GB LVM.. Needless to say, I DID NOT setup the 250 GB with an LVM!
So.. here\’s how I did it.
First, I boot with the install CD and typed “linux rescue” at the prompt. At the shell, I typed “lvm” and because running pvscan, vgscan, lvscan, etc.. to see what the problem was.. Well, it was reporting that the Logical Volume was damaged due to a missing drive.
I fixed that by doing this:
lvm>vgreduce VolGroup01 –removemissing
Ok.. so that fixed the Physical Volume and Volume Group.. but.. LogVol01 is still missing.. :/
So, after reading a lot.. I read that you can create the LogVol01 with the exact same parameters, and poof.. it should be back to the way it was.. Well, before I read about the lvcreate -Zy option, I created the LogVol01 back.. and zero\’d everything.. so the Logical Volume was there.. but was not listed as an EXT3 like it should be, and could not be mounted. Great..
lvm>lvcreate –size 110G /dev/VolGroup01/LogVol01
So, now I have an “empty” Logical Volume.. Now the part that took a few days to figure out.. and really should have been a no-brainer..
lvm>lvchange -vay /dev/VolGroup01/LogVol01
lvm>lvscan
ACTIVE \’/dev/VolGroup01/LogVol01\’ [110.66 GB] inherit
Should now show ACTIVE
Then, back at the shell
#e2fsck -fy /dev/VolGroup01/LogVol01
Now.. It could have complained about all the INode errors because I created the LV with the wrong paramters.. I\’m not sure.. But it went through and fixed them all.. After this, I was able to mount the volume and move my data to the new 250 GB drive..
#mount /dev/VolGroup01/LogVol01 /mnt/tmp
Yeah.. a happy ending!