[dm-crypt] Accesing dm-crypt volume after failed resizing with mdadm/RAID1, dm-crypt/LUKS, LVM
arno at wagner.name
Fri Aug 5 16:50:04 CEST 2011
One indirect iteration as I accidentally relayed direct instead
of to list. -- Arno
On Thu, Aug 04, 2011 at 05:03:21PM +0200, Paul Menzel wrote:
> 2011/8/4 Arno Wagner <arno at wagner.name>:
> > Sorry for this, but I think your set-up is overly complex
> > and basically a mess.
> It was a standard option from the Debian Installer at least some years
> ago and probably also today .
>  http://d-i.alioth.debian.org/manual/en.amd64/ch06s03.html#di-partition
Passed me by. But I partition myself before doing any
installation. So this is still a mess, but you are not
the one responsible.
> > It is quite possible that you made a small error somewhere
> > that brought the whole house of cards down.
> > I would say the only safe way to resize anything in this
> > setup is to do a complete data backup and recreate the setup
> > from scratch.
> Noted for the next time.
> > Some remarks:
> > - The old RAID superblocks are at the end of the
> > ??device, the new ones are at the start.
> > - LVM superblocks are at the start
> > - If your header backup is good, you can open a LUKS
> > ??device but the contents may be trash.
> I did not understand the last item. Do you mean because the LVM
> metadata might be lost?
> > I think what killed you was the RAID superblock. If I
> > remember correctly the old one is still the default.
> I found the exact commands I used tonight.
> mdadm --verbose --create /dev/md1 \
> --assume-clean \
> --level=1 \
> --raid-devices=2 \
> --uuid=52ff2cf2:40981859:e58d8dd6:5faec42c \
> /dev/sda2 missing
> This created a RAID1 with metadata 1.2 which seems to be the default
> in mdadm - v3.1.4 - 31st August 2010, at least in Grml. The command
According to this
Format 1.2 is at "4K from the beginning of the device"
Now that makes 3 alternatives. Seems this is a difficult
problem, but just keeping changing the lokation is probably
the worst solution.
> cryptsetup luksOpen /dev/md1 md1_crypt
> did not work though afterward not being able to find the md metadata.
> After stopping the RAID, `service mdadm-raid stop`,
> cryptsetup luksOpen /dev/sda2 sda2_crypt
> still worked. Only after this also adding `--metadata=0.90` too
> `/dev/md1` was detected as a LUKS partition/container but entering the
> passphrase failed. Metadata v1.2 seems to be the default according to
> the manual. So the Wiki page  is not up to date.
Hmm. Milan found some corruption in your header backup. Is
it possible that this is the version 1.2 RAID superblock 4k from
the beginning of the device?
> -e, --metadata=
> 1, 1.0, 1.1, 1.2 default
> Use the new version-1 format superblock. This
> has few restrictions. The different sub-versions store the
> superblock at different locations on the device,
> either at the end (for 1.0), at the start (for 1.1) or 4K from the
> start (for 1.2). "1" is equivalent to "1.0".
> "default" is equivalent to "1.2".
>  https://raid.wiki.kernel.org/index.php/RAID_superblock_formats#Total_Size_of_superblock
> > But as you worked on the unmapped device, you would
> > have killed the superblock regardless of where it was.
> > Don't believe what people on IRC tell you.
> You mean the md/RAID superblock, right?
> > Anyways, you need to get way more comforateble with
> > cryptsetup, lvm and mdadm for a meaningful diagnostic
> > of what happened. The the process would go something
> > like this:
> > 1. Get some sleep! After 10 hours of debugging you are bound
> > ?? to make it worse.
> Definitely true.
> > 2. Make a sector-wise copy of the whole drive. As your filesystem
> > ?? resize did work, chances are the data is intact.
> > ?? Work only on the copy to prevent further damage.
> `ddrescue` is running since some hours already to clone the drive.
Good. Most important secondary mistake not made.
> > 3. Identify where the LUKS header should be. If I understand
> > ?? correctly, it should be in the same place as on the
> > ?? 500G drive, as you did not move the start of md1.
> > ?? If you did move the LUKS header, find it.
> `cryptsetup luksHeaderBackup /dev/sda2 --header-backup-file h` works
> fine but something seems a miss.
Yes, milan found some obvious corryption as he posted.
> > 4. luksOpen the whole disk with an offset that matches the
> > ?? header offset. This should give you the rest of the disk
> > ?? in decrypted form (verify with hex-dump tool).
> > ?? Your enlarged filesystem should be in there somewhere.
> > ?? If this does not work, overwrite the header with the backup
> > ?? header and retry.
> > 5. Find the filesystem. There are some tools to find an ext2 superblock.
> > 6. You may have to remapp the decrypted device with LVM to
> > ?? get the start of the filesystem to the start of a partition.
> I have not gotten to the last three points yet. asalor on IRC could
> not spot anything fishy in the debug output of `luksOpen` .
>  http://www.saout.de/pipermail/dm-crypt/2011-August/001858.html
Ah, I am not talking about your restored LUKS header as that may
be in the wrong position, but the original LUKS header, that
just seems not to be aligned to the current partitions.
> > From then onwards you can read your enlarged filesystem.
> > Be careful and mount it ro. Make a backup of all data.
> > Wipe disk and recreate a simpler set-up from scratch ;-)
> Back to point three and understanding what happened.
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
If it's in the news, don't worry about it. The very definition of
"news" is "something that hardly ever happens." -- Bruce Schneier
More information about the dm-crypt