[dm-crypt] "not a valid LUKS device" after distro change

Arno Wagner arno at wagner.name
Fri Aug 22 00:54:26 CEST 2014


This looks to me like some massive issues with user-space tools,
starting with the md-devices, that in Fedora are md0 and md2,
but in Ubuntu are md126 and md127. 

This likely means they are of the completely braindead "new"
RAID-superblock formats where the RAID assembly is not done
by the RAID controller (i.e. the kernel) as it should be, but
by something in userspace. Change the userspace, lose proper
RAID assembly. The idiots responsible for that have not 
understood what layering is and why you do it and have 
implemented a massive step backwards. 

The way around this is to use the old superblock format 0.90 
with partition type 0xfd. With that and any kernel doing RAID
auto-assembly, you always get the same evices. Of course, some
distros are utterly brain-dead and re-map the 0.90 devices to 
new names via user-space re-assembly, which are then persistent. 
Had that with one "recovery CD once", that I needed an hour to 
recover from booting.

Yes, the 1.x formats have better capabilities, but what use
is that when they are fundamentally broken otherwise? "Better
capabilities" is not a valid reason to break things that worked
before.

My advice around this whole mess would be to make a simple, 
flat patitioning and RAID schema yourself manually (superblock
0.90, type 0xfd, no LVM, no partitioning in RAID, just make
more RAID arrays) and install into that. That seems the only
way to get KISS with regard to RAID and partitioning these
days and KISS is the only thing that gives you reliability
and ease of debugging.

I think I will add a warning to the FAQ that changing distros
can make your LUKS containers hard or impossible to access due
to LVM and new RAID superblock userspace differences and that
it may mess things up enough that a move back to the old distro
may not be sufficient to fix things. How these jokers expect
people to deal with multiple distros on the same machine is
beyond me. 

Arno






On Thu, Aug 21, 2014 at 22:46:04 CEST, John Wells wrote:
> Attached below, I've included the output of running these commands from
> first Ubuntu and then Fedora. Note, apparently they map the raid device
> numbers differently. Also, you'll note there's a cryptd mounted in Ubuntu.
> That's the one i can successfully mount from Ubuntu.
> 
> To make things even stranger, I was ***able to luksOpen and then mount
> /dev/MORE_VG/MORE_LV this time while booted in the Fedora 20*** live cd,
> while as you recall I was unable to before. I'm also including the results
> of dumping the head of the dev file from Fedora.
> 
> I'm completely at a loss. At this point I feel I should probably backup the
> entire machine and re-install from scratch, but I'll hold off a bit longer.
> The inconsistent behavior is really disconcerting.
> 
> Thanks,
> John
> 
> ####################### FIRST FROM UBUNTU ########################
> # lsblk
> NAME                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> sda                                     8:0    0  37.3G  0 disk
> ├─sda1                                  8:1    0   100M  0 part
> └─sda2                                  8:2    0  37.2G  0 part
> sdb                                     8:16   0   1.8T  0 disk
> └─sdb1                                  8:17   0   1.8T  0 part
>   └─md0                                 9:0    0   1.8T  0 raid1
>     └─MORE_VG-MORE_LV (dm-1)          252:1    0   800G  0 lvm
>       └─cryptd (dm-7)                 252:7    0   800G  0 crypt /home
> sdc                                     8:32   0   1.8T  0 disk
> └─sdc1                                  8:33   0   1.8T  0 part
>   └─md2                                 9:2    0   1.8T  0 raid1
>     ├─FINALFRONTIER_VG-HOME_LV (dm-2) 252:2    0 698.5G  0 lvm
>     ├─FINALFRONTIER_VG-SWAP_LV (dm-3) 252:3    0  22.4G  0 lvm   [SWAP]
>     ├─FINALFRONTIER_VG-OPT_LV (dm-4)  252:4    0 139.7G  0 lvm   /opt
>     ├─FINALFRONTIER_VG-VMS_LV (dm-5)  252:5    0 102.5G  0 lvm   /vms
>     └─FINALFRONTIER_VG-DATA_LV (dm-6) 252:6    0 279.4G  0 lvm   /data
> sdd                                     8:48   0   1.8T  0 disk
> └─sdd1                                  8:49   0   1.8T  0 part
>   └─md2                                 9:2    0   1.8T  0 raid1
>     ├─FINALFRONTIER_VG-HOME_LV (dm-2) 252:2    0 698.5G  0 lvm
>     ├─FINALFRONTIER_VG-SWAP_LV (dm-3) 252:3    0  22.4G  0 lvm   [SWAP]
>     ├─FINALFRONTIER_VG-OPT_LV (dm-4)  252:4    0 139.7G  0 lvm   /opt
>     ├─FINALFRONTIER_VG-VMS_LV (dm-5)  252:5    0 102.5G  0 lvm   /vms
>     └─FINALFRONTIER_VG-DATA_LV (dm-6) 252:6    0 279.4G  0 lvm   /data
> sde                                     8:64   0 167.7G  0 disk
> ├─sde1                                  8:65   0     3G  0 part  /boot
> ├─sde2                                  8:66   0    12G  0 part
> └─sde3                                  8:67   0 152.8G  0 part
>   └─SSDROOT_VG-ROOT_LV (dm-0)         252:0    0  97.7G  0 lvm   /
> sdf                                     8:80   0   1.8T  0 disk
> └─sdf1                                  8:81   0   1.8T  0 part
> sdg                                     8:96   0   1.8T  0 disk
> └─sdg1                                  8:97   0   1.8T  0 part
>   └─BACKUPVG-BACKUPLV (dm-8)          252:8    0   1.2T  0 lvm   /backup
> sr0                                    11:0    1  1024M  0 rom
> # dmsetup ls
> FINALFRONTIER_VG-VMS_LV (252:5)
> BACKUPVG-BACKUPLV (252:8)
> MORE_VG-MORE_LV (252:1)
> FINALFRONTIER_VG-OPT_LV (252:4)
> FINALFRONTIER_VG-SWAP_LV (252:3)
> SSDROOT_VG-ROOT_LV (252:0)
> cryptd (252:7)
> FINALFRONTIER_VG-DATA_LV (252:6)
> FINALFRONTIER_VG-HOME_LV (252:2)
> # dmsetup deps
> FINALFRONTIER_VG-VMS_LV: 1 dependencies : (9, 2)
> BACKUPVG-BACKUPLV: 1 dependencies : (8, 97)
> MORE_VG-MORE_LV: 1 dependencies : (9, 0)
> FINALFRONTIER_VG-OPT_LV: 1 dependencies : (9, 2)
> FINALFRONTIER_VG-SWAP_LV: 1 dependencies : (9, 2)
> SSDROOT_VG-ROOT_LV: 1 dependencies : (8, 67)
> cryptd: 1 dependencies : (252, 1)
> FINALFRONTIER_VG-DATA_LV: 1 dependencies : (9, 2)
> FINALFRONTIER_VG-HOME_LV: 1 dependencies : (9, 2)
> 
> ########################## THEN FROM FEDORA #########################
> 
> # lsblk
> NAME                           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
> sda                              8:0    0  37.3G  0 disk
> ├─sda1                           8:1    0   100M  0 part
> └─sda2                           8:2    0  37.2G  0 part
> sdb                              8:16   1  14.6G  0 disk
> ├─sdb1                           8:17   1   953M  0 part
>  /run/initramfs/live
> ├─sdb2                           8:18   1   4.9M  0 part
> └─sdb3                           8:19   1  19.7M  0 part
> sdc                              8:32   0   1.8T  0 disk
> └─sdc1                           8:33   0   1.8T  0 part
>   └─md126                        9:126  0   1.8T  0 raid1
>     └─MORE_VG-MORE_LV          253:4    0   800G  0 lvm
> sdd                              8:48   0   1.8T  0 disk
> └─sdd1                           8:49   0   1.8T  0 part
>   └─md127                        9:127  0   1.8T  0 raid1
>     ├─FINALFRONTIER_VG-HOME_LV 253:5    0 698.5G  0 lvm
>     ├─FINALFRONTIER_VG-SWAP_LV 253:6    0  22.4G  0 lvm
>     ├─FINALFRONTIER_VG-OPT_LV  253:7    0 139.7G  0 lvm   /opt
>     ├─FINALFRONTIER_VG-VMS_LV  253:8    0 102.5G  0 lvm
>     └─FINALFRONTIER_VG-DATA_LV 253:9    0 279.4G  0 lvm
> sde                              8:64   0   1.8T  0 disk
> └─sde1                           8:65   0   1.8T  0 part
>   └─md127                        9:127  0   1.8T  0 raid1
>     ├─FINALFRONTIER_VG-HOME_LV 253:5    0 698.5G  0 lvm
>     ├─FINALFRONTIER_VG-SWAP_LV 253:6    0  22.4G  0 lvm
>     ├─FINALFRONTIER_VG-OPT_LV  253:7    0 139.7G  0 lvm   /opt
>     ├─FINALFRONTIER_VG-VMS_LV  253:8    0 102.5G  0 lvm
>     └─FINALFRONTIER_VG-DATA_LV 253:9    0 279.4G  0 lvm
> sdf                              8:80   0 167.7G  0 disk
> ├─sdf1                           8:81   0     3G  0 part
> ├─sdf2                           8:82   0    12G  0 part
> └─sdf3                           8:83   0 152.8G  0 part
>   └─SSDROOT_VG-ROOT_LV         253:3    0  97.7G  0 lvm
> sdg                              8:96   0   1.8T  0 disk
> └─sdg1                           8:97   0   1.8T  0 part
> sdh                              8:112  0   1.8T  0 disk
> └─sdh1                           8:113  0   1.8T  0 part
>   └─BACKUPVG-BACKUPLV          253:10   0   1.2T  0 lvm   /backup
> sr0                             11:0    1  1024M  0 rom
> loop0                            7:0    0     8K  1 loop
> loop1                            7:1    0   1.2M  1 loop
> └─live-osimg-min               253:2    0     4G  1 dm
> loop2                            7:2    0 886.8M  1 loop
> loop3                            7:3    0     4G  1 loop
> ├─live-rw                      253:0    0     4G  0 dm    /
> ├─live-base                    253:1    0     4G  1 dm
> └─live-osimg-min               253:2    0     4G  1 dm
> loop4                            7:4    0   512M  0 loop
> └─live-rw                      253:0    0     4G  0 dm    /
> # dmsetup ls
> live-base (253:1)
> FINALFRONTIER_VG-VMS_LV (253:8)
> BACKUPVG-BACKUPLV (253:10)
> MORE_VG-MORE_LV (253:4)
> FINALFRONTIER_VG-OPT_LV (253:7)
> live-osimg-min (253:2)
> FINALFRONTIER_VG-SWAP_LV (253:6)
> live-rw (253:0)
> SSDROOT_VG-ROOT_LV (253:3)
> FINALFRONTIER_VG-DATA_LV (253:9)
> FINALFRONTIER_VG-HOME_LV (253:5)
> # dmsetup deps
> live-base: 1 dependencies : (7, 3)
> FINALFRONTIER_VG-VMS_LV: 1 dependencies : (9, 127)
> BACKUPVG-BACKUPLV: 1 dependencies : (8, 113)
> MORE_VG-MORE_LV: 1 dependencies : (9, 126)
> FINALFRONTIER_VG-OPT_LV: 1 dependencies : (9, 127)
> live-osimg-min: 2 dependencies : (7, 1) (7, 3)
> FINALFRONTIER_VG-SWAP_LV: 1 dependencies : (9, 127)
> live-rw: 2 dependencies : (7, 4) (7, 3)
> SSDROOT_VG-ROOT_LV: 1 dependencies : (8, 83)
> FINALFRONTIER_VG-DATA_LV: 1 dependencies : (9, 127)
> FINALFRONTIER_VG-HOME_LV: 1 dependencies : (9, 127)
> 
> ###################### AND NOW THE dumps of both devs from FEDORA
> ############################
> # head -c 1024 /dev/FINALFRONTIER_VG/HOME_LV | hexdump -C
> 00000000  47 4e 55 20 50 61 72 74  65 64 20 4c 6f 6f 70 62  |GNU Parted
> Loopb|
> 00000010  61 63 6b 20 30 00 00 00  00 00 00 00 00 00 00 00  |ack
> 0...........|
> 00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> *
> 00000400
> 
> 
> 
> # head -c 1024 /dev/MORE_VG/MORE_LV | hexdump -C
> 00000000  4c 55 4b 53 ba be 00 01  61 65 73 00 00 00 00 00
>  |LUKS....aes.....|
> 00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000020  00 00 00 00 00 00 00 00  78 74 73 2d 70 6c 61 69
>  |........xts-plai|
> 00000030  6e 36 34 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |n64.............|
> 00000040  00 00 00 00 00 00 00 00  73 68 61 31 00 00 00 00
>  |........sha1....|
> 00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000060  00 00 00 00 00 00 00 00  00 00 10 00 00 00 00 40
>  |...............@|
> 00000070  4e 94 98 c0 51 a9 25 bb  74 73 88 a4 a7 6e c7 d3
>  |N...Q.%.ts...n..|
> 00000080  6f 18 71 fa 94 9a f4 5e  d1 e8 5b 1a 34 3a 9f 9f
>  |o.q....^..[.4:..|
> 00000090  1d 79 1f 61 f5 dd 98 09  e1 d6 2e ed c4 29 af 1a
>  |.y.a.........)..|
> 000000a0  23 c9 59 da 00 00 77 a1  36 63 63 31 38 38 64 62
>  |#.Y...w.6cc188db|
> 000000b0  2d 63 63 64 62 2d 34 63  38 66 2d 39 37 62 32 2d
>  |-ccdb-4c8f-97b2-|
> 000000c0  65 34 31 31 39 38 65 63  36 65 34 34 00 00 00 00
>  |e41198ec6e44....|
> 000000d0  00 ac 71 f3 00 01 df d7  79 85 dd f0 29 59 98 63
>  |..q.....y...)Y.c|
> 000000e0  0b 69 80 fe 48 61 8c 40  5b 3b 57 0f 82 9c ae 90  |.i..Ha.@
> [;W.....|
> 000000f0  36 57 45 e2 03 82 26 c5  00 00 00 08 00 00 0f a0
>  |6WE...&.........|
> 00000100  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000120  00 00 00 00 00 00 00 00  00 00 02 00 00 00 0f a0
>  |................|
> 00000130  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000150  00 00 00 00 00 00 00 00  00 00 03 f8 00 00 0f a0
>  |................|
> 00000160  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000180  00 00 00 00 00 00 00 00  00 00 05 f0 00 00 0f a0
>  |................|
> 00000190  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 000001b0  00 00 00 00 00 00 00 00  00 00 07 e8 00 00 0f a0
>  |................|
> 000001c0  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 000001e0  00 00 00 00 00 00 00 00  00 00 09 e0 00 00 0f a0
>  |................|
> 000001f0  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000210  00 00 00 00 00 00 00 00  00 00 0b d8 00 00 0f a0
>  |................|
> 00000220  00 00 de ad 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000230  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 00000240  00 00 00 00 00 00 00 00  00 00 0d d0 00 00 0f a0
>  |................|
> 00000250  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
>  |................|
> 
> 
> 
> 
> On Thu, Aug 21, 2014 at 12:13 PM, Jonas Meurer <jonas at freesources.org>
> wrote:
> 
> > Hello John,
> >
> > Am 21.08.2014 um 17:55 schrieb John Wells:
> > > Thanks. How would I use this information to track back to a potential
> > > problem?
> >
> > I'm not convinced yet, that your system screwed up partition and/or
> > device ordering. First you should try to find out which devices are
> > recognized both in Fedora and Ubuntu and compare these.
> >
> > Try to run 'lsblk', 'dmsetup ls' and 'dmsetup deps' on both systems and
> > compare or post the result here.
> >
> > Which Ubuntu version do you run? Do you use an installed Ubuntu or
> > rather a live image? It would help if you could provide some more
> > information regarding your system.
> >
> > Kind regards,
> >  jonas
> >
> > >
> > > On Aug 21, 2014 10:41 AM, "Robert Nichols" <rnicholsNOSPAM at comcast.net
> > > <mailto:rnicholsNOSPAM at comcast.net>> wrote:
> > >
> > >     On 08/21/2014 09:00 AM, John Wells wrote:
> > >
> > >         I will try what you say. To add to the weirdness, I came into
> > >         work this morning
> > >         with a unresponsive machine. I hard reset, booted Ubuntu, but at
> > >         that point
> > >         *Ubuntu wouldn't recognize either partition, and both had the
> > >         same "GNU Parted
> > >         Loopback 0" in the output of "head -c 1024 /volume | hd".
> > >         Neither partition was
> > >         recognized by luksDump. I panicked.
> > >
> > >         Rebooted, and guess what? /dev/MORE_VG/MORE_LV was back to
> > >         normal and I could
> > >         mount it. /dev/FINALFRONTIER_VG/HOME_LV was still corrupted with
> > >         the "GNU Parted
> > >         Loopback 0" output.
> > >
> > >         This makes no sense to me. How could the leading bits be
> > >         different each time I
> > >         booted up? Could datamapper be assigning the wrong device to the
> > >         logical volume
> > >         in some way? It just makes no sense.
> > >
> > >
> > >     You can run (as root) "dmsetup deps" on each device in /dev/mapper
> > >     to see
> > >     the major and minor device numbers needed for each entry.
> > >
> > >     --
> > >     Bob Nichols     "NOSPAM" is really part of my email address.
> > >                     Do NOT delete it.
> > >
> > >     _________________________________________________
> > >     dm-crypt mailing list
> > >     dm-crypt at saout.de <mailto:dm-crypt at saout.de>
> > >     http://www.saout.de/mailman/__listinfo/dm-crypt
> > >     <http://www.saout.de/mailman/listinfo/dm-crypt>
> > >
> > >
> > >
> > > _______________________________________________
> > > dm-crypt mailing list
> > > dm-crypt at saout.de
> > > http://www.saout.de/mailman/listinfo/dm-crypt
> > >
> >
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt at saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> >

> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno at wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato


More information about the dm-crypt mailing list