[dm-crypt] Anoter Raid Restoe

Arno Wagner arno at wagner.name
Sun May 6 19:33:30 CEST 2012


On Sun, May 06, 2012 at 10:55:23AM +0200, Daniel wrote:
> Hello i have following Problem. 
> Unfortunately through a power loss all superblocks of my RAID were
> destroyed.

That should not be possible. The superblocks are not 
written in normal operation, hence power-loss should
not write them either.

I suspect something else happened here.

> My setup
> 
> 8 x 1TB in a RAID 6
> 3 x 2TB in a RAID 5
> 
> The crucial data are on both of these arrays. 

You lost two arrays? That is even less likely with
power-loss as the cause.

Unix/Linux filesystems and raid arrays are reasonably 
resistant against power-loss, as they typically run 24/7
and in many set-ups power-loss is actually the "normal"
way to take filesystem and RAID down.

> After googling a while i recognized that several users encountered that
> problem.

Can you guive some examples? I have never heard array
loss because of power loss. People overwriting their 
superblocks (if at the start or 4k offset) and creating 
filesystems in their raw RAID components (killing the MD 
superblock depending on placement) are fairly common.

Side note: Unfortunately the current md folks have run
amock and there are three (!) different placement
optiond of the RAID superblock. The old format (0.90)
was at the end. The new format superblocks can be
at the end (1.0), start (1.1) or 4k from the start (1.2).
This is one thing that makes recovery tricky.

> Most of them were able to restore their arrays by creating the array
> again with excatly the same parameters mdadm --create ..... 

That is risky. If you do just one thing wrong, you may
kill other things, like a LUKS header, in the array.
You also need to specify all old values for akk the 
defaults that may have changed.

For example, if you recreate an 1.0 array with version 1.1
superblock, you kill a (potential) LUKS header.

> i tried that with the 3 x 2TB Array. Unfortunately after that
> cryptsetup luksOpen tels me "md5 is  no valid luks device". Then I had
> a look on the fist bytes of the array. Unfortunately it dose seem to be
> crap and no luks header. I unfortunately do not have a luks-header
> backup of this array. 
> I do have a backup of the luks header of the 8 x
> 1 TB array. Bevore i Try the same with the 8 x 1 TB Array i would like
> to make deadly sure there are no other options. 

Whatever you do, you can make backups before. For example,
backing up the first and last 10MB of each disk before messing
with RAID recivery, you should protect both residual superblock
data and potential LUKS headers.

In the RAID superblock dumps below, I do not find any
8 disk arrays, just 4 disk plus spaere and a dumping tool
that seems to be older than the RAID arrays, as it does
not display RAID level. I also see a mix if 0.90 and 1.2
superblocks. 

For the 1.2 superblock, the raid header is at 4k offset.
The data would start after that. You hexdumped the
area before.



That all said, I think this is not a cryptsetup problem.
Better ask some md-folks what to do. With backing up the
first and last 10MB of each disk, you should be able to
protect the contents from botched RAID repair and should
be able to go back to what you had before.


However, I see a larger problem here. First, power-loss is 
very, very unlikely to kill md-RAID headers even on one disk.
Then I see different header versions and the number of disks 
in the headers not matching your array description. Then I 
see the raid-level as -unknown- in some headers.


Care to explain the situation a bit more?


Arno


 


> Maby someone can Help me 
> Here some information that might be helpful:
> 
> Hexdump of the 3x 2TB Array
> 00000000  30 af 62 61 58 4f 88 1a  a4 d2 2f ed d6 c0 98 c3
>  |0.baXO..../.....|
> 00000010  1d 46 54 5f 12 38 d1 db  b5 56 77 42 fb 2d df db
>  |.FT_.8...VwB.-..|
> 00000020  88 6c 45 8d 2e 0c e8 39  62 88 2c b7 d1 c7 84 eb
>  |.lE....9b.,.....|
> 00000030  90 25 b6 3f 0d 4c 2b 9a  38 ee 27 5a 4a 6b 75 1a
>  |.%.?.L+.8.'ZJku.|
> 00000040  34 c3 58 fb 24 d1 b5 c8  ee ee ee 96 88 9a 41 c3
>  |4.X.$.........A.|
> 00000050  c8 ab c1 27 3f 78 5c 06  8b 1c cc 1b f1 95 5c ba
>  |...'?x\.......\.|
> 00000060  8c ce 11 e8 ba e0 8c 90  ca 5e 41 a0 d3 de 3c 4d
>  |.........^A...<M|
> 00000070  4c 5d 0d 4f cc ab 06 f1  89 d2 06 04 60 29 90 da
>  |L].O........`)..|
> 00000080  3d 38 50 e6 72 81 8b 18  89 92 8b 22 90 e4 e6 3c
>  |=8P.r......"...<|
> 00000090  dd 66 23 8f f1 08 94 38  34 20 98 3f a9 16 9f 22  |.f#....84
> .?..."|
> 000000a0  a8 93 10 03 8f 6b 33 47  4e 31 e7 e9 e5 c3 85 87
>  |.....k3GN1......|
> 000000b0  70 76 2b 68 e0 29 31 34  ee b2 e6 d9 5b be c7 96
>  |pv+h.)14....[...|
> 000000c0  9d 2b 33 c1 7f 96 0b 7f  f8 7e 05 0e 85 51 de ce
>  |.+3......~...Q..|
> 000000d0  e7 08 3e c4 c0 0e bc 43  05 74 41 ff e3 af e0 ab
>  |..>....C.tA.....|
> 000000e0  7d ce 5a 3d dd 88 d6 bb  b1 88 fa 85 bd 65 73 57
>  |}.Z=.........esW|
> 000000f0  f7 7c 8d 5e 37 9a 69 10  2e 30 84 a7 c6 21 72 2c
>  |.|.^7.i..0...!r,|
> 00000100  51 46 80 31 79 c2 0a ec  08 04 12 a7 a6 7b 4d 3e
>  |QF.1y........{M>|
> 00000110  b3 bc f7 db 20 f8 ad b6  58 c8 52 23 b9 3d 96 de  |....
> ...X.R#.=..|
> 00000120  6c 6d c2 2a 8a 5d ec 3c  7c 8b d5 4f ca b8 75 5f
>  |lm.*.].<|..O..u_|
> 00000130  ef b1 2c 55 cb 6a c5 f9  2f fa e9 18 fa 42 7f 32
>  |..,U.j../....B.2|
> 00000140  ed e7 4a 40 71 67 c3 30  b3 e8 a0 a4 4f 46 98 59
>  |..J at qg.0....OF.Y|
> 00000150  43 50 74 f1 b5 42 e9 bd  90 57 59 67 fc 77 e0 d6
>  |CPt..B...WYg.w..|
> 00000160  6e c5 40 56 9f 06 4d d0  e8 32 b3 2c 73 ec a1 db
>  |n. at V..M..2.,s...|
> 00000170  cd d7 80 31 01 04 b2 ac  4e 3e fa ee cf a2 6a df
>  |...1....N>....j.|
> 00000180  8f 74 e3 4f 8f 4a 08 8f  df 75 fe 0b 35 d9 73 32
>  |.t.O.J...u..5.s2|
> 00000190  37 e7 d2 71 95 b6 c4 43  f5 db cd 09 d1 ab 12 02
>  |7..q...C........|
> 000001a0  bd 8f 8e 2d b0 71 6a b8  ba cf d4 44 7c 0a 34 13
>  |...-.qj....D|.4.|
> 000001b0  ee 1f ec e1 c1 46 14 2c  66 94 a0 a5 7e a4 fe 0a
>  |.....F.,f...~...|
> 000001c0  d7 9e 0d 9c 40 a6 57 49  f4 a0 4b b0 79 87 0b 03
>  |.... at .WI..K.y...|
> 000001d0  29 34 cc 79 1d 42 60 93  bc d0 e2 8d a3 20 f1 28
>  |)4.y.B`...... .(|
> 000001e0  56 3d a8 a1 43 c4 fc b7  e6 8e 2c 07 f4 d3 73 01
>  |V=..C.....,...s.|
> 000001f0  c4 49 1a 08 10 ac 25 0e  31 91 d2 a7 3e 85 f6 be
>  |.I....%.1...>...|
> 00000200  cb e8 5f ec 1c 69 23 92  c3 84 43 aa 38 58 25 ef
>  |.._..i#...C.8X%.|
> 00000210  19 90 e9 86 7a ab 47 62  f9 7b 1a 2e 09 9f 70 f5
>  |....z.Gb.{....p.|
> 00000220  0a 67 9f d2 01 db ce 07  40 86 7c c4 7e cf 61 7f
>  |.g...... at .|.~.a.|
> 00000230  0e df a7 44 61 dc c5 56  f7 e1 14 8c 01 8e a8 0d
>  |...Da..V........|
> 00000240  eb 5b 35 70 45 16 60 be  85 ce 57 9a 09 b1 82 fd
>  |.[5pE.`...W.....|
> 00000250  9d 59 41 00 79 fd c0 3e  e5 18 18 4f 33 9c ee 22
>  |.YA.y..>...O3.."|
> 00000260  f4 2b ff 06 b2 fe 9c 0a  88 01 e6 a4 fd d5 35 01
>  |.+............5.|
> 00000270  3e dd 31 84 55 5d 71 9b  73 57 94 46 72 7d c9 c0
>  |>.1.U]q.sW.Fr}..|
> 
> mdadm -D /dev/md5
> 
> /dev/md5:
>         Version : 1.2
>   Creation Time : Sat May  5 01:30:36 2012
>      Raid Level : raid5
>      Array Size : 3907021568 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 1953510784 (1863.01 GiB 2000.40 GB)
>    Raid Devices : 3
>   Total Devices : 3
>     Persistence : Superblock is persistent
> 
>     Update Time : Sun May  6 07:16:52 2012
>           State : clean 
>  Active Devices : 3
> Working Devices : 3
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>            Name : stonith:5  (local to host stonith)
>            UUID : ea42de9b:87279279:b278cefd:4686698b
>          Events : 21
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       17        0      active sync   /dev/sdb1
>        1       8       49        1      active sync   /dev/sdd1
>        3       8       65        2      active sync   /dev/sde1
> 
> 
> 
> mdadm -E all devices 
> 
> /dev/sdb:
>    MBR Magic : aa55
> Partition[0] :   3907024002 sectors at           63 (type fd)
> /dev/sdb1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : ea42de9b:87279279:b278cefd:4686698b
>            Name : stonith:5  (local to host stonith)
>   Creation Time : Sat May  5 01:30:36 2012
>      Raid Level : raid5
>    Raid Devices : 3
> 
>  Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>      Array Size : 7814043136 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : 38cca227:a2b01df0:a4f71bd4:26fc69ab
> 
>     Update Time : Sun May  6 07:16:52 2012
>        Checksum : f843636c - correct
>          Events : 21
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>    Device Role : Active device 0
>    Array State : AAA ('A' == active, '.' == missing)
> 
> 
> 
> /dev/sdc:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type fd)
> /dev/sdc1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 22:21:42 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 5
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 22:24:06 2012
>           State : active
>  Active Devices : 0
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 5
>        Checksum : 829a0f47 - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     4       8       97        4      spare   /dev/sdg1
> 
>    0     0       8      177        0      spare   /dev/sdl1
>    1     1       8      161        1      spare   /dev/sdk1
>    2     2       8      145        2      spare
>    3     3       8      113        3      spare   /dev/sdh1
>    4     4       8       97        4      spare   /dev/sdg1
> /dev/sdd:
>    MBR Magic : aa55
> Partition[0] :   3907024002 sectors at           63 (type fd)
> /dev/sdd1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : ea42de9b:87279279:b278cefd:4686698b
>            Name : stonith:5  (local to host stonith)
>   Creation Time : Sat May  5 01:30:36 2012
>      Raid Level : raid5
>    Raid Devices : 3
> 
>  Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>      Array Size : 7814043136 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : ddc9eea2:7ecb67cf:ecd45f8f:8171667b
> 
>     Update Time : Sun May  6 07:16:52 2012
>        Checksum : de19ce91 - correct
>          Events : 21
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>    Device Role : Active device 1
>    Array State : AAA ('A' == active, '.' == missing)
> /dev/sde:
>    MBR Magic : aa55
> Partition[0] :   3907024002 sectors at           63 (type fd)
> /dev/sde1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : ea42de9b:87279279:b278cefd:4686698b
>            Name : stonith:5  (local to host stonith)
>   Creation Time : Sat May  5 01:30:36 2012
>      Raid Level : raid5
>    Raid Devices : 3
> 
>  Avail Dev Size : 3907021954 (1863.01 GiB 2000.40 GB)
>      Array Size : 7814043136 (3726.03 GiB 4000.79 GB)
>   Used Dev Size : 3907021568 (1863.01 GiB 2000.40 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : 0e9d2be3:ab0c2188:e353ac39:f62f8178
> 
>     Update Time : Sun May  6 07:16:52 2012
>        Checksum : 7e77205d - correct
>          Events : 21
> 
>          Layout : left-symmetric
>      Chunk Size : 128K
> 
>    Device Role : Active device 2
>    Array State : AAA ('A' == active, '.' == missing)
> /dev/sdf:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type fd)
> /dev/sdf1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 22:21:42 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 5
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 22:24:06 2012
>           State : active
>  Active Devices : 0
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 5
>        Checksum : 829a0f73 - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     2       8      145        2      spare
> 
>    0     0       8      177        0      spare   /dev/sdl1
>    1     1       8      161        1      spare   /dev/sdk1
>    2     2       8      145        2      spare
>    3     3       8      113        3      spare   /dev/sdh1
>    4     4       8       97        4      spare   /dev/sdg1
> /dev/sdg:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type fd)
> /dev/sdg1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 22:21:42 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 5
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 22:24:06 2012
>           State : active
>  Active Devices : 0
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 5
>        Checksum : 829a0f81 - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     1       8      161        1      spare   /dev/sdk1
> 
>    0     0       8      177        0      spare   /dev/sdl1
>    1     1       8      161        1      spare   /dev/sdk1
>    2     2       8      145        2      spare
>    3     3       8      113        3      spare   /dev/sdh1
>    4     4       8       97        4      spare   /dev/sdg1
> /dev/sdh:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type 83)
> /dev/sdh1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 22:21:42 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 5
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 22:24:06 2012
>           State : active
>  Active Devices : 0
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 5
>        Checksum : 829a0f55 - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     3       8      113        3      spare   /dev/sdh1
> 
>    0     0       8      177        0      spare   /dev/sdl1
>    1     1       8      161        1      spare   /dev/sdk1
>    2     2       8      145        2      spare
>    3     3       8      113        3      spare   /dev/sdh1
>    4     4       8       97        4      spare   /dev/sdg1
> /dev/sdi:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type fd)
> /dev/sdi1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 22:21:42 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 5
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 22:24:06 2012
>           State : active
>  Active Devices : 0
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 5
>        Checksum : 829a0f8f - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     0       8      177        0      spare   /dev/sdl1
> 
>    0     0       8      177        0      spare   /dev/sdl1
>    1     1       8      161        1      spare   /dev/sdk1
>    2     2       8      145        2      spare
>    3     3       8      113        3      spare   /dev/sdh1
>    4     4       8       97        4      spare   /dev/sdg1
> /dev/sdk:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type 83)
> /dev/sdk1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 20:50:05 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 1
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 20:51:28 2012
>           State : active
>  Active Devices : 0
> Working Devices : 1
>  Failed Devices : 0
>   Spare Devices : 1
>        Checksum : 8299e0fc - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     0       8       33        0      spare   /dev/sdc1
> 
>    0     0       8       33        0      spare   /dev/sdc1
> /dev/sdl:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type 83)
> /dev/sdl1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 20:09:44 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 2
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 20:10:29 2012
>           State : active
>  Active Devices : 0
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 2
>        Checksum : 8299cdfc - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     1       8       17        1      spare   /dev/sdb1
> 
>    0     0       8       33        0      spare   /dev/sdc1
>    1     1       8       17        1      spare   /dev/sdb1
> /dev/sdm:
>    MBR Magic : aa55
> Partition[0] :   1953520002 sectors at           63 (type fd)
> /dev/sdm1:
>           Magic : a92b4efc
>         Version : 0.90.00
>            UUID : 00000000:00000000:00000000:00000000
>   Creation Time : Thu May  3 20:09:44 2012
>      Raid Level : -unknown-
>    Raid Devices : 0
>   Total Devices : 2
> Preferred Minor : 0
> 
>     Update Time : Thu May  3 20:10:29 2012
>           State : active
>  Active Devices : 0
> Working Devices : 2
>  Failed Devices : 0
>   Spare Devices : 2
>        Checksum : 8299ce0a - correct
>          Events : 1
> 
> 
>       Number   Major   Minor   RaidDevice State
> this     0       8       33        0      spare   /dev/sdc1
> 
>    0     0       8       33        0      spare   /dev/sdc1
>    1     1       8       17        1      spare   /dev/sdb1
> 
> 
> ----------------------------------------------
> Das ist ein Service von austria.com
> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 


More information about the dm-crypt mailing list