[dm-crypt] LUKS in failover cluster

Sohl, Jacob (LNG-SEA) jacob.sohl at applieddiscovery.com
Wed Oct 12 22:26:08 CEST 2011


Yeah, it was a "crazy-fun-project" laugh. An opportunity to learn a lot and create something cool.

> -----Original Message-----
> From: Scott McCarty [mailto:smccarty at redhat.com]
> Sent: Wednesday, October 12, 2011 12:55 PM
> To: Sohl, Jacob (LNG-SEA)
> Cc: LNG-SEA Systems Engineers ADI; Arno Wagner; dm-crypt at saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> That was a good laugh, maybe a chuckle even :-)
> 
> Scott McCarty, RHCE, RHCSA, LPIC-1
> Solutions Architect
> Email: smccarty at redhat.com
> Phone: 312-660-3535
> Web: http://people.redhat.com/smccarty
> 
> 
> ----- Original Message -----
> From: "Jacob Sohl (LNG-SEA)" <jacob.sohl at applieddiscovery.com>
> To: "Arno Wagner" <arno at wagner.name>, dm-crypt at saout.de
> Cc: "LNG-SEA Systems Engineers ADI" <LNG-
> SEASystemsEngineersADI at lexisnexis.com>, "Scott McCarty"
> <smccarty at redhat.com>
> Sent: Wednesday, October 12, 2011 3:20:51 PM
> Subject: RE: [dm-crypt] LUKS in failover cluster
> 
> 
> 
> > -----Original Message-----
> > From: dm-crypt-bounces at saout.de [mailto:dm-crypt-bounces at saout.de] On
> > Behalf Of Arno Wagner
> > Sent: Tuesday, October 11, 2011 8:13 PM
> > To: dm-crypt at saout.de
> > Subject: Re: [dm-crypt] LUKS in failover cluster
> >
> > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA)
> wrote:
> > [...]
> > > > You can map ("decrypt") the devices and never use them. The
> > > > LVM/mount/whatever is completely optional.
> > > >
> > > > In fact the mapper tool (cryptsetup) does nothing except
> > > > to decrypt the raw encrypted device to a raw decrypted device
> > > > under /dev/mapper/.
> > > >
> > >
> > > Ok, thank you. That seemed logical, but I just recently started
> > working
> > > with encryption so I'm still learning.
> >
> > That is fine. Making sure now will safe you grief later.
> >
> > > >
> > > > > Also, scalability is a requirement in my design, hence XFS. I
> was
> > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > > > > filesystem. But I would end up with multiple LUKS devices to
> keep
> > > > track
> > > > > of. I recently found out that LUKS can resize.
> > > >
> > > > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > > > device size. So if you enlarge a device/partition with an intact
> > > > LUKS header, "cryptsetup luksOpen" will just map the larger
> > > > device.
> > > >
> > > > If you have multiple devices, you can "slave" the additional ones
> > > > to a "master" device using something like "decrypt_derived".
> > > > This just takes the master key from the opened "master"
> container,
> > > > runns it though a hash and uses this as key for the "slave"
> > > > device.
> > > >
> > > > > Would it be better to
> > > > > create one LUKS device on top of LVM? Then create a filesystem
> on
> > > > that?
> > > > > (Though that would affect resource dependencies.)
> > > >
> > > > Sre you sure you need LVM at all?
> > >
> > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN
> admins
> > > can/will only attach storage in 4TB increments. So I have to
> combine
> > all
> > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS
> > > filesystem. That's why I ask about encrypting multiple physical
> LUNS
> > vs
> > > a single Logical Volume.
> >
> > 50TB? Ok, that is a bit larger. I did have such a dataset
> > for research a few years back on tape. Took several days
> > days to get 15TB of it to disk for my largest measurement.
> >
> > Well, yes, I think LVM is the best way here.
> >
> > > >
> > > > > But basically:
> > > > >
> > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server
> > > > >
> > > > >
> > > > >
> > > > > Other people will be accessing/managing this system, so I want
> > > > > manageability through simplicity.
> > > >
> > > > Hence the question whether you actually need LVM.
> > > > It strikes me that typically LVM is not needed and
> > > > just complicates matters.
> > > >
> > > > > Don't want to have the wrong volumes
> > > > > (re)encrypted, headers damaged, etc.
> > > >
> > > > I think this setup is already too complicated for most people
> > > > to manage. A full backup should be part of your plans.
> > > > Resize with backup is actually easier, as you can just
> > > > backup, kill everything and do a clean new config.
> > > >
> > >
> > > I will be making full backups with xfsdump. But it doesn't seem
> > > practical to be moving 50TB around. Both LVM and XFS allow for
> > designed
> > > for online resize. I was planning to "pvcreate" an encrypted LUN
> then
> > > expand the Logical Volume and Filesystem.
> >
> > For that volume, run a full compare as well. I found that
> > with good hardware, and an IBM Tape-Library I still had
> > something like 1 unexplained bit-error every 10TB or so
> > when pulling the data from the tape-library again. That was
> > a few years back, but still.
> >
> 
> Thanks, I've never been very good at doing backups so any advice is
> appreciated.
> 
> > > > > Anyways, thanks for your help.
> > > > >
> > > >
> > > > Just to make sure: You _have_ read the FAQ?
> > >
> > > Yes I have read the FAQ.
> >
> > Good!
> >
> > > I guess I am just a little nervous about this project.
> >
> > Understandable.
> >
> > > This is my first time working with encryption and 50TB of
> > > critical client data is at stake. Just being thorough and making
> sure
> > I
> > > fully understand everything.
> >
> > And that is all you can do. I think you are approaching this
> > perfectly right. I take it the 50TB are alredy on XFS+LVM
> > at this time and you are just adding the encryption layer?
> 
> Currently the data is spread across multiple ext3 volumes of 8TB or
> less
> and is a pain to manage. That is the reason we are moving to XFS. The
> encryption is currently being handled by a Brocade switch blade. The
> encryption blade caused us major issues last year, resulting in a
> multi-day outage. That is the why we are moving to LUKS.
> 
> >
> > Make sure to use XTS mode or plain64 with ESSIV for these
> > volume sizes.
> 
> I don't know what this means. Haha. But I will look it up.
> 
> >
> > And let us know what your expeiences are!
> >
> 
> Sure thing. This project has been my first exposure to XFS, LUKS and
> clustering. And I'm the main Linux guy on my team. I've been working
> with Red Hat, but the solution architect laughed when he saw what we
> were trying to do. Haha. But he's been doing research and talking to
> his
> Red Hat contacts. And you guys have helped out tremendously! Thank you!
> 
> Jacob
> 
> > Arno
> > --
> > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> > arno at wagner.name
> > GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50
> 1E25
> > 338F
> > ----
> > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> >
> > If it's in the news, don't worry about it.  The very definition of
> > "news" is "something that hardly ever happens." -- Bruce Schneier
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt at saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt


More information about the dm-crypt mailing list