[dm-crypt] dm-crypt + mdadm

Arno Wagner arno at wagner.name
Sat Nov 23 12:58:03 CET 2013

On Sat, Nov 23, 2013 at 09:10:08 CET, Milan Broz wrote:
> On 22.11.2013 23:53, Arno Wagner wrote:
> >My bottom line is that
> >a) I do not trust the CentOS "stable" kernel one bit.
> >b) 2.6.32 is so old, doing anything on it to improve
> >    performance is probably a waste of time. 2.6.32 is 4 years
> >    old by now and the 2.6.x line is 10 years old...
> Sorry, but this is way too far generalization and typical mistake
> with "enterprise" kernels :)

I see you feel strongly about this ;-)

But I don't think so. Maybe I was not clear enough.
Certainly I did not give the details I base that 
statement on.
> 2.6.32 is baseline for REHL6/CentOS 6 kernels. But many patches
> are backported there and it get a lot of more testing.
> (And CentOS should be just rebuild of RHEL kernel.)
> Yes, they can mess things up sometimes, that's why there are stable
> kernels (no new features) and paid support for RHEL.

I had 3 immediate issues with older, reliable hardware that 
never gave me any trouble with any other kernels:

1. Kernel panic due to missing radeon firmware image.
   Newer kernels do not panic on this. Older ones do not 
   even detect the integrated graphis as anything special, 
   just as VGA, and hence do not panic either. 
   I finally had to switch off the inegrated graphics to get 
   it to boot at all.

2. A perfectly fine RTL8139 card was not even recognized.

3. A perfectly fine RTL8168 card caused the driver to spam 
   the syslog and text-console with warnings and made it unusable.
   (No X11 on this installation....)

Sorry, but that is not what I would call a stable kernel. Feels
more like something hacked together...

The only possible explantion I have is that this very standard 
hardware is not "enterpriese" enough (being cheap) and did not 
make it on the hardware compatibility list and hence does not
get tested well or at all. I did not check that list and usually 
I do not use RHEL/CentOS. This was a customer installation that 
I evaluated, and I was shocked to run into 3 serious issues with 
the kernel right away. 

> In many areas kernel number means nothing, it contains
> features from 3.12 etc (specially for device-mapper it is
> almostalways true).
> (And my experience is that RHEL/CentOS kernel are pretty stable
> in comparison with manual upstream builds. I would not
> suggest any customer to build own kernel for these enterprise 
> distributions

The suggestion was for tests, not to create a stable installation
this way. If there is still a performance-hit with a current 
kernel, then it is likely not a kernel issue. If not, then
it is and moving away from CentOS may be a viable solution...

> - note you need to update and maintain userspace bit as well,
> handle security updates and you have to know some internal differences.)

True. For example with newer kernels I had issues that some
include files were moved around and the include-path for
Debian stable gcc is not sufficient anymore. Fixing that
was a bit tricky.
> But in the RHEL6 dmcrypt case: yes, there is single threaded implementation
> (every dmcrypt device has own single thread). This is kind of exception,
> because usually RHEL device-mapper stack follows upstream.

That was the part why I called it "too old". Not in general, not
driver-wise, and not stability-wise, but with a really old kernel
some things cannot reasonably be back-ported and bottlenecks that
require architecural fixes may be impossible. Hence my statement
that it does not make sense to do performance optimization on an
old kernel architecture.

> This is no longer true for upstream where it uses per-cpu threads
> (limitation is that it always processes work on cpu which submitted it,
> so there are still some issues).
> Backporting this to RHEL6 is near to impossible (not only technical
> reasons, think certification etc) but I cannot speak for Red Hat anymore.
> (Just you can be sure you are not the first requesting it and I spent
> quite a lot time playing with it.)

I did not request this, I know it does not really make sense ;-)
> So complaining to RHEL/CentOS kernel here makes no sense, please
> fill bugzilla or use their support channel.

Not a complaint, just a comment.  

> For upstream, please test current kernel. You should get better
> performance (depends on scenario).
> (Mikulas posted several times another approach to dmcrypt parallelization
> bus I am still not convinced it really helps.  And currently there are
> some changes upstream in MD and block devices subsystem which influences
> it as well).
> Anyway, general suggestion now is to use fast CPU with AES-NI switched on.
> (This can saturate even very fast RAID arrays with single thread,
> IIRC I saw throughput >500MB for recent server hw & RHEL6 kernel).

Yes, I would think so. And that way you prevent possible 
incompatibilities between kernel and the rest of the distribution.
And if you pay for RHEL, that config gets you support, while a
custom kernel (understandably) gets you a shrug.

Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno at wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
There are two ways of constructing a software design: One way is to make it
so simple that there are obviously no deficiencies, and the other way is to
make it so complicated that there are no obvious deficiencies. The first
method is far more difficult.  --Tony Hoare

More information about the dm-crypt mailing list