[dm-crypt] poor mysqldump performance

Arno Wagner arno at wagner.name
Fri Feb 24 21:05:29 CET 2012

On Fri, Feb 24, 2012 at 11:07:57AM -0800, Tracy Reed wrote:
> On Fri, Feb 24, 2012 at 09:18:58AM +0100, Milan Broz spake thusly:
> > RHEL5 uses old dmcrypt code which is stable but has known
> > limitations.
> Aside from being single-threaded (not the problem in our case) what are
> the limitations?  My extensive googling hasn't turned up anything relevant
> short of reading large amounts of the dm-crypt list archive.
> > The last change in RHEL was backporting suport for XTS mode.
> I can't find any good info on exactly what this is but I wonder if it is
> related to block size since. 

It is a block cipher mode. Shpuld not have any relation to your problem.

> One thing I have been wondering about is block size and CBC. mysqldump is
> probably doing a lot of tiny reads. Just how much data does dm-crypt have
> to read to pull a single piece of data from the disk?  Could the use of
> cipher block chaining be causing it to read a lot more than it otherwise
> would so it can decrypt the piece of data that it needs? 

Decryption is in 512 byte blocks. There is really no way around that
if you want to be secure. But 512 byte is already the smallest
unit you can read from disk.

> I have a basic
> crypto education (university class, read Applied Cryptography, used it
> plenty as a sysadmin/security guy) but don't know the details of how the
> IV is generated from the previous block in dm-crypt.

It is not, at least not from the previous disk block. In disk 
encryption you have to generate the IV somehow from the block 
number (e.g. via ESSIV) or do without (e.g. XTS mode). You still
have to decrypt each block fully. Blocks used in dm-crypt are
allways 512 Bytes in size, hence no extra reads or writes
compared to non-encrypted disks should ever be required. 

Disk encryption is a bit different than normal data-stream
encryption and not covered well or at all in the usual
crypto literature. 

> It looks like XTC mode uses the sector number as IV which might result in
> reading less data. Perhaps I should try ECB mode instead of my current:
> Cipher mode:    cbc-essiv:sha256

Please to not. ECB is massively insecure. And it will not result
in reading less data. Well, you can do it as an experiement,
but please do not use it when it has to be secure.

> > Also please note that this is exactly where RHEL customer
> > requests helps - and there were no such requests.
> > So other things get priority.
> We have RHEL also and can deploy this solution on RHEL and run the
> question by RedHat if it comes to that, no problem.  But it will likely
> still be on RHEL5.  However, this allows me to make a good argument for
> getting things upgraded to RHEL6.
> > So if you are using CentOS my advice is simple - try to upgrade
> > to CentOS6 and test it. It should be in some aspect better but still
> > database performance over dmcrypt can have problems.
> I'll try an ECB mode (I am aware of the cryptographic downside as far as
> identical plaintext blocks go) just to see if that is the issue.  

It should not. But if you notice a difference, it may be interesting 
to see what it is.

There is also another possible factor that may make a larger or 
smaller difference: If I remember correctly, you said the dump 
was going to the same disk, but from different partitions for
encrypted and non-encrypted. This may influence speeds. First,
head seeks are dependend on seek distance. The farther the heads 
have to move, the slower. And second, different disk regions
have different data density. My observation is that disk linear
throughput drops to less than half towards the end of many disks.
And third, you may have a different fragmentation or data placement 
situation or filesystem parameter on both source disks.

Generally moving a lot of data around on one disk is subject to 
all sorts of wired  effects. Often it is faster to move the data 
over a second disk as intermediate step.

So maybe you should try this with exactly the same partitions 
first, with exactly the same filesystems and data, just once
with dm-crypt and once without. Maybe even put the same binary
disk image in both, once on the raw partition and once on the
encrypted one. Or try with dumping to /dev/null. As far as I 
remember, mysqldump just produces an output data stream that
should be redirectable to STDOUT. Has been a few years that
I used MySQL last, though.

> Then I'll try RHEL/CentOS 6 and XTC.

Sounds like a good plan.


> Thanks!
> -- 
> Tracy Reed
> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

More information about the dm-crypt mailing list