[dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

"C. Dominik Bódi" dominik.bodi at gmx.de
Tue Feb 25 18:21:05 CET 2014

Hash: SHA512

(please CC me, I'm not subscribed to the list)


I've discovered some strange behaviour on my Debian system, with
dm-crypt on raid 5 showing slow performance and I am seeking ways to
remedy that.

My setup:
Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI)
Intel X58 chipset
Debian sid/unstable
kernels tried: 3.12.9, 3.13.4 (Debian kernels)
4 x 3 TB SATA disks on a Intel X58 AHCI controller.
That controller has 6 SATA ports and connects directly to the chipsets
South Bridge, which connects to the North Bridge via a very fast DMI link.
Second PCIe-SATA controller on motherboard (ACHI driver, probably
JMicron chipset) - 2 SATA ports.

On that disk I have a RAID5 array (source media are the 4th partition on
each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS
volume. That LUKS volume is a PV for LVM.

4 disks => RAID5 => LUKS Volume => LVM PV

Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read
speed. Each single disk has about 170MB/s with hdparm, so that is ok.

However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s.
Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.

Now I connected two of the four disks to the second SATA controller

hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the
RAID5 array dropped to ca. 400MB/s. That was to be expected, as that
second controller seems somewhat slower.

And now comes the funny part:
Now I did a hdparm -t /dev/mapper/encrypted_raid5
and that got me ca. 300MB/s

Running top whilst doing the hdparm tests showed me that, when all 4
disks were on the Intel X58 AHCI controller, only 1 kworker thread was
running on a single CPU core, utilizing 100% CPU time.

When 2 of those disks were connected to the second SATA controller, 2
kworker threads were running on two different CPU cores.

It seems that when the disks sit on two different controllers, dm-crypt
is using 2 kworker threads to handle the data, but uses only one kworker
thread when all disks sit on a single controller.

My guess is that one single cpu core can decrypt/encrypt the LUKS volume
at about 150MB/s, which seems to be reasonable for that kind of CPU.
Thus, dm-crypt using more than one kworker threads leads to doubling of
the decryption speed, even if the two disks are slightly slower when
connected to the second controller.

I'd like dm-crypt to use as many threads as possible (e.g. one kworker
for each disk in the RAID5 array). That would give me reasonable speeds
for the encrypted LUKS volume, close to that of the unencrypted RAID5

I've experimented with the RAID5 arrays group_thread_cnt sysfs setting.
That is set to zero by default, disabling multithreaded RAID5. However,
enabling that and setting it to 2 or 4 did not change anything speed-wise.

I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity
setting, yet. Would that help?

Is there a way to enforce dm-crypt using more than one kworker thread to
encrypt/decrypt data for a crypt-on-raid5 setup?

Version: GnuPG v1


More information about the dm-crypt mailing list