[dm-crypt] dm-crypt on raid 5 - having all disks on single controller makes dm-crypt slower

"C. Dominik Bódi" dominik.bodi at gmx.de
Tue Feb 25 18:21:05 CET 2014


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

(please CC me, I'm not subscribed to the list)

Hello,

I've discovered some strange behaviour on my Debian system, with
dm-crypt on raid 5 showing slow performance and I am seeking ways to
remedy that.

My setup:
Gigabyte EX-58UDR3 motherboeard with an Intel Corei7 920 (no AES-NI)
Intel X58 chipset
Debian sid/unstable
kernels tried: 3.12.9, 3.13.4 (Debian kernels)
4 x 3 TB SATA disks on a Intel X58 AHCI controller.
That controller has 6 SATA ports and connects directly to the chipsets
South Bridge, which connects to the North Bridge via a very fast DMI link.
Second PCIe-SATA controller on motherboard (ACHI driver, probably
JMicron chipset) - 2 SATA ports.

On that disk I have a RAID5 array (source media are the 4th partition on
each disk) called /dev/md/luks. On top of that RAID5 I've created a LUKS
volume. That LUKS volume is a PV for LVM.

4 disks => RAID5 => LUKS Volume => LVM PV

Doing tests with hdparm -t, the RAID5 arrays shows ca. 520MB/s read
speed. Each single disk has about 170MB/s with hdparm, so that is ok.

However, hdparm -t /dev/mapper/encrypted_raid5 gets me only ca. 150MB/s.
Performing dd tests on the mounted LVM filesystems gets me about 160MB/s.

Now I connected two of the four disks to the second SATA controller

hdparm for those disks dropped to ca. 130MB/s, and hdparm -t for the
RAID5 array dropped to ca. 400MB/s. That was to be expected, as that
second controller seems somewhat slower.

And now comes the funny part:
Now I did a hdparm -t /dev/mapper/encrypted_raid5
and that got me ca. 300MB/s

Running top whilst doing the hdparm tests showed me that, when all 4
disks were on the Intel X58 AHCI controller, only 1 kworker thread was
running on a single CPU core, utilizing 100% CPU time.

When 2 of those disks were connected to the second SATA controller, 2
kworker threads were running on two different CPU cores.

It seems that when the disks sit on two different controllers, dm-crypt
is using 2 kworker threads to handle the data, but uses only one kworker
thread when all disks sit on a single controller.

My guess is that one single cpu core can decrypt/encrypt the LUKS volume
at about 150MB/s, which seems to be reasonable for that kind of CPU.
Thus, dm-crypt using more than one kworker threads leads to doubling of
the decryption speed, even if the two disks are slightly slower when
connected to the second controller.

I'd like dm-crypt to use as many threads as possible (e.g. one kworker
for each disk in the RAID5 array). That would give me reasonable speeds
for the encrypted LUKS volume, close to that of the unencrypted RAID5
volume.

I've experimented with the RAID5 arrays group_thread_cnt sysfs setting.
That is set to zero by default, disabling multithreaded RAID5. However,
enabling that and setting it to 2 or 4 did not change anything speed-wise.

I haven't tried to fiddle with the raid5's or luks_volume's rq_affinity
setting, yet. Would that help?

Is there a way to enforce dm-crypt using more than one kworker thread to
encrypt/decrypt data for a crypt-on-raid5 setup?

Regards,
Dominik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJTDND9AAoJEH57oErWeAO1IHgP/jghJGdRS/mWZ0SU3goeWSB3
0Bii9xRb0Eo1tPS+TTXRad308pCL8BXloNdgLJXL9T1P/TLOaX0e7QxVRCC3j6Uj
siBZ7rMBp60bHQu6JNUyXPwssgQhyRhL7Web2tHm7RQYDUoNaVdIjL0p4KvCRAX6
4RE2FzMEze/DgumZtXYgSlBBxVb9Za87Y/iwH+SJQ87kloKPkD7kgX2hByYJ5D7h
NswCns1slj5KYyOSqikcoJM1Z9Sd1eJjMhRJ37/hH5ERMiP7m+LMO07ZsxtPCdE0
Fyi1J6Ra28YzKJPTcaBSa2oy+zeauennCIgLbA2px+7t3zEoaUiMSg4WrFbrV8i4
x1WWiHHrZV23dza2Hgy4BcE1lXAGxEKCuOSZr0Js/b20DfCeoV84oc4GIlMo3ZL4
LztRJ4yrtOUehH9KUwTqFLFk3r9xWDMC3ZlrbyyoxVAB5Nr+w1FnAzQRnvC7DOHb
wU2lj9zWloNT8hUsCwtdpLxQWSj9iWw1j70xuqSUGIP3SLBgBQdUfpZP0Xwwe0Bi
577yt6CbQcaqGgjhtlgetwL9e8Vj8rMXbKZUKbyimimwmHEAJ++pdOWnBTdz7NOQ
IKXP29mbI7+uIrzcgQvH6o5NS6psxdIcuxjfgYoR8CLPcNB87xKDcMXF9hU8JkZA
fVE8Z2X5LPDksJrs3eb4
=5Mww
-----END PGP SIGNATURE-----


More information about the dm-crypt mailing list