[dm-crypt] slow read performance, but fast writes?

Arno Wagner arno at wagner.name
Wed Sep 14 17:04:00 CEST 2011


The RAID6 calculations should not matter much. For example
here is the RAID6 bootup-benchmark from my elderly
Athlon 64 X2 5600+:
 ...
 raid6: using algorithm sse2x2 (4867 MB/s)

What I suspect for your problem on reading is an I/O
chip or attachment with asymmetrical performance.
This can happen, e.g. when the buffering is asymetrical
or one direction has a long switchover time when going 
to/from different targets while the other does not.
Reads have to read one stripe in turn from each disk, 
(with head movement, rotational latency, etc.) while
writes go to 1. OS buffer and 2. disk buffer.

So writes are in principle faster on any Unix, as they 
get buffered by the kernel and the disks, while reads do not. 
Writes used to be slower because disk heads would need to
be positioned a lot better, but this difference has mostly 
vanished in modern drives.

Still, I suspect a hardware problem in the contoller(s)
that the disks attach to or their attachment to the system.

Arno


On Wed, Sep 14, 2011 at 04:04:28PM +0200, Peter Merhaut wrote:
> Hi,
> 
> i've got a really strange problem with my setup, and i'm not quite sure
> why.
> First of all, the setup. Got 6x 2TB Samsung hdds, each one encrypted
> with dm_crypt, on top of the encrypted block devices, there's a raid6.
> Normally you would assume, that writes are slower than reads, since
> dm-crypt has to encrypt 2 additional parity stripes (and md raid6 has to
> calculate the 2 parity stripes).
> Let's take a look on the write performance first. Here's an average of
> 10 seconds iostat while running "dd if=/dev/zero of=/daten/testfile bs=1M"
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.12    0.05   94.15    2.17    0.00    3.50
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sdb               9.30 15200.00    1.80 2972.70    44.40 73472.40   
> 49.43    15.53    5.21  201.72    5.09   0.26  76.68
> sdd               6.30 14495.00    1.60 3759.60    31.60 73756.00   
> 39.24     2.91    0.77  133.44    0.71   0.19  73.08
> sdf               3.10 14519.30    1.70 3650.00    19.20 73429.20   
> 40.23     2.86    0.78  137.06    0.72   0.20  72.52
> sde               6.10 14240.10    1.30 3964.20    29.60 73572.40   
> 37.12     2.80    0.70  173.38    0.65   0.18  71.70
> sdg               3.10 14859.40    1.10 3306.50    16.80 73410.00   
> 44.40     2.72    0.82    3.45    0.82   0.19  63.06
> sdc               0.20 14276.20    1.00 3909.40     4.80 73450.00   
> 37.57     2.65    0.67   41.80    0.66   0.17  65.29
> md3               0.00     0.00    0.00 4609.60     0.00 294870.40  
> 127.94     0.00    0.00    0.00    0.00   0.00   0.00
> 
> Works as expected. Since this machine is powered by an quadcore, all
> this crypto and raid6 threads are nicely balanced over all 4 cores.
> Sequential Write averages out at about 300MB/s.
> So the CPU is capable of encrypting 6 times 72MB per second (while
> calculating parity for raid6).
> 
> Now for the Reads.
> 
> 10 seconds iostat while running "dd if=/dev/md3 of=/dev/null bs=1M"
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.15    0.05   37.61   19.90    0.00   42.29
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sdb            5700.20     0.00 1141.90    0.00 27507.20     0.00   
> 48.18     6.37    5.52    5.52    0.00   0.42  48.27
> sdd            5420.60     0.00 1419.80    0.00 27462.40     0.00   
> 38.68     4.47    3.14    3.14    0.00   0.30  42.12
> sdf            5646.90     0.00 1198.10    0.00 27418.00     0.00   
> 45.77     5.50    4.57    4.57    0.00   0.36  42.91
> sde            5837.10     0.00 1006.40    0.00 27507.20     0.00   
> 54.66     6.69    6.61    6.61    0.00   0.49  49.45
> sdg            5671.10     0.00 1171.30    0.00 27401.20     0.00   
> 46.79     4.65    3.95    3.95    0.00   0.34  39.99
> sdc            5994.20     0.00  851.30    0.00 27485.60     0.00   
> 64.57     7.11    8.27    8.27    0.00   0.57  48.44
> md3               0.00     0.00 40927.60    0.00 163710.40     0.00    
> 8.00     0.00    0.00    0.00    0.00   0.00   0.00
> 
> 42% idle and almost 20% iowait?
> I've already tried everything i thought of. changed kernel CONFIG_HZ,
> tried a tickless Kernel, enabled/disabled ncq, changed bios from ahci to
> ide mode, tuned read ahead for harddisks, crypto-devices and raid array
> (with different combinations).
> Same thing happens if i start badblocks on all 6 crypto devices.
> Througput drops to 29MB/s on each drive. When i overclock the CPU (AMD
> 910e) about 20%, the read rate increases at exactly 20%.
> Any advice would be greatly appreciated.
> 
> thanks and best regards, Peter

> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 


More information about the dm-crypt mailing list