[dm-crypt] ext3 + dm_crypt performance impact (CentOS 5.4 AMD64)

Robert.Heinzmann at deutschepost.de Robert.Heinzmann at deutschepost.de
Tue Dec 28 16:19:52 CET 2010


Hello again, 

> Von: mbroz at redhat.com [mailto:mbroz at redhat.com] 
> I do not want to speculate what exactly causes latency issues 
> - it is not one thing (testing of per-cpu worker threads 
> changes shows that page cache and fs are also important part 
> which can contribute to some problems here etc.)

I reran the test on 2.6.32 again and looked at vmstat for the test run and found something interesing: 

  Run1: ext3 with /dev/sdX backend 
  Run2: ext3 with dmcrypt backend (dm_crypt is backed by the same /dev/sdX)
  Run3: xfs with dmcrypt backend (dm_crypt is backed by the same /dev/sdX)

Run2 has a very stuttery I/O pattern, no constant I/O flow. For me this could explain the result of -35% performance für ext3 on dm_crypt. While Run1) and Run3) work in a pipeline fashion (where latency is not so much of an issue) - Run2 clearly distinguished different phases (write to buffer cache / encrypt / write to disk). Here tiny latency increases in the I/O will pile up in phase (2) encrypt.

This looks like a very odd behaviour / incompatiblity to me :/

Unfortunately I'm not a kernel expert, so I can't debug further. 

Regards, 
Robert 

----
Run1: 
======
ext3: Run on un-encrypted backend (plain disk): 

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 3621816 131712 3504132    0    0     8  2790  196  345  2  5 82 10  1
 2  0      0 3521500 135872 3586616    0    0    24     4  712  151  2 26 65  0  7
 0  3      0 3467760 138568 3628056    0    0    16 54916 2792 2003  2 17 11 69  1
 0  3      0 3408284 142220 3673428    0    0    12 46132 3516 2526  2 19  1 77  1
 1  2      0 3343768 145108 3726176    0    0    12 47828 2965 2206  2 15  1 78  4
 0  3      0 3271328 148116 3786512    0    0    12 45212 3087 2218  2 17  1 77  4
 1  2      0 3187632 151352 3856728    0    0    12 45992 3215 2370  2 17  0 80  1
 0  3      0 3095036 154432 3936248    0    0    16 45888 3215 2150  2 17 10 72  0
 2  2      0 2988832 157912 4026676    0    0     8 39804 2734 1066  2 22 47 30  0
 1  1      0 2937600 160608 4066824    0    0    16 36164 2524 1532  1 15 12 69  2
 1  1      0 2877724 164160 4115332    0    0    40 48636 1476  208  3 21 23 50  4
 1  1      0 2789056 168604 4187808    0    0     8 45516 1923  238  2 23 13 62  0
 1  2      0 2722748 171612 4244048    0    0    16 48520 1855  268  1 17 29 50  2
 0  3      0 2647884 174764 4307504    0    0    12 45768 2446 2401  2 18 23 54  3
 2  2      0 2563828 177716 4380532    0    0    12 37536 2713 2080  2 16  0 80  1
 1  3      0 2463276 181088 4465828    0    0    12 44952 3320 2230  3 20  0 75  2
 1  2      0 2385192 184224 4530712    0    0    12 47352 2708 2307  1 18  0 81  0
 0  3      0 2323952 187452 4577956    0    0    12 47512 3346 2289  3 18  0 77  2
 2  2      0 2268744 190812 4620344    0    0    12 47636 3346 2600  2 17  0 81  0
 1  2      0 2177296 194916 4695412    0    0     8 49964 3625 2421  2 21  0 74  3
 1  2      0 2648832 195028 4265308    0    0     0 29352 1665 1643  2 32  0 64  2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  3      0 3068280 195028 3865160    0    0     0 34360 1922 2481  1 29  9 61  0
 0  3      0 3103564 195076 3834508    0    0    16 48092 1925 1621  0  4 42 54  0
 0  3      0 3139772 195116 3801664    0    0    12 47444 2049 1845  0  3 52 44  0
 0  3      0 3196236 195120 3764352    0    0     0 39396 1852 2226  0  4 40 55  0
 0  3      0 3444792 195200 3531164    0    0    16 38064 2391 1594  1 11 55 33  0
 1  1      0 3736860 195224 3269744    0    0     4 23356 2991  700  1 19 54 25  2
 1  1      0 4244964 195224 2803476    0    0     0 18492 3625  435  1 33  4 62  0
 0  1      0 4254208 195328 2796780    0    0    20 50000 1431  112  0  3 46 51  0
 1  0      0 4532788  97148 2725008    0    0     1   136  835  458 17  6 66  9  2
 1  0      0 4532712  97148 2724992    0    0    41     5  807   93 32  0 65  0  2

 
Run2
======
ext3: Same run with encrypted Backend (dm_crypt, aes_cbc) 
 
 2  1      0 4161032 412308 2725000    0    0    40 315112 1463 1648  2 23 66  2  6
 2  2      0 4166480 412308 2725004    0    0     0     0 2844 2727  0 23 50 25  1
 1  1      0 4174364 412308 2725004    0    0     0     0 4138 3636  0 26 41 30  3
 1  1      0 4183044 412308 2725004    0    0     0     0 1887 1932  0 24 60 14  2
 1  1      0 4192220 412308 2725004    0    0     0     0 2335 2264  0 29 61 10  1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1      0 4200280 412316 2724996    0    0     0    28 2063 2155  0 23 59 18  0
 1  1      0 4533280  97208 2725084    0    0   105     4 1018 1241  0  8 54 39  0
 2  0      0 4438560  98168 2809272    0    0    32   108  844 5537 16 14 65  2  2
 1  0      0 4400668 100340 2838468    0    0    16     0  718 10816 23  9 65  1  2
 1  0      0 4367456 102364 2864068    0    0     8     0  680 8588 22  7 66  0  4
 1  0      0 4336292 104296 2887836    0    0    16     0  690 8833 19  7 67  1  5
 2  0      0 4303192 106016 2913276    0    0     8     0  685 9194 21  9 66  0  4
 1  2      0 4271408 106300 2916456    0    0     8 191356 1580  961  3 26 37 31  3
 1  2      0 4279096 106300 2916464    0    0     0     0 1775   87  0 17 59 24  0
 1  2      0 4286164 106300 2916464    0    0     0     0 1947   90  0 18 60 22  0
 0  3      0 4291620 106300 2916464    0    0     0     0 1704   91  0 15 58 27  0
 1  0      0 4283820 107044 2930500    0    0    16  9064  977 6038 14 10 48 26  2
 1  0      0 4266336 107648 2945788    0    0     0 17576 1116 7134 18 10 47 23  2
 1  0      0 4235284 108668 2972812    0    0     8     0  728 12201 26  6 64  1  3
 1  0      0 4204896 109732 2999168    0    0     8     0  743 11814 27  6 63  0  3
 1  0      0 4176104 110716 3024536    0    0     0     0  762 11966 27  6 66  0  2
 1  0      0 4147956 111676 3049548    0    0     8    56  791 11842 26  6 67  0  1
 2  1      0 4106776 112384 3068496    0    0     0 119492 1386 8563 26 28 10 22 14
 1  2      0 4113688 112408 3069244    0    0     4     0 1936  471  1 20  0 78  1
 1  0      0 4121364 112536 3071740    0    0     8  4688 1689 1174  2 16  5 72  4
 1  0      0 4097144 113432 3093496    0    0     0  4092 1151 10074 23  9 54  9  5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 4070032 114444 3117072    0    0     8     0  709 10846 23  6 65  1  6
 2  1      0 4038420 115340 3136936    0    0     0 66320 1681 9014 30 39  7 11 13
 0  3      0 4029340 115756 3147208    0    0     4 29604 1838 5212 14 30  2 39 15
 1  0      0 4018948 116400 3161668    0    0     8  2772 2072 6793 14  8 29 48  1
 1  0      0 3989892 117436 3187080    0    0     0     0  763 11976 27  6 66  0  1
 1  0      0 3959948 118748 3212344    0    0     8     0  745 10322 24  7 66  1  3
 3  2      0 3930692 119544 3228896    0    0     0 75952 1560 7265 25 34  7 19 15
 1  1      0 3914316 120640 3249400    0    0    12  4496 1569 9340 27 23 25 17  8
 1  0      0 3872148 122724 3290044    0    0     8     4  749 10154 21 11 66  1  2
 1  0      0 3754576 127996 3386976    0    0    24     0  748  212  5 28 66  1  0
 1  0      0 3662420 133292 3457428    0    0    32     0  770  285  6 26 66  2  0
 1  2      0 3607292 134544 3471488    0    0     4 247188 1545  114  1 30  4 59  7
 1  2      0 3613864 134544 3471488    0    0     0     0 1899   93  0 17  0 83  0
 1  2      0 3620684 134552 3471480    0    0     0    16 1810   96  0 17  0 83  0
 1  2      0 3626636 134552 3471488    0    0     0     0 1543   83  0 13  0 87  0
 1  2      0 3633580 134552 3471488    0    0     0     0 1835   88  0 16  0 84  0
 1  1      0 3628936 135296 3478356    0    0    24 19200 1659  115  1 24  9 63  4
 1  1      0 3554984 138016 3538396    0    0     8 77976 1143  116  2 43 13 25 16
 0  2      0 3554612 138016 3538428    0    0     0     0 1029   54  0  9 54 37  0
 0  2      0 3552644 138404 3545736    0    0     0     0 1397   95  1  6 50 43  0
 0  2      0 3545124 138752 3554636    0    0     4     0 2847  171  0  8 50 41  1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 3418976 143204 3665872    0    0    20     0  884  126  3 24 63  9  0
 1  2      0 3344964 144892 3709656    0    0     4 167920 1498  110  0 38 10 45  7
 1  2      0 3352404 144892 3709656    0    0     0     0 2006   99  0 20  4 75  0
 1  2      0 3360216 144892 3709656    0    0     0     0 1963   95  0 19  4 77  0
 1  0      0 3361600 145276 3716428    0    0    16 11456 1368  114  0 16 22 61  2
 1  0      0 3232156 150392 3827480    0    0    16     0  737  186  5 27 67  1  1
 1  2      0 3165872 152620 3863048    0    0    12 160908 1496  141  3 36 38 14  8
 0  3      0 3173188 152620 3863048    0    0     0     0 1929   93  0 20 60 20  0
 1  2      0 3180628 152620 3863048    0    0     0     0 1980   98  0 19 61 20  0
 1  2      0 3152568 154376 3884704    0    0    16 40216 1490  106  1 37 34 18 10
 1  0      0 3139400 155832 3901336    0    0     4    20 2173  167  1 11 18 70  0
 1  2      0 3095536 157908 3932828    0    0    12 47652 1582  148  3 35  9 45  9
 1  0      0 3005852 162460 4012048    0    0    24  5452  852  188  3 26 56 12  3
 1  0      0 2853664 167780 4143148    0    0    16     0  777  176  5 28 66  1  0
 2  0      0 2715040 172524 4259044    0    0    16  5704  809  197  3 31 62  1  3
 2  1      0 2613312 175684 4317960    0    0    12 243488 1553  113  3 56 16 10 16
 0  2      0 2592604 175684 4317944    0    0     0 158036 1681   82  0 20 12 65  3
 1  1      0 2600292 175684 4317944    0    0     0     0 2078   94  0 20  5 75  0
 1  1      0 2606244 175684 4317944    0    0     0     0 1622   79  0 16  3 81  0
 0  2      0 2613560 175700 4317932    0    0     4    16 1940  108  0 19  4 77  0
 1  1      0 2620752 175700 4317944    0    0     0     0 1892   90  0 18  4 78  0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  2      0 2626332 175700 4317944    0    0     0     0 1582   74  0 15  4 81  0
 0  2      0 2632284 175700 4317944    0    0     0     0 1612   81  0 15  4 81  0
 0  2      0 2642452 175700 4317944    0    0     0     0 1708   82  0 13  3 83  0
 2  1      0 2626680 176584 4331580    0    0     0 17220 2422  140  1 19  5 69  6
 1  0      0 2589492 179104 4363236    0    0    28 19496  975  143  2 27 17 39 14
 1  0      0 2495704 184692 4436920    0    0    24     0  753  284  6 27 66  1  0
 1  0      0 2351244 190080 4562020    0    0    16     0  761  189  4 29 66  1  0


Run3: 
=======
xfs: Now a xfs on the same device (dm_crypt, aes_cbc) 

0  0      0 4532784  97404 2724988    0    0     0     0   32   12  0  0 100  0  0
 0  0      0 4532816  97404 2724992    0    0     0     0   31   11  0  0 100  0  0
 0  0      0 4532816  97404 2724992    0    0     0     0   28   17  0  0 100  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 4532792  97412 2724984    0    0     0    68  211  340  1  1 97  0  0
 0  1      0 4533312  97428 2724980    0    0    20 10574  945  849  1  7 55 36  1
 1  0      0 4446004  97420 2805168    0    0   179  4404 1119 3817 11  9 73  5  2
 1  0      0 4423212  97420 2823960    0    0     0  6984 1101 7801 19 10 60  0 11
 1  0      0 4392896  97420 2847932    0    0     0 10924 1408 8646 20 15 61  0  4
 1  0      0 4370208  97420 2867048    0    0     0  7644  997 6719 15  9 73  0  3
 1  0      0 4343520  97420 2888740    0    0     0 10768 1236 7424 21 13 58  0  8
 1  0      0 4317964  97428 2909456    0    0     0 11408 1335 6238 20 12 59  0  9
 1  0      0 4297124  97428 2926644    0    0     0 10040 1158 5890 16 10 74  0  1
 1  0      0 4281416  97428 2940348    0    0     0  4296  735 6210 14  4 79  0  3
 1  0      0 4265972  97428 2954104    0    0     0  2868  691 6297 13  4 81  0  3
 1  0      0 4242736  97428 2975012    0    0     0  4564  955 10067 21  8 66  0  5
 1  0      0 4217148  97428 2996796    0    0     0  7040 1059 10220 23  9 61  0  8
 1  0      0 4192024  97428 3019292    0    0     0  5256 1107 10753 24  8 63  0  4
 1  0      0 4165704  97428 3043512    0    0     0  4240 1135 11862 26  7 66  0  1
 1  0      0 4139056  97428 3067328    0    0     0  5184 1179 11436 25  9 63  0  3
 1  0      0 4114124  97428 3090204    0    0     0  4200 1042 11076 23  7 64  0  6
 1  0      0 4093592  97428 3103028    0    0     0 27320 1137 6365 12 15 58 11  4
 1  0      0 4077564  97428 3122936    0    0     0  5000  957 9662 20  7 68  0  4
 2  0      0 4056956  97428 3139108    0    0     0 22064 1426 7383 17 15 60  0  8
 2  0      0 4043508  97428 3151752    0    0     0 18389 1228 5802 14 11 69  0  6
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 4032116  97428 3161168    0    0     0 48057 2334 4014 12 29 55  0  4
 0  0      0 4022456  97428 3170792    0    0     0 42444 2208 3864 11 25 58  0  5
 2  0      0 4012436  97428 3176364    0    0     0 35000 1310 3259  8 17 71  0  4
 1  0      0 4007912  97428 3182172    0    0     0 46332 2095 3005  8 24 61  3  5
 2  0      0 3999120  97428 3188468    0    0     0 47804 2062 2820  9 24 59  3  5
 2  0      0 3996056  97428 3193124    0    0     0 45692 2248 2641  7 22 57 10  4
 0  1      0 3986408  97428 3198288    0    0     0 48204 1703 2621  5 18 57 18  2
 1  0      0 3984172  97428 3203504    0    0     0 41144 2119 2729  5 22 63  6  4
 1  0      0 3979080  97428 3210696    0    0     0 39480 1582 3259  8 20 65  3  4
 1  0      0 3964364  97428 3223272    0    0     0 10394 1140 5035 12  9 73  0  6
 1  0      0 3940640  97428 3243824    0    0     0  5640 1014 9436 22  8 66  0  3
 1  0      0 3912652  97428 3267928    0    0     0  5604 1120 11204 24  9 65  0  1
 1  0      0 3875980  97428 3302288    0    0     0 10744 1571 8815 16 18 57  0  9
 1  0      0 3817940  97428 3349316    0    0     0 25240 2875 2942  1 36 54  0  8
 1  0      0 3755244  97428 3400224    0    0     0 31032 3200 3339  3 42 48  0  8
 2  0      0 3661368  97428 3481976    0    0     0 31164 3279 3474  3 43 48  0  6
 1  0      0 3602928  97428 3534648    0    0     0 33411 2293 2334  1 29 62  0  8
 0  0      0 3579180  97428 3554604    0    0     0 48600 2403 2431  0 30 66  0  4
----




More information about the dm-crypt mailing list