[dm-crypt] Re : Re : Poor performances with nfs and Kernel 3.x

Mickael mki2268 at yahoo.com
Wed Feb 8 18:55:21 CET 2012


Here are more tests focusing the CPU load with htop snapshot:

with Kernel 3.x, nfsd threads are using more CPU !?
Then kworker (process used for encryption) has less CPU time.
(The server is a simple fileserver using nfs, dm_crypt and ssh for administration.)

About the low non-encrypted performance; this is certainly due to AMD Kool&Quiet.
As CPU load is low, the CPU frequency stay at its minimum value 1GHz during non-encrypted access.
But when using encryption, the frequency is set to its maximum, 2.3GHz


*** KERNEL 2.6 ***

NFS Reading:
cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
 read:   2.032 GB [   2032330895 B]    avg:  81.293 MB/sec [    25 sec]

  CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Tasks: 36, 68 kthr; 3 running
  Mem[||||||||||||||||||||||||||||||||||||||||||||81/998MB]     Load average: 2.68 2.93 1.79
  Swp[                                            0/1951MB]     Uptime: 00:20:20

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
   41 root      20   0     0     0     0 R 84.0  0.0  2:05.48 kworker/0:1
 1020 root      20   0     0     0     0 D  1.0  0.0  0:10.72 nfsd
 1015 root      20   0     0     0     0 D  1.0  0.0  0:07.29 nfsd
 1019 root      20   0     0     0     0 D  1.0  0.0  0:08.35 nfsd
 1022 root      20   0     0     0     0 D  1.0  0.0  0:11.00 nfsd
 1016 root      20   0     0     0     0 D  1.0  0.0  0:07.41 nfsd
 1017 root      20   0     0     0     0 D  1.0  0.0  0:08.79 nfsd
 1021 root      20   0     0     0     0 D  1.0  0.0  0:08.41 nfsd
   42 root      20   0     0     0     0 S  1.0  0.0  0:06.89 kswapd0
 1018 root      20   0     0     0     0 D  1.0  0.0  0:11.10 nfsd
 1460 root      20   0     0     0     0 R  0.0  0.0  0:01.34 kworker/0:0
 1475 root      20   0 21972  1556  1148 R  0.0  0.2  0:01.79 htop
    1 root      20   0 24188  2088  1168 S  0.0  0.2  0:00.48 /sbin/init


----------
NFS Writing:
cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi
 read:   2.032 GB [   2032330895 B]    avg:  50.808 MB/sec [    40 sec]

  CPU[|||||||||||||||||||||||||||||||||              61.8%]     Tasks: 36, 68 kthr; 2 running
  Mem[||||||||||||||||||||||||                    86/998MB]     Load average: 3.35 2.32 1.81
  Swp[                                            0/1951MB]     Uptime: 00:27:15

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
   41 root      20   0     0     0     0 S 41.0  0.0  3:21.84 kworker/0:1
 1015 root      20   0     0     0     0 S  9.0  0.0  0:09.85 nfsd
 1020 root      20   0     0     0     0 S  4.0  0.0  0:15.22 nfsd
 1021 root      20   0     0     0     0 S  2.0  0.0  0:11.35 nfsd
 1018 root      20   0     0     0     0 R  1.0  0.0  0:15.18 nfsd
 1460 root      20   0     0     0     0 S  0.0  0.0  0:02.39 kworker/0:0
 1017 root      20   0     0     0     0 S  0.0  0.0  0:12.03 nfsd
 1488 root      20   0     0     0     0 S  0.0  0.0  0:01.66 flush-251:3
 1016 root      20   0     0     0     0 S  0.0  0.0  0:10.30 nfsd
 1442 root      20   0     0     0     0 S  0.0  0.0  0:03.41 jbd2/dm-3-8
   42 root      20   0     0     0     0 S  0.0  0.0  0:08.25 kswapd0
 1022 root      20   0     0     0     0 S  0.0  0.0  0:14.34 nfsd
 1019 root      20   0     0     0     0 S  0.0  0.0  0:11.05 nfsd
   91 root      20   0     0     0     0 S  0.0  0.0  0:01.27 kworker/0:2
    1 root      20   0 24188  2088  1168 S  0.0  0.2  0:00.48 /sbin/init



========================================

*** KERNEL 3.0 ***

NFS Reading:
cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
 read:   2.032 GB [   2032330895 B]    avg:  30.793 MB/sec [    66 sec]

  CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Tasks: 36, 65 kthr; 3 running
  Mem[||||||||||||||||||||||||||||||||            69/998MB]     Load average: 2.02 0.47 0.16
  Swp[                                            0/1951MB]     Uptime: 00:03:02

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
   38 root      20   0     0     0     0 R 36.0  0.0  0:05.17 kworker/0:1
 1045 root      20   0     0     0     0 R 36.0  0.0  0:03.07 nfsd
 1046 root      20   0     0     0     0 D  3.0  0.0  0:02.23 nfsd
 1049 root      20   0     0     0     0 D  3.0  0.0  0:00.52 nfsd
 1044 root      20   0     0     0     0 D  3.0  0.0  0:00.70 nfsd
 1047 root      20   0     0     0     0 D  3.0  0.0  0:00.54 nfsd
 1042 root      20   0     0     0     0 D  3.0  0.0  0:00.51 nfsd
 1043 root      20   0     0     0     0 D  3.0  0.0  0:00.49 nfsd
 1048 root      20   0     0     0     0 D  3.0  0.0  0:00.50 nfsd
 1446 root      20   0 21972  1552  1148 R  0.0  0.2  0:00.68 htop
    1 root      20   0 24188  2188  1284 S  0.0  0.2  0:00.46 /sbin/init


----------
NFS Writing:
cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi
 read:   2.032 GB [   2032330895 B]    avg:  36.951 MB/sec [    55 sec]

  CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Tasks: 36, 61 kthr; 3 running
  Mem[|||||||||||||||||||||||||||||||             83/998MB]     Load average: 2.39 2.26 1.10
  Swp[                                            0/1951MB]     Uptime: 00:07:27

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
   38 root      20   0     0     0     0 R 49.0  0.0  0:52.87 kworker/0:1
 1048 root      20   0     0     0     0 R 49.0  0.0  0:12.66 nfsd
 1045 root      20   0     0     0     0 D  0.0  0.0  0:16.80 nfsd
 1044 root      20   0     0     0     0 D  0.0  0.0  0:09.27 nfsd
 1043 root      20   0     0     0     0 D  0.0  0.0  0:10.83 nfsd
 1042 root      20   0     0     0     0 D  0.0  0.0  0:08.08 nfsd
 1047 root      20   0     0     0     0 D  0.0  0.0  0:09.70 nfsd
 1467 root      20   0 21972  1556  1152 R  0.0  0.2  0:00.36 htop
    1 root      20   0 24188  2072  1168 S  0.0  0.2  0:00.46 /sbin/init



> ----- Mail original -----
> De : Arno Wagner <arno at wagner.name>
> À : dm-crypt at saout.de
> Cc :
> Envoyé le : Mercredi 8 février 2012 16h26
> Objet : Re: [dm-crypt] Re : Poor performances with nfs and Kernel 3.x
>
>
> Have you looked at CPU load on the server? Maybe the difference is
> just the CPU-power that NFS consumes and decryption is already
> CPU-Limited before. The low non-encrypted performance,
> compared to local performance, is also a bit fishy.
>
> Pure speculation, but maybe something is consuming CPU
> on the server or the CPU is slowed-down in some way.
> If this not a CPU-power issue, then the CPU should have
> significant idle percentage during encrypted accesses.
>
> Arno
>
>
> On Wed, Feb 08, 2012 at 03:04:57PM +0000, Mickael wrote:
> > >De?: Arno Wagner <arno at wagner.name>
> > >??: dm-crypt at saout.de
> > >Cc?:
> > >Envoy? le : Mardi 7 f?vrier 2012 9h33
> > >Objet?: Re: [dm-crypt] Poor performances with nfs and Kernel 3.x
> > >
> > >O.k., no such obervation with 3.2.2 on client and server. I got
> > >
> > >root /gate/tmp>cat ttt | wcs > /dev/null
> > >read:?  2.147 GB [?  2147483648 B]? ? avg:? 69.274 MB/sec [? ? 31 sec]
> > >
> > >(wcs, a.k.a. wc-stream is a small tool I wrote to do real-time
> > >monitoring of pipeline throughput and byte count, sources
> > >below.) This is with some ramp-up and almost 100% CPU load
> > >on the (slower) server.
> > >
> > >I noticed though that nfsiostat is not the right tool to measure, as
> > >it gives you performance over the whole time the device has been
> > >mounted, i.e. the throughput number keeps being updated in
> > >real-time.
> > >
> > >Maybe re-run with wcs as shown above (compile instructions are in
> > >the header of wcs.c) and also monitor CPU usage on the server
> > >while this is rrunning, e.g. with "top". I am not saying you
> > >imagine the issue, but lets be sure the measurement is good.
> > >
> > >If you do not want to compile anything, you could also use
> > >something like "time cat ttt > /dev/null" and calculate
> > >throughput manually.
> > >
> > >Arno
> > >
> > >----- CUTed -----
> > >
> >
> >
> > Hello Arno,
> > Here are the new tests with wcs: results are still the same.
> >
> > /tmp is the tmp directory on the client (7200 rpm HD)
> > /CRYPT is a crypted partition on the server (Green HD ~5600 rpm)
> > /SAV is a non-crypted partition on the server (Green HD ~5600 rpm)
> > CPU monitored with htop
> >
> >
> > *** Server with Kernel 3.0.0-15-server #26
> >
> > * From client, using NFS on crypted partition:
> >
> > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 41.476 MB/sec [??? 49 sec]
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 29.454 MB/sec [??? 69 sec]??? <---------------
> >
> > -----------
> >
> > * From client, using NFS on non-crypted partition
> >
> > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 65.559 MB/sec [??? 31 sec]
> >
> > -----------
> >
> >
> > * From Server:
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]
> >
> > -----------
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 56.454 MB/sec [??? 36 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]
> >
> >
> >
> > ===============================
> >
> > *** Server with Kernel 2.6.38-13-server #52
> >
> > * From client, using NFS on crypted partition:
> >
> > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]??? CPU=100%
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec]??? CPU=98%??? ??? <-------------------
> >
> > -----------
> >
> > * From client, using NFS on non-crypted partition
> >
> > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=50%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=25%
> >
> >
> >
> > * From Server:
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 84.680 MB/sec [??? 24 sec]??? CPU=98%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]??? CPU=25%
> >
> > -----------
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 52.111 MB/sec [??? 39 sec]??? CPU=50%-100%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 47.264 MB/sec [??? 43 sec]??? CPU=75%-100%
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt at saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> >
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
> ----
> One of the painful things about our time is that those who feel certainty
> are stupid, and those with any imagination and understanding are filled
> with doubt and indecision. -- Bertrand Russell
> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
http://www.saout.de/mailman/listinfo/dm-crypt


More information about the dm-crypt mailing list