[dm-crypt] Re : Re : Poor performances with nfs and Kernel 3.x
Mickael
mki2268 at yahoo.com
Wed Feb 8 18:55:21 CET 2012
Here are more tests focusing the CPU load with htop snapshot:
with Kernel 3.x, nfsd threads are using more CPU !?
Then kworker (process used for encryption) has less CPU time.
(The server is a simple fileserver using nfs, dm_crypt and ssh for administration.)
About the low non-encrypted performance; this is certainly due to AMD Kool&Quiet.
As CPU load is low, the CPU frequency stay at its minimum value 1GHz during non-encrypted access.
But when using encryption, the frequency is set to its maximum, 2.3GHz
*** KERNEL 2.6 ***
NFS Reading:
cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
read: 2.032 GB [ 2032330895 B] avg: 81.293 MB/sec [ 25 sec]
CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 68 kthr; 3 running
Mem[||||||||||||||||||||||||||||||||||||||||||||81/998MB] Load average: 2.68 2.93 1.79
Swp[ 0/1951MB] Uptime: 00:20:20
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
41 root 20 0 0 0 0 R 84.0 0.0 2:05.48 kworker/0:1
1020 root 20 0 0 0 0 D 1.0 0.0 0:10.72 nfsd
1015 root 20 0 0 0 0 D 1.0 0.0 0:07.29 nfsd
1019 root 20 0 0 0 0 D 1.0 0.0 0:08.35 nfsd
1022 root 20 0 0 0 0 D 1.0 0.0 0:11.00 nfsd
1016 root 20 0 0 0 0 D 1.0 0.0 0:07.41 nfsd
1017 root 20 0 0 0 0 D 1.0 0.0 0:08.79 nfsd
1021 root 20 0 0 0 0 D 1.0 0.0 0:08.41 nfsd
42 root 20 0 0 0 0 S 1.0 0.0 0:06.89 kswapd0
1018 root 20 0 0 0 0 D 1.0 0.0 0:11.10 nfsd
1460 root 20 0 0 0 0 R 0.0 0.0 0:01.34 kworker/0:0
1475 root 20 0 21972 1556 1148 R 0.0 0.2 0:01.79 htop
1 root 20 0 24188 2088 1168 S 0.0 0.2 0:00.48 /sbin/init
----------
NFS Writing:
cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi
read: 2.032 GB [ 2032330895 B] avg: 50.808 MB/sec [ 40 sec]
CPU[||||||||||||||||||||||||||||||||| 61.8%] Tasks: 36, 68 kthr; 2 running
Mem[|||||||||||||||||||||||| 86/998MB] Load average: 3.35 2.32 1.81
Swp[ 0/1951MB] Uptime: 00:27:15
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
41 root 20 0 0 0 0 S 41.0 0.0 3:21.84 kworker/0:1
1015 root 20 0 0 0 0 S 9.0 0.0 0:09.85 nfsd
1020 root 20 0 0 0 0 S 4.0 0.0 0:15.22 nfsd
1021 root 20 0 0 0 0 S 2.0 0.0 0:11.35 nfsd
1018 root 20 0 0 0 0 R 1.0 0.0 0:15.18 nfsd
1460 root 20 0 0 0 0 S 0.0 0.0 0:02.39 kworker/0:0
1017 root 20 0 0 0 0 S 0.0 0.0 0:12.03 nfsd
1488 root 20 0 0 0 0 S 0.0 0.0 0:01.66 flush-251:3
1016 root 20 0 0 0 0 S 0.0 0.0 0:10.30 nfsd
1442 root 20 0 0 0 0 S 0.0 0.0 0:03.41 jbd2/dm-3-8
42 root 20 0 0 0 0 S 0.0 0.0 0:08.25 kswapd0
1022 root 20 0 0 0 0 S 0.0 0.0 0:14.34 nfsd
1019 root 20 0 0 0 0 S 0.0 0.0 0:11.05 nfsd
91 root 20 0 0 0 0 S 0.0 0.0 0:01.27 kworker/0:2
1 root 20 0 24188 2088 1168 S 0.0 0.2 0:00.48 /sbin/init
========================================
*** KERNEL 3.0 ***
NFS Reading:
cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
read: 2.032 GB [ 2032330895 B] avg: 30.793 MB/sec [ 66 sec]
CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 65 kthr; 3 running
Mem[|||||||||||||||||||||||||||||||| 69/998MB] Load average: 2.02 0.47 0.16
Swp[ 0/1951MB] Uptime: 00:03:02
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
38 root 20 0 0 0 0 R 36.0 0.0 0:05.17 kworker/0:1
1045 root 20 0 0 0 0 R 36.0 0.0 0:03.07 nfsd
1046 root 20 0 0 0 0 D 3.0 0.0 0:02.23 nfsd
1049 root 20 0 0 0 0 D 3.0 0.0 0:00.52 nfsd
1044 root 20 0 0 0 0 D 3.0 0.0 0:00.70 nfsd
1047 root 20 0 0 0 0 D 3.0 0.0 0:00.54 nfsd
1042 root 20 0 0 0 0 D 3.0 0.0 0:00.51 nfsd
1043 root 20 0 0 0 0 D 3.0 0.0 0:00.49 nfsd
1048 root 20 0 0 0 0 D 3.0 0.0 0:00.50 nfsd
1446 root 20 0 21972 1552 1148 R 0.0 0.2 0:00.68 htop
1 root 20 0 24188 2188 1284 S 0.0 0.2 0:00.46 /sbin/init
----------
NFS Writing:
cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi
read: 2.032 GB [ 2032330895 B] avg: 36.951 MB/sec [ 55 sec]
CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 61 kthr; 3 running
Mem[||||||||||||||||||||||||||||||| 83/998MB] Load average: 2.39 2.26 1.10
Swp[ 0/1951MB] Uptime: 00:07:27
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
38 root 20 0 0 0 0 R 49.0 0.0 0:52.87 kworker/0:1
1048 root 20 0 0 0 0 R 49.0 0.0 0:12.66 nfsd
1045 root 20 0 0 0 0 D 0.0 0.0 0:16.80 nfsd
1044 root 20 0 0 0 0 D 0.0 0.0 0:09.27 nfsd
1043 root 20 0 0 0 0 D 0.0 0.0 0:10.83 nfsd
1042 root 20 0 0 0 0 D 0.0 0.0 0:08.08 nfsd
1047 root 20 0 0 0 0 D 0.0 0.0 0:09.70 nfsd
1467 root 20 0 21972 1556 1152 R 0.0 0.2 0:00.36 htop
1 root 20 0 24188 2072 1168 S 0.0 0.2 0:00.46 /sbin/init
> ----- Mail original -----
> De : Arno Wagner <arno at wagner.name>
> À : dm-crypt at saout.de
> Cc :
> Envoyé le : Mercredi 8 février 2012 16h26
> Objet : Re: [dm-crypt] Re : Poor performances with nfs and Kernel 3.x
>
>
> Have you looked at CPU load on the server? Maybe the difference is
> just the CPU-power that NFS consumes and decryption is already
> CPU-Limited before. The low non-encrypted performance,
> compared to local performance, is also a bit fishy.
>
> Pure speculation, but maybe something is consuming CPU
> on the server or the CPU is slowed-down in some way.
> If this not a CPU-power issue, then the CPU should have
> significant idle percentage during encrypted accesses.
>
> Arno
>
>
> On Wed, Feb 08, 2012 at 03:04:57PM +0000, Mickael wrote:
> > >De?: Arno Wagner <arno at wagner.name>
> > >??: dm-crypt at saout.de
> > >Cc?:
> > >Envoy? le : Mardi 7 f?vrier 2012 9h33
> > >Objet?: Re: [dm-crypt] Poor performances with nfs and Kernel 3.x
> > >
> > >O.k., no such obervation with 3.2.2 on client and server. I got
> > >
> > >root /gate/tmp>cat ttt | wcs > /dev/null
> > >read:? 2.147 GB [? 2147483648 B]? ? avg:? 69.274 MB/sec [? ? 31 sec]
> > >
> > >(wcs, a.k.a. wc-stream is a small tool I wrote to do real-time
> > >monitoring of pipeline throughput and byte count, sources
> > >below.) This is with some ramp-up and almost 100% CPU load
> > >on the (slower) server.
> > >
> > >I noticed though that nfsiostat is not the right tool to measure, as
> > >it gives you performance over the whole time the device has been
> > >mounted, i.e. the throughput number keeps being updated in
> > >real-time.
> > >
> > >Maybe re-run with wcs as shown above (compile instructions are in
> > >the header of wcs.c) and also monitor CPU usage on the server
> > >while this is rrunning, e.g. with "top". I am not saying you
> > >imagine the issue, but lets be sure the measurement is good.
> > >
> > >If you do not want to compile anything, you could also use
> > >something like "time cat ttt > /dev/null" and calculate
> > >throughput manually.
> > >
> > >Arno
> > >
> > >----- CUTed -----
> > >
> >
> >
> > Hello Arno,
> > Here are the new tests with wcs: results are still the same.
> >
> > /tmp is the tmp directory on the client (7200 rpm HD)
> > /CRYPT is a crypted partition on the server (Green HD ~5600 rpm)
> > /SAV is a non-crypted partition on the server (Green HD ~5600 rpm)
> > CPU monitored with htop
> >
> >
> > *** Server with Kernel 3.0.0-15-server #26
> >
> > * From client, using NFS on crypted partition:
> >
> > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 41.476 MB/sec [??? 49 sec]
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 29.454 MB/sec [??? 69 sec]??? <---------------
> >
> > -----------
> >
> > * From client, using NFS on non-crypted partition
> >
> > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 65.559 MB/sec [??? 31 sec]
> >
> > -----------
> >
> >
> > * From Server:
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]
> >
> > -----------
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 56.454 MB/sec [??? 36 sec]
> >
> > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]
> >
> >
> >
> > ===============================
> >
> > *** Server with Kernel 2.6.38-13-server #52
> >
> > * From client, using NFS on crypted partition:
> >
> > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]??? CPU=100%
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec]??? CPU=98%??? ??? <-------------------
> >
> > -----------
> >
> > * From client, using NFS on non-crypted partition
> >
> > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=50%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=25%
> >
> >
> >
> > * From Server:
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 84.680 MB/sec [??? 24 sec]??? CPU=98%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /dev/null
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]??? CPU=25%
> >
> > -----------
> >
> > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 52.111 MB/sec [??? 39 sec]??? CPU=50%-100%
> >
> > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi
> > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 47.264 MB/sec [??? 43 sec]??? CPU=75%-100%
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt at saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> >
>
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno at wagner.name
> GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
> ----
> One of the painful things about our time is that those who feel certainty
> are stupid, and those with any imagination and understanding are filled
> with doubt and indecision. -- Bertrand Russell
> _______________________________________________
> dm-crypt mailing list
> dm-crypt at saout.de
http://www.saout.de/mailman/listinfo/dm-crypt
More information about the dm-crypt
mailing list