[dm-crypt] bcache gets stuck flushing writeback cache when used in combination with LUKS/dm-crypt and non-default bucket size

Eric Wheeler bcache at lists.ewheeler.net
Wed May 11 03:38:38 CEST 2016


On Sun, 8 May 2016, James Johnston wrote:

> Hi,
> 
> [1.] One line summary of the problem:
> 
> bcache gets stuck flushing writeback cache when used in combination with
> LUKS/dm-crypt and non-default bucket size
> 
> [2.] Full description of the problem/report:
> 
> I've run into a problem where the bcache writeback cache can't be flushed to
> disk when the backing device is a LUKS / dm-crypt device and the cache set has
> a non-default bucket size.  

You might try LUKS atop of bcache instead of under it.  This might be 
better for privacy too, otherwise your cached data is unencrypted.

> # Make cache set on second drive
> # IMPORTANT:  Problem does not occur if I omit --bucket parameter.
> make-bcache --bucket 2M -C /dev/sdb

2MB is quite large, maybe it exceeds the 256-bvec limit.  I'm not sure if 
Ming Lei's patch got in to 4.6 yet, but try this:
  https://lkml.org/lkml/2016/4/5/1046

and maybe Shaohua Li's patch too:
  http://www.spinics.net/lists/raid/msg51830.html


--
Eric Wheeler

> # Set up LUKS/dm-crypt on second drive.
> # IMPORTANT:  Problem does not occur if I omit the dm-crypt layer.
> cryptsetup luksFormat /dev/sdc
> cryptsetup open --type luks /dev/sdc backCrypt
> # Make bcache backing device & enable writeback
> make-bcache -B /dev/mapper/backCrypt
> bcache-super-show /dev/sdb | grep cset.uuid | \
> cut -f 3 > /sys/block/bcache0/bcache/attach
> echo writeback > /sys/block/bcache0/bcache/cache_mode
> 
> 4.  Finally, this is the kill sequence to bring the system to its knees:
> 
> sudo -s
> cd /sys/block/bcache0/bcache
> echo 0 > sequential_cutoff
> # Verify that the cache is attached (i.e. does not say "no cache").  It should
> # say that it's clean since we haven't written anything yet.
> cat state
> # Copy some random data.
> dd if=/dev/urandom of=/dev/bcache0 bs=1M count=250
> # Show current state.  On my system approximately 20 to 25 MB remain in
> # writeback cache.
> cat dirty_data
> cat state
> # Detach the cache set.  This will start the cache set disk thrashing.
> echo 1 > detach
> # After a few moments, confirm that the cache set is not going anywhere.  On
> # my system, only a few MB have been flushed as evidenced by a small decrease
> # in dirty_data.  State remains dirty.
> cat dirty_data
> cat state
> # At this point, the hypervisor system reports hundreds of MB/second of reads
> # to the underlying physical SSD coming from the virtual machine; the hard drive
> # light is stuck on...  hypervisor status bar shows the activity is on cache
> # set.  No writes seem to be occurring on any disk.
> 
> [8.] Environment
> [8.1.] Software (add the output of the ver_linux script here)
> Linux bcachetest2 4.6.0-040600rc6-generic #201605012031 SMP Mon May 2 00:33:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> 
> Util-linux              2.27.1
> Mount                   2.27.1
> Module-init-tools       22
> E2fsprogs               1.42.13
> Xfsprogs                4.3.0
> Linux C Library         2.23
> Dynamic linker (ldd)    2.23
> Linux C++ Library       6.0.21
> Procps                  3.3.10
> Net-tools               1.60
> Kbd                     1.15.5
> Console-tools           1.15.5
> Sh-utils                8.25
> Udev                    229
> Modules Loaded          8250_fintek ablk_helper aesni_intel aes_x86_64 ahci async_memcpy async_pq async_raid6_recov async_tx async_xor autofs4 btrfs configfs coretemp crc32_pclmul crct10dif_pclmul cryptd drm drm_kms_helper e1000 fb_sys_fops fjes gf128mul ghash_clmulni_intel glue_helper hid hid_generic i2c_piix4 ib_addr ib_cm ib_core ib_iser ib_mad ib_sa input_leds iscsi_tcp iw_cm joydev libahci libcrc32c libiscsi libiscsi_tcp linear lrw mac_hid mptbase mptscsih mptspi multipath nfit parport parport_pc pata_acpi ppdev psmouse raid0 raid10 raid1 raid456 raid6_pq rdma_cm scsi_transport_iscsi scsi_transport_spi serio_raw shpchp syscopyarea sysfillrect sysimgblt ttm usbhid vmw_balloon vmwgfx vmw_vmci vmw_vsock_vmci_transport vsock xor
> 
> [8.2.] Processor information (from /proc/cpuinfo):
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 42
> model name      : Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
> stepping        : 7
> microcode       : 0x29
> cpu MHz         : 2491.980
> cache size      : 3072 KB
> physical id     : 0
> siblings        : 1
> core id         : 0
> cpu cores       : 1
> apicid          : 0
> initial apicid  : 0
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm ida arat pln pts
> bugs            :
> bogomips        : 4983.96
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 42 bits physical, 48 bits virtual
> power management:
> 
> [8.3.] Module information (from /proc/modules):
> ppdev 20480 0 - Live 0x0000000000000000
> vmw_balloon 20480 0 - Live 0x0000000000000000
> vmw_vsock_vmci_transport 28672 1 - Live 0x0000000000000000
> vsock 36864 2 vmw_vsock_vmci_transport, Live 0x0000000000000000
> coretemp 16384 0 - Live 0x0000000000000000
> joydev 20480 0 - Live 0x0000000000000000
> input_leds 16384 0 - Live 0x0000000000000000
> serio_raw 16384 0 - Live 0x0000000000000000
> shpchp 36864 0 - Live 0x0000000000000000
> vmw_vmci 65536 2 vmw_balloon,vmw_vsock_vmci_transport, Live 0x0000000000000000
> i2c_piix4 24576 0 - Live 0x0000000000000000
> nfit 40960 0 - Live 0x0000000000000000
> 8250_fintek 16384 0 - Live 0x0000000000000000
> parport_pc 32768 0 - Live 0x0000000000000000
> parport 49152 2 ppdev,parport_pc, Live 0x0000000000000000
> mac_hid 16384 0 - Live 0x0000000000000000
> ib_iser 49152 0 - Live 0x0000000000000000
> rdma_cm 53248 1 ib_iser, Live 0x0000000000000000
> iw_cm 49152 1 rdma_cm, Live 0x0000000000000000
> ib_cm 45056 1 rdma_cm, Live 0x0000000000000000
> ib_sa 36864 2 rdma_cm,ib_cm, Live 0x0000000000000000
> ib_mad 49152 2 ib_cm,ib_sa, Live 0x0000000000000000
> ib_core 122880 6 ib_iser,rdma_cm,iw_cm,ib_cm,ib_sa,ib_mad, Live 0x0000000000000000
> ib_addr 20480 3 rdma_cm,ib_sa,ib_core, Live 0x0000000000000000
> configfs 40960 2 rdma_cm, Live 0x0000000000000000
> iscsi_tcp 20480 0 - Live 0x0000000000000000
> libiscsi_tcp 24576 1 iscsi_tcp, Live 0x0000000000000000
> libiscsi 53248 3 ib_iser,iscsi_tcp,libiscsi_tcp, Live 0x0000000000000000
> scsi_transport_iscsi 98304 4 ib_iser,iscsi_tcp,libiscsi, Live 0x0000000000000000
> autofs4 40960 2 - Live 0x0000000000000000
> btrfs 1024000 0 - Live 0x0000000000000000
> raid10 49152 0 - Live 0x0000000000000000
> raid456 110592 0 - Live 0x0000000000000000
> async_raid6_recov 20480 1 raid456, Live 0x0000000000000000
> async_memcpy 16384 2 raid456,async_raid6_recov, Live 0x0000000000000000
> async_pq 16384 2 raid456,async_raid6_recov, Live 0x0000000000000000
> async_xor 16384 3 raid456,async_raid6_recov,async_pq, Live 0x0000000000000000
> async_tx 16384 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor, Live 0x0000000000000000
> xor 24576 2 btrfs,async_xor, Live 0x0000000000000000
> raid6_pq 102400 4 btrfs,raid456,async_raid6_recov,async_pq, Live 0x0000000000000000
> libcrc32c 16384 1 raid456, Live 0x0000000000000000
> raid1 36864 0 - Live 0x0000000000000000
> raid0 20480 0 - Live 0x0000000000000000
> multipath 16384 0 - Live 0x0000000000000000
> linear 16384 0 - Live 0x0000000000000000
> hid_generic 16384 0 - Live 0x0000000000000000
> usbhid 49152 0 - Live 0x0000000000000000
> hid 122880 2 hid_generic,usbhid, Live 0x0000000000000000
> crct10dif_pclmul 16384 0 - Live 0x0000000000000000
> crc32_pclmul 16384 0 - Live 0x0000000000000000
> ghash_clmulni_intel 16384 0 - Live 0x0000000000000000
> aesni_intel 167936 0 - Live 0x0000000000000000
> aes_x86_64 20480 1 aesni_intel, Live 0x0000000000000000
> lrw 16384 1 aesni_intel, Live 0x0000000000000000
> gf128mul 16384 1 lrw, Live 0x0000000000000000
> glue_helper 16384 1 aesni_intel, Live 0x0000000000000000
> ablk_helper 16384 1 aesni_intel, Live 0x0000000000000000
> cryptd 20480 3 ghash_clmulni_intel,aesni_intel,ablk_helper, Live 0x0000000000000000
> vmwgfx 237568 1 - Live 0x0000000000000000
> ttm 98304 1 vmwgfx, Live 0x0000000000000000
> drm_kms_helper 147456 1 vmwgfx, Live 0x0000000000000000
> syscopyarea 16384 1 drm_kms_helper, Live 0x0000000000000000
> psmouse 131072 0 - Live 0x0000000000000000
> sysfillrect 16384 1 drm_kms_helper, Live 0x0000000000000000
> sysimgblt 16384 1 drm_kms_helper, Live 0x0000000000000000
> fb_sys_fops 16384 1 drm_kms_helper, Live 0x0000000000000000
> drm 364544 4 vmwgfx,ttm,drm_kms_helper, Live 0x0000000000000000
> ahci 36864 2 - Live 0x0000000000000000
> libahci 32768 1 ahci, Live 0x0000000000000000
> e1000 135168 0 - Live 0x0000000000000000
> mptspi 24576 0 - Live 0x0000000000000000
> mptscsih 40960 1 mptspi, Live 0x0000000000000000
> mptbase 102400 2 mptspi,mptscsih, Live 0x0000000000000000
> scsi_transport_spi 32768 1 mptspi, Live 0x0000000000000000
> pata_acpi 16384 0 - Live 0x0000000000000000
> fjes 28672 0 - Live 0x0000000000000000
> 
> [8.6.] SCSI information (from /proc/scsi/scsi)
> Attached devices:
> Host: scsi3 Channel: 00 Id: 00 Lun: 00
>   Vendor: ATA      Model: VMware Virtual S Rev: 0001
>   Type:   Direct-Access                    ANSI  SCSI revision: 05
> Host: scsi4 Channel: 00 Id: 00 Lun: 00
>   Vendor: NECVMWar Model: VMware SATA CD01 Rev: 1.00
>   Type:   CD-ROM                           ANSI  SCSI revision: 05
> Host: scsi5 Channel: 00 Id: 00 Lun: 00
>   Vendor: ATA      Model: VMware Virtual S Rev: 0001
>   Type:   Direct-Access                    ANSI  SCSI revision: 05
> Host: scsi6 Channel: 00 Id: 00 Lun: 00
>   Vendor: ATA      Model: VMware Virtual S Rev: 0001
>   Type:   Direct-Access                    ANSI  SCSI revision: 05
> 
> Best regards,
> 
> James Johnston
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


More information about the dm-crypt mailing list