[dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

Ryan Lynch ryan.b.lynch at gmail.com
Sun Apr 25 19:48:10 CEST 2010

On Sun, Apr 25, 2010 at 05:38, Milan Broz <mbroz at redhat.com> wrote:
> short summary because there is mix of problems and dangerous ideas :-)

Milan, first of all, thank you for taking the time to help me
understand this. It's intimidating for me to approach a complex topic
like this, from a cold start. Your input is invaluable.

> Multiple problems here, but probably the most important is that
> system see two devices - managing cache separately, so you can
> get data corruption - one device see old data, (f)sync will not help here.

So, if I under stand you correctly, the problem is with *READS FROM*
invalidated cache, not writes to cache that don't hit the disk? Let me
see if I get this right:

    - I have an unmapped encrypted volume, /dev/cipher.
    - I create two identical dm-crypt mappings, one as /dev/plainA,
the other as /dev/plainB.
    - I read the first 512 bytes of /dev/plainA, which caches
/dev/cipher's bytes 0-512 in the /dev/plainA-specific block buffer.
    - I write the first 512 bytes of /dev/plainB, using the direct-io
flag, which alters /dev/cipher's bytes 0-512, but /dev/plainA's block
buffer doesn't get notified that its cache is now invalid.
    - I read the first 512 bytes of /dev/plainA, again, and see the
unchanged invalid cache contents instead of the updated contents on
the underlying /dev/cipher.

I understand how O_DIRECT affects write operations. But I can't find a
clear reference about how it affects reads. Do block device reads
still get buffered, or do read requests always go directly to the
underlying device? In the above example, if I passed the the 'direct'
or 'cio' flags to 'dd' for the read operations, would I be reading
from the cache or directly from underlying device?

As for a cluster FS like GFS2, I believe the lock managers take
responsibility for dropping their invalid read caches when one host
notifies the others of a write operation. Otherwise, how would GFS2
work, at all?


More information about the dm-crypt mailing list