[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openldap-devel
Subject:    LMDB and fsync failures
From:       Howard Chu <hyc () symas ! com>
Date:       2024-02-09 10:09:04
Message-ID: 7cce887c-c1a1-ecf7-7508-c5abf4eec2a8 () symas ! com
[Download RAW message or body]

If anyone remembers fsync-gate https://danluu.com/fsyncgate/ which showed a \
lot of vulnerabilities in other popular DBMSs, some other research was \
published on the topic as well  \
https://www.usenix.org/conference/atc20/presentation/rebello

I originally discussed this on twitter back in 2020 but wanted to summarize \
again here.

As usual with these types of reports, there are a lot of flaws in their \
test methodology, which invalidates some of their conclusions.

In particular, I question the validity of the failure scenarios their \
CuttleFS simulator produces. Specifically, they claim that multiple systems \
exhibit False Failures after fsync reports a failure, but actually \
(partially) succeeded. In the case of LMDB, where a 1-page synchronous \
write is involved, this is just an invalid test.

They assume that the relevant sector that LMDB cares about is successfully \
written, but an I/O error occurs on some other sector in the page. And so \
while LMDB invalidates the commit in memory, a cache flush and subsequent \
page-in will read the updated sector. But in the real world, if there are \
hard I/O errors on these other sectors, they will most likely also be \
unreadable, and a subsequent page-in will also fail. So at least for LMDB, \
there would be no false failure.

The failure modes they're modeling don't reflect reality.

Leaving that issue aside, there's also the point that modern storage \
devices are now using 4KB sectors, and still guarantee atomic sector \
writes, so the partial success scenario they describe can't even happen. \
This is a bunch of academic speculation, with a total absence of real world \
modeling to validate the failure scenarios they presented.

The other failures they report, on ext4fs with journaled data, are \
certainly disturbing. But we always recommend turning that journaling off \
with LMDB; it's redundant with LMDB's own COW strategy and harms perf for \
no benefit.

Of course, you don't even need to trust the filesystem, you can just use \
LMDB on a raw block device.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/


[prev in list] [next in list] [prev in thread] [next in thread]