[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openldap-devel
Subject:    LMDB and fsync failures
From:       Howard Chu <hyc () symas ! com>
Date:       2024-02-09 10:09:04
Message-ID: 7cce887c-c1a1-ecf7-7508-c5abf4eec2a8 () symas ! com
[Download RAW message or body]

If anyone remembers fsync-gate https://danluu.com/fsyncgate/ which showed a lot of \
vulnerabilities in other popular DBMSs, some other research was published on the \
topic as well  https://www.usenix.org/conference/atc20/presentation/rebello

I originally discussed this on twitter back in 2020 but wanted to summarize again \
here.

As usual with these types of reports, there are a lot of flaws in their test \
methodology, which invalidates some of their conclusions.

In particular, I question the validity of the failure scenarios their CuttleFS \
simulator produces. Specifically, they claim that multiple systems exhibit False \
Failures after fsync reports a failure, but actually (partially) succeeded. In the \
case of LMDB, where a 1-page synchronous write is involved, this is just an invalid \
test.

They assume that the relevant sector that LMDB cares about is successfully written, \
but an I/O error occurs on some other sector in the page. And so while LMDB \
invalidates the commit in memory, a cache flush and subsequent page-in will read the \
updated sector. But in the real world, if there are hard I/O errors on these other \
sectors, they will most likely also be unreadable, and a subsequent page-in will also \
fail. So at least for LMDB, there would be no false failure.

The failure modes they're modeling don't reflect reality.

Leaving that issue aside, there's also the point that modern storage devices are now \
using 4KB sectors, and still guarantee atomic sector writes, so the partial success \
scenario they describe can't even happen. This is a bunch of academic speculation, \
with a total absence of real world modeling to validate the failure scenarios they \
presented.

The other failures they report, on ext4fs with journaled data, are certainly \
disturbing. But we always recommend turning that journaling off with LMDB; it's \
redundant with LMDB's own COW strategy and harms perf for no benefit.

Of course, you don't even need to trust the filesystem, you can just use LMDB on a \
raw block device.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/


[prev in list] [next in list] [prev in thread] [next in thread]