'Re: apr_dbm and concurrency'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       apr-dev
Subject:    Re: apr_dbm and concurrency
From:       Branko_Čibej <brane () apache ! org>
Date:       2023-09-29 11:35:30
Message-ID: c38214af-a8ff-4dc2-8943-92f1bdabcf4b () apache ! org
[Download RAW message or body]

On 27.09.2023 14:37, Joe Orton wrote:
> On Mon, Sep 25, 2023 at 05:37:00PM +0200, Branko Čibej wrote:
>> IIRC, Berkeley DB multi-process concurrency is managed through an on-disk
>> "register" file external to the actual key/value store. The key/value store
>> format is not affected by the presence of this file. The DB_REGISTER
>> mechanism was introduced in BDB 4.4 (now long defunct) and can be used for
>> both concurrency control and automatic database recovery. The client-side
>> code for this can be lifted from Subversion.
>>
>> (I was involved in designing this mechanism for BDB and implementing its use
>> in Subversion, but that was ages ago -- back in 2005. There may be better
>> ways do do this in newer versions of Berkeley DB).
>>
>> TL;DR: all upstream supported versions of BDB should have this mechanism
>> available and APR can detect if it's being used without changing the API,
>> and even "upgrade" existing databases with the register file on the fly
>> without affecting the actual database.
> Thanks a lot for the response Branko, this is helpful.
>
> I spent a lot more time playing with this and I can get apr_dbm to use a
> DB_ENV in the way you suggest with locking. However, the path handling
> becomes very complicated/impossible - with a DB_ENV the database is
> assumed to be inside the "db_home" directory but we want to load/save a
> database by absolute path name.
>
> Using BDB in this way also appears incompatible with apr_dbm since the
> _usednames API assumes that at most two state files are used and with a
> full BDB environment the state is a whole directory. (mod_dav relies on
> this feature working correctly)
>
> So I don't want to appear like I'm shrugging this off but *at best* it
> might be possible to fix BDB for some versions (possibly, in theory,
> with compat concerns), and that leaves older BDB releases plus NDBM
> still broken.
>
> As an httpd developer I really don't care about database internals, I
> just want this to work. I also know that Berkeley DB is deprecated and
> will be removed from my Linux distribution of choice, so I don't want to
> invest too much time in it. :(

Yes, the whole environment is a directory, not a file. Also we should 
call it "Oracle Berkeley DB", as it was gobbled up shortly after the 
register mechanism was made available...

Given all of the above it would be better to remove support from BDB 
from httpd, or at least mark it "there but don't use unless you're 
exceptionally adventurous."

-- Brane
[Attachment #3 (text/html)]

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">On 27.09.2023 14:37, Joe Orton wrote:<br>
    </div>
    <blockquote type="cite" cite="mid:ZRQiFNVs2Co3mHir@redhat.com">
      <pre class="moz-quote-pre" wrap="">On Mon, Sep 25, 2023 at 05:37:00PM +0200, \
Branko Čibej wrote: </pre>
      <blockquote type="cite">
        <pre class="moz-quote-pre" wrap="">IIRC, Berkeley DB multi-process \
concurrency is managed through an on-disk "register" file external to the actual \
key/value store. The key/value store format is not affected by the presence of this \
file. The DB_REGISTER mechanism was introduced in BDB 4.4 (now long defunct) and can \
be used for both concurrency control and automatic database recovery. The client-side
code for this can be lifted from Subversion.

(I was involved in designing this mechanism for BDB and implementing its use
in Subversion, but that was ages ago -- back in 2005. There may be better
ways do do this in newer versions of Berkeley DB).

TL;DR: all upstream supported versions of BDB should have this mechanism
available and APR can detect if it's being used without changing the API,
and even "upgrade" existing databases with the register file on the fly
without affecting the actual database.
</pre>
      </blockquote>
      <pre class="moz-quote-pre" wrap="">
Thanks a lot for the response Branko, this is helpful.

I spent a lot more time playing with this and I can get apr_dbm to use a 
DB_ENV in the way you suggest with locking. However, the path handling 
becomes very complicated/impossible - with a DB_ENV the database is 
assumed to be inside the "db_home" directory but we want to load/save a 
database by absolute path name.

Using BDB in this way also appears incompatible with apr_dbm since the 
_usednames API assumes that at most two state files are used and with a 
full BDB environment the state is a whole directory. (mod_dav relies on 
this feature working correctly)

So I don't want to appear like I'm shrugging this off but *at best* it 
might be possible to fix BDB for some versions (possibly, in theory, 
with compat concerns), and that leaves older BDB releases plus NDBM 
still broken.

As an httpd developer I really don't care about database internals, I 
just want this to work. I also know that Berkeley DB is deprecated and 
will be removed from my Linux distribution of choice, so I don't want to 
invest too much time in it. :(</pre>
    </blockquote>
    <br>
    Yes, the whole environment is a directory, not a file. Also we
    should call it "Oracle Berkeley DB", as it was gobbled up shortly
    after the register mechanism was made available...<br>
    <br>
    Given all of the above it would be better to remove support from BDB
    from httpd, or at least mark it "there but don't use unless you're
    exceptionally adventurous."<br>
    <br>
    -- Brane<br>
  </body>
</html>



[prev in list] [next in list] [prev in thread] [next in thread]
Configure | About | News | Add a list | Sponsored by KoreLogic