[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openldap-devel
Subject:    Dynamically managed groups, etc
From:       Howard Chu <hyc () symas ! com>
Date:       2024-02-26 15:54:56
Message-ID: 3f787c5f-140f-31ae-2d37-53faaa26693c () symas ! com
[Download RAW message or body]

The recent work on expanding dynamic group functionality in the dynlist overlay seems to have been
a bad idea. It makes an already fairly complex overlay even more complicated, and it puts a lot more
work into the read side of operations, which adds up to quite noticeable slowdowns in search performance
on deployments that make heavy use of dynamic groups with large memberships.

It appears that in these situations, the approach used in the autogroup overlay (which has been
in contrib since 2007) is better. It moves all of the membership management into the write side
of operations, updating memberships whenever dynamic group definitions are modified, or when
member entries are written/updated/etc. As such, it allows dynamically defined group memberships
to be read/searched at full speed, as if they were static groups. The search performance difference
between autogroup and the dynlist approach is pretty drastic when large groups and large numbers
of groups are in use. As such I believe autogroup should be recommended, going forward, and we
should move it from contrib into the core code.

For similar reasons, dynamically managing memberOf attributes in dynlist also has a major
impact on search performance. And as alluded to before, adding that functionality to dynlist has
made the code quite a bit more complex. Again for performance reasons, it's better to manage
this attribute on the write side, updating a real attribute in the DB when groups and memberships
are modified, instead of doing the lookup work on the read/search side. As such, we should
reverse the decision to deprecate the memberof overlay. There were a couple problems that the
overlay presented in replication environments before, that prompted us to deprecate it, but I
believe those problems have now been resolved. The first one, referenced in ITS#7400, had to do
with the actual memberof attributes getting replicated even though they weren't meant to be.
The solution for that should have been to specify that memberof should be excluded from replication
using "exattr" in a syncrepl consumer config. The exattr option wasn't behaving as intended in
the past, and using it would cause data desyncs, but that problem was fixed long ago.

The other problem was simply due to the lack of ordering guarantees in syncrepl refreshes,
which prevented the memberof overlay from updating an member's memberof attribute if the group
entry got replicated before the member entry. That problem has been solved in ITS#10167, simply
adding a check whenever new entries are added, to see if they're already members of any existing groups.

As such, the memberof overlay should be perfectly fine for use in replication scenarios now.
More testing of those scenarios is welcome.

With statically managed member and memberof attributes, the major hits to search performance have
been removed. We're still trying dynamically managed nesting of groups though, which is what the
new nestgroup overlay (ITS#10161) is for. Again, we need testing to see how much performance impact
there is from this much-reduced overhead in the read/search side of things. The config is also a
lot easier to understand in its own overlay, instead of shoe-horned into the dynlist config.

Testing and feedback appreciated.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
[prev in list] [next in list] [prev in thread] [next in thread]