[prev in list] [next in list] [prev in thread] [next in thread]
List: keycloak-dev
Subject: [keycloak-dev] Storage and migration issues
From: srossillo () smartling ! com (Scott Rossillo)
Date: 2015-07-21 18:40:30
Message-ID: 9F48F64D-CEF6-4657-837F-B3D6A0575E40 () smartling ! com
[Download RAW message or body]
I don?t see a problem with that.
> On Jul 21, 2015, at 12:07 PM, Scott Rossillo <srossillo at smartling.com> wrote:
>
>
> Please do not stick community users without an upgrade path. I know upgrades can be \
> painful but isn?t this part of the reason for using Liquibase? Also consider that \
> some of today?s community users may wish to become paying customers. If we can?t \
> get from 1.x to whatever becomes product, we can?t pay RedHat. :) Besides, keeping \
> the upgrades working keeps the product honest and fosters a more open community of \
> users and contributors.
> IMO, the only real advantage of Mongo over an RDBMS for Keycloak is that it can be \
> easily replicated and sharded. So, in theory, performance and uptime can be \
> improved over a relational DB. However, in practice, some of the RDBMS systems can \
> be set up for serious HA.
> I?ve been reading the threads on compatibility and I?m not so sure I understand why \
> deprecated APIs and SPIs are required to be kept around to support an upgrade? Is \
> that for non RDBMS data stores?
>
> > On Jul 21, 2015, at 11:23 AM, Bill Burke <bburke at redhat.com> wrote:
> >
> > I'm not convinced any of this will make things any better. It just
> > might introduce different problems. We're still going to have to
> > support a relational model for User storage so we're still going to have
> > to deal with the headaches of supporting different RDBMS dbs. I wish we
> > could just drop Mongo. Does Red Hat even support Mongo in product? What
> > are the advantages of storing in Mongo anyways?
> >
> > I do agree that its a pain that at least one of these model backends
> > seems to break all the time. Our migration and import/export facilities
> > are not well tested.
> >
> > * Representations are just going to always be different than the data
> > model. This could be because of REST API backwards compatibility and
> > usability. There also seems to be this mandate that exported datamodels
> > must be human creatable and readable. This means referencing everything
> > by name instead of ID and/or creating more hierarchical structures that
> > look nice to humans. All of which don't map very well to a functional
> > datastore.
> > * The way we do caching gives us a lot of flexibility. We can have
> > specific caching logic for various parts of the data model. All of
> > which is datastore type agnostic.
> > * A lot of users are using UserSession RDBMS store because they can't
> > use, or don't know how to use Infinispan in the cloud.
> > * I don't think users are going to want to manage 2 different types of
> > databases, i.e. Mongo + RDBMS.
> > * User databases need to be queryable in all sorts of ways. This pretty
> > much requires a relational model.
> > * Would client databases need to be queryable?
> >
> >
> > I do not agree at all that we need to be able to migrate any old version
> > of community Keycloak. I want 1st version of product to be clean and
> > not have tons of baggage from deprecated and obsolete APIs, SPIs, and
> > data models. Keycloak will be close to 3 years old when we go to
> > product. It will have changed a lot. Sure, we should support migrating
> > between any version of product because thats one of the things people
> > pay us for. But community?
> >
> > It just seems like our queue is so big and the benefits of this are
> > questionable.
> >
> > On 7/21/2015 2:50 AM, Stian Thorgersen wrote:
> > > Maintaining the storage providers are a incredible time sync and very error \
> > > prone. On top of that we also have migration to deal with. It would be worth to \
> > > improve this before we go to product.
> > > Why is storage so horrible?:
> > >
> > > * Multiple implementations - JPA + Mongo + Representations + Cache
> > > * Multiple database providers - every time we add anything it seems to break on \
> > > at least one database
> > > * Migration - we have 4 different types of migration to deal with. \
> > > Representations, JPA, Mongo and the generic one as well
> > > Not only do we spend a lot of time even adding the simplest thing to the model, \
> > > but we also spend a lot of time getting it to work on all supported databases \
> > > before release.
> > > This all made me think that we have to many options! I don't think users care \
> > > how we store realm and client meta-data. They may care about users though. With \
> > > that regards I think we should have a single realm store that is managed by \
> > > Keycloak instead of supporting an external store. For users we'd still have the \
> > > option to use JPA or Mongo, but that's a significantly simpler model, in fact I \
> > > don't think we've barely touched it for a long time. We could also use the same \
> > > managed store for users OOTB.
> > > For the realm store I'd like to use either a document store or a key-value \
> > > store. Relational databases are just to much of a pain to get to work, \
> > > migration is painful and it makes users expect that the database of their \
> > > choice will work as well.
> > > Alternatives could be:
> > >
> > > * Cassandra
> > > * MongoDB
> > > * Plain files - we'd have a revision number of a realm or client and we'd \
> > > replicate the details with Infinispan and store on all nodes
> > > * Infinispan store - Infinispan has built-in support for persisting data
> > > * DMR - in domain mode WildFly manages atomic updates to all nodes. Not sure if \
> > > standalone.xml etc can deal with the amount of data we have \
> > > though.
> > > * Continue storing in any store, but store a JSON blob for realms and clients. \
> > > In relational database we'd have two tables, realms and clients, with an id, \
> > > revision and blob
> > > With regards to migration I think we DO need to support updating from any \
> > > version. In community this is an incredibly nice to have, but could also be \
> > > more than that if a user is stuck on an old version and can't update to the \
> > > intermediate version due to a bug. However, in product it's going to be \
> > > required. Think about it a major release is supported for 8 years or something. \
> > > We could end up with having 4 or more major versions out there at any given \
> > > time. We can't expect a customer that's been on product v1 to upgrade to v2, \
> > > then v3 to be able to upgrade to v4. \
> > > _______________________________________________ keycloak-dev mailing list
> > > keycloak-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> > >
> >
> > --
> > Bill Burke
> > JBoss, a division of Red Hat
> > http://bill.burkecentral.com
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic