[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-dev
Subject: Re: Compact Storage and SuperColumn Tables in 4.0/trunk
From: Aleksey Yeshchenko <aleksey () apple ! com>
Date: 2017-09-19 17:58:44
Message-ID: etPan.59c15ad4.14a15d21.3ad () apple ! com
[Download RAW message or body]
4.0 should also fail startup (very early) if it still sees any non-migrated tables, \
probably.
—
AY
On 19 September 2017 at 18:35:11, J. D. Jordan (jeremiah.jordan@gmail.com) wrote:
Thanks for the clarification. +1 for adding a "DROP COMPACT STORAGE" option in 3.x \
and then not allowing it to be specified in 4.0.
On Sep 19, 2017, at 1:27 PM, Alex P <oleksandr.petrov@gmail.com> wrote:
> > If we provide a way to drop the flag, but still access the data, I think that is \
> > fine and perfectly reasonable. If the proposal here is that users who have data \
> > in COMPACT STORAGE tables have no way to upgrade to 4.0 and still access that \
> > data without exporting it to a brand new table, then I am against it. Can you \
> > clarify which thing is being proposed? It is not clear to me.
>
>
> A bit of details on how compact storage and all thrift tables are implemented:
>
> When a table is created through thrift or with COMPACT STORAGE flag, it has a \
> `value` column, which is invisible when doing any CQL queries and only seen through \
> Thrift. With SuperColumn families, internally (on the storage level) have a \
> partition key, clustering and a value column that has a type of `map<>`. Thrift \
> exposes it as a "normal" super column family. CQL, however, does not expose this \
> `map` column. Instead, it translates the key of the map into the second clustering \
> and a map value as a regular column.
> All of this requires quite some special-casing everywhere across the CQL layer, in \
> order to hide/show and translate the columns depending on whether the table is \
> dense, super and so on.
>
>
> For more details you can take a look at 8099 or 12373.
>
> In short: dropping a COMPACT STORAGE flag means that your tables will be accessible \
> and their internal representation (e.g. hidden value column) will be exposed as if \
> it was a normal column. No data will be lost, no data will be inaccessible. You can \
> take a look at the details of CASSANDRA-10857 if you want more details.
> As regards SuperColumn families, my proposal is to have a 100% support in 3.0/3.11 \
> (LWTs, counters, all sorts of queries, exactly like they were accessible through \
> CQL in 2.2).
> There will be a clear upgrade path, but I suggest that the DROP COMPACT STORAGE has \
> to be in 3.x only.
> 4.x will still make the same data available, but expose the whole internal CQL \
> structure, together with a usually "hidden" compact value column, without any \
> legacy special-casing.
>
>
>
>
>
>
> > On 19 Sep 2017, at 17:23, Jeremiah D Jordan <jeremiah.jordan@gmail.com> wrote:
> >
> > I think that all the work to support Compact Storage tables from CQL seems like \
> > wasted effort if we are going to tell people "just kidding, you have to migrate \
> > all your data". I do not think supporting "COMPACT STORAGE" as a table option \
> > matters one way or the other. But I do think being able to read the data that was \
> > in a table created that way is something we need to have a path forward for.
> > > since thrift is not supported on trunk/4.0, it makes it much less appealing or \
> > > even necessary
> >
> > I think that the fact thrift is not supported on trunk/4.0 makes accessing said \
> > data from CQL *MORE* necessary and appealing.
> > > possibility drop a Compact Storage flag and expose them as "normal" tables, \
> > > there was an idea of removing the Compact Tables from 4.x altogether.
> >
> > If we provide a way to drop the flag, but still access the data, I think that is \
> > fine and perfectly reasonable. If the proposal here is that users who have data \
> > in COMPACT STORAGE tables have no way to upgrade to 4.0 and still access that \
> > data without exporting it to a brand new table, then I am against it. Can you \
> > clarify which thing is being proposed? It is not clear to me.
> > -Jeremiah
> >
> >
> > > On Sep 19, 2017, at 7:10 AM, Oleksandr Petrov <oleksandr.petrov@gmail.com \
> > > <mailto:oleksandr.petrov@gmail.com>> wrote:
> > > As you may know, SuperColumn Tables did not work in 3.x the way they worked in \
> > > 2.x. In order to provide everyone with a reasonable upgrade path, we've been \
> > > working on CASSANDRA-12373[1], that brings in support for SuperColumn tables as \
> > > close to 2.x as possible. The patch is planned to land cassandra-3.0 and \
> > > cassandra-3.11 branches only, since the patch for trunk will require even more \
> > > work and, since thrift is not supported on trunk/4.0, it makes it much less \
> > > appealing or even necessary. The idea behind the support for SuperColumns was \
> > > always only to allow people to smoothly migrate off them in 3.0/3.11 world, not \
> > > to have them as a primary feature.
> > > SuperColumns are not the only type of Compact Table, there are more. After \
> > > CASSANDRA-8099[2], Compact Tables are special-cased and have special schema \
> > > layout with some columns hidden from CQL, that allows them to be used from \
> > > Thrift. But, except for the fact they're accessible from Thrift, there are no \
> > > advantages to use them with the new storage. In order to allow people to \
> > > "expose" the internal structure of the compact tables to make them fully \
> > > accessible in CQL, CASSANDRA-10857[3] was created.
> > > In the light of the fact that 4.0 will not have reasonable SuperColumn support \
> > > (due to related complexity and amount of special-cases required to support it \
> > > in 4.0) and a possibility drop a Compact Storage flag and expose them as \
> > > "normal" tables, there was an idea of removing the Compact Tables from 4.x \
> > > altogether.
> > >
> > > Leaving Compact Storage in 3.x only will make the table metadata a bit lighter \
> > > and allow us to remove some special cases required for their support. Doing it \
> > > during the major release, provided with a reasonable upgrade path (same \
> > > functionality from both Thrift and CQL for all compact tables, including Super \
> > > Column ones) through 3.x/3.11, sounds like the best option that we have right \
> > > now.
> > > It'd be good if you could voice your support of this idea (or raise possible \
> > > concerns, if there are any).
> > >
> > > There will be additional discussion and a proposal on how to allow "online" \
> > > COMPACT STORAGE flag drop in CASSANDRA-10857 later this (or the following \
> > > week).
> > > Best Regards,
> > > Alex
> > >
> > > [1] https://issues.apache.org/jira/browse/CASSANDRA-12373 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-12373> \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-12373 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-12373>> [2] \
> > > https://issues.apache.org/jira/browse/CASSANDRA-8099 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-8099> \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-8099 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-8099>> [3] \
> > > https://issues.apache.org/jira/browse/CASSANDRA-10857 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-10857> \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-10857 \
> > > <https://issues.apache.org/jira/browse/CASSANDRA-10857>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org \
> > <mailto:dev-unsubscribe@cassandra.apache.org> For additional commands, e-mail: \
> > dev-help@cassandra.apache.org <mailto:dev-help@cassandra.apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic