[prev in list] [next in list] [prev in thread] [next in thread]
List: postgresql-general
Subject: [HACKERS] On columnar storage (2)
From: Alvaro Herrera <alvherre () 2ndQuadrant ! com>
Date: 2015-08-31 22:53:28
Message-ID: 20150831225328.GM2912 () alvherre ! pgsql
[Download RAW message or body]
As discussed in
https://www.postgresql.org/message-id/20150611230316.GM133018@postgresql.org
we've been working on implementing columnar storage for Postgres.
Here's some initial code to show our general idea, and to gather
comments related to what we're building. This is not a complete patch,
and we don't claim that it works! This is in very early stages, and we
have a lot of work to do to get this in working shape.
This was proposed during the Developer's Unconference in Ottawa earlier
this year. While some questions were raised about some elements of our
design, we don't think they were outright objections, so we have pressed
forward on the expectation that any limitations can be fixed before this
is final if they are critical, or in subsequent commits if not.
The commit messages for each patch should explain what we've done in
enough technical detail, and hopefully provide a high-level overview of
what we're developing.
The first few pieces are "ready for comment" -- feel free to speak up
about the catalog additions, the new COLUMN STORE bits we added to the
grammar, the way we handle column stores in the relcache, or the
mechanics to create column store catalog entries.
The later half of the patch series is much less well cooked yet; for
example, the colstore_dummy module is just a simple experiment to let us
verify that the API is working. The planner and executor code are
mostly stubs, and we are not yet sure of what are the executor nodes
that we would like to have: while we have discussed this topic
internally a lot, we haven't yet formed final opinions, and of course
the stub implementations are not doing the proper things, and in many
cases they are even not doing anything at all.
Still, we believe this shows the general spirit of things, which is that
we would like these new objects be first-class citizens in the Postgres
architecture:
a) so that the optimizer will be able to extract as much benefit as is
possible from columnar storage: it won't be at arms-length through an
opaque interface, but rather directly wired into plans, and have Path
representation eventually.
b) so that it is possible to implement things such as tables that live
completely in columnar storage, as mentioned by Tom regarding Salesforce
extant columnar storage.
Please don't think that the commits attached below represent development
history. We played with the early pieces for quite a while before
settling on what you see here. The presented split is intended to ease
reading. We continue to play with the planner and executor code,
getting ourselves familiar with it enough that we can write something
that actually works.
This patch is joint effort of Tomáš Vondra and myself, with
contributions from Simon Riggs. There's a lot of code attribute to me
in the commit messages that was actually authored by Tomáš. (Git
decided to lay blame on me because I split the commits.)
The research leading to these results has received funding from the
European Union's Seventh Framework Programme (FP7/2007-2015) under grant
agreement n 318633.
--
Álvaro Herrera Developer, http://www.PostgreSQL.org/
["0001-initial-README-for-column-stores.patch" (text/x-diff)]
From 100a12f6bdad1563b40cbcaee63892d20cda6b53 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Sat, 18 Jul 2015 18:58:15 +0200
Subject: [PATCH 01/24] initial README for column stores
---
src/backend/colstore/README | 187 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 187 insertions(+)
create mode 100644 src/backend/colstore/README
diff --git a/src/backend/colstore/README b/src/backend/colstore/README
new file mode 100644
index 0000000..4ab5598
--- /dev/null
+++ b/src/backend/colstore/README
@@ -0,0 +1,187 @@
+Column Store API
+================
+
+The goal of the column store implementations is to allow vertical partitioning
+of tables, with the benefits of
+
+* Increasing storage capacity thanks to better compression ratios, which is
+ possible because the vertical partitioning makes the data more homogenous.
+ This is true both for explicit compression (implemented in the column store)
+ or implicit (implemented at the filesystem level, thus transparent to the
+ database).
+
+* Lower I/O usage, thanks to only reading the 'vertical partitions' necessary
+ for the particular query, and also thanks to compression.
+
+* Improved CPU efficiency thanks to specialized encoding schemes.
+
+* Storage formats optimized for various kind of specialized devices (GPU, FPGA).
+
+The aim of the CS API is not to implement a column store with all those benefits
+(because some of those may be actually conflicting), but providing an API that
+makes it easier to implement a custom columnar storage.
+
+This endorses the extensibility principle, which is one of the core ideas in
+PostgreSQL project.
+
+We do envision the API to be eventually used both internally (for built-in
+column store implementations), and from extensions (with all the limitations
+inherent to code delivered as an extension).
+
+
+CREATE TABLE syntax
+-------------------
+
+A simple CREATE TABLE statement with column store(s) might look like this:
+
+ CREATE TABLE lineitem (
+ l_orderkey BIGINT,
+ l_partkey INTEGER,
+ l_suppkey INTEGER,
+ l_linenumber INTEGER,
+ l_quantity REAL,
+ l_extendedprice REAL,
+ l_discount REAL,
+ l_tax REAL,
+ l_returnflag CHAR(1),
+ l_linestatus CHAR(1),
+ l_shipdate DATE,
+ l_commitdate DATE,
+ l_receiptdate DATE,
+ l_shipinstruct CHAR(25) COLUMN STORE shipinstr USING cstam1 WITH (...),
+ l_shipmode CHAR(10) COLUMN STORE shipmode USING cstam2 WITH (...),
+ l_comment VARCHAR(44) COLUMN STORE comment USING cstam2 WITH (...),
+
+ COLUMN STORE prices
+ USING cstam1 (l_quantity, l_extendedprice, l_discount, l_tax),
+
+ COLUMN STORE dates
+ USING cstam1 (l_shipdate, l_commitdate, l_receiptdate)
+ WITH (compression lzo)
+
+);
+
+If you're familiar with TPC-H benchmark, this table should be familiar to you,
+this is the largest table in that data set. But take this example only as an
+illustration of the syntax, not as a recommendation of how to define the column
+stores in practice.
+
+The example defines a number of column stores - some at column level, some at
+table level. The column stores defined at a column level only contain a single
+column, the stores defined at table level may contain multiple columns. This is
+quite similar to CHECK constraints, for example.
+
+The COLUMN STORE syntax for stores defined at the column level is this:
+
+ COLUMN STORE <name> USING <am> WITH (<options>)
+
+and for stores defined at table level:
+
+ COLUMN STORE <name> USING <am> (<columns>) WITH (<options>)
+
+The <name> is a column store name, unique within a table - once again, this is
+similar to constraints (constraint names are unique in a table).
+
+The <am> stands for 'access method' and references a particular implementation
+of a column store API, listed in pg_cstore_am.
+
+Of course, <columns> is a list of columns in the column store.
+
+This syntax is consistent with indexes, which use the same syntax
+
+ <name> USING <am> (<columns>)
+
+Currently we only allow each column to be assigned to a single column store,
+although this may be changed in the future (allowing overlapping column stores).
+The columns that are not assigned to a column store remain in the heap.
+
+
+Inheritance
+-----------
+TODO
+
+
+Columns Store Handler
+---------------------
+
+To implement a column store, you need a implement an API defined in
+
+ colstore/colstoreapi.h
+
+The design mostly follows the technical ideas from foreign data wrapper API,
+so the API has a form of handler function, returning a pointer to a struct:
+
+ typedef struct ColumnStoreRoutine
+ {
+ NodeTag type;
+
+ /* insert a single row into the column store */
+ ExecColumnStoreInsert_function ExecColumnStoreInsert;
+
+ /* insert a batch of rows into the column store */
+ ExecColumnStoreBatchInsert_function ExecColumnStoreBatchInsert;
+
+ /* fetch values for a single row */
+ ExecColumnStoreFetch_function ExecColumnStoreFetch;
+
+ /* fetch a batch of values for a single row */
+ ExecColumnStoreBatchFetch_function ExecColumnStoreBatchFetch;
+
+ /* discard a batch of deleted rows from the column store */
+ ExecColumnStoreDiscard_function ExecColumnStoreDiscard;
+
+ /* prune the store - keep only the valid rows */
+ ExecColumnStorePrune_function ExecColumnStorePrune;
+
+ } ColumnStoreRoutine;
+
+that implement various tasks for querying, modification and maintenance.
+
+You also need to define a 'handler' which is a function that creates and
+populates the routine structure with pointers to your implementation. This
+function is very simple - takes no arguments and returns a pointer to the
+structure as cstore_handler.
+
+So if the function is called my_colstore_handler(), you may do this:
+
+ CREATE FUNCTION my_colstore_handler()
+ RETURNS cstore_handler
+ AS 'MODULE_PATHNAME'
+ LANGUAGE C STRICT;
+
+to define the handler.
+
+
+Column Store Access Methods
+---------------------------
+
+The column store access method binds a name to a handler (again, this is similar
+to how foreign data wrappers binds name and FDW handler). To define a new access
+method, use
+
+ CREATE COLUMN STORE ACCESS METHOD <name> METHOD <handler>
+
+so using the previously defined handler, you may do
+
+ CREATE COLUMN STORE ACCESS METHOD my_colstore METHOD my_colstore_handler
+
+and then use 'my_colstore' in CREATE TABLE statements.
+
+
+Catalogs
+--------
+- pg_cstore_am - access method (name + handler)
+- pg_cstore - column stores defined for a table (similar to pg_index)
+- pg_class - column stores defined for a table (relations)
+- pg_attribute - attributes for a column store
+
+
+Plan nodes
+----------
+TODO - explain what plan nodes are supported, etc.
+
+
+Limitations
+-----------
+- all column stores have to be defined at CREATE TABLE time
+- each column may belong to heap or to a single column store
--
2.1.4
["0002-New-docs-section-on-Data-Definition-Column-Stores.patch" (text/x-diff)]
From 048991c9fa5179841811a759816697cc6e9749fc Mon Sep 17 00:00:00 2001
From: Simon Riggs <simon@2ndQuadrant.com>
Date: Fri, 21 Aug 2015 10:18:39 +0100
Subject: [PATCH 02/24] New docs section on Data Definition - Column Stores
---
doc/src/sgml/ddl.sgml | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 91 insertions(+)
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index af1bda6..92098e1 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3310,6 +3310,97 @@ ANALYZE measurement;
</sect2>
</sect1>
+ <sect1 id="ddl-column-store">
+ <title>Column Store</title>
+
+ <indexterm>
+ <primary>column store</primary>
+ </indexterm>
+ <indexterm>
+ <primary>columnar storage</primary>
+ </indexterm>
+ <indexterm>
+ <primary>vertical partitioning</primary>
+ </indexterm>
+
+ <para>
+ By default, <productname>PostgreSQL</productname> places all columns
+ of a table together within a single main data structure, referred
+ to internally as the heap.
+ </para>
+
+ <para>
+ Optionally, <productname>PostgreSQL</productname> allows you to
+ specify that columns can be held in secondary data structures,
+ known as column stores. Each column store has a unique name and
+ contains data only for its master table. If a column definition
+ specifies a column store then values of that column for all rows
+ of a table are stored only in the column store - and not within
+ the main heap. This is completely different from TOAST, which is
+ designed to handle oversized attributes by breaking them up into
+ chunks and storing them in a single common subtable for all columns.
+ It is possible to store multiple columns within the same column store.
+ </para>
+
+ <para>
+ With this feature <productname>PostgreSQL</productname> can be described
+ as a hybrid row/column store. <productname>PostgreSQL</productname>
+ implements column stores by providing a flexible column store API,
+ allowing different types of storage to be designed for different datatypes
+ or for different use cases.
+ Column stores are also sometimes referred to as columnar storage or
+ vertical paritioning. This section describes why and how to implement column
+ stores or vertical partitioning as part of your database design.
+ </para>
+
+ <para>
+ Column stores provide the following advantages
+ <itemizedlist>
+ <listitem>
+ <para>
+ By grouping related columns together, data not required for the current
+ query or action will be completely avoided, significantly reducing query
+ time.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ By grouping related columns together it is possible to take advantage of
+ high compression rates, reducing data storage requirements and reducing
+ query times for very large databases.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ By separating data into multiple column stores, datatype-specific
+ storage optimizations will become possible, further extending
+ <productname>PostgreSQL</productname>'s support for custom datatypes.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ By separating data into multiple subtables, it will be possible to
+ store tables that are much larger than the maximum table size for a single
+ heap (32TB).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Reusing storage technology from other projects and/or using technology
+ with different licencing restrictions (GPL licence etc).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Provides an architecture that may eventually allow us to have tables
+ with more than 1600 columns.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+
+ </sect1>
+
<sect1 id="ddl-foreign-data">
<title>Foreign Data</title>
--
2.1.4
["0003-Add-RELKIND_COLUMN_STORE-to-pg_class.h.patch" (text/x-diff)]
From 6af6cbd07863d9b83a10641f55ed024d25c052ed Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 23 Jun 2015 18:03:28 +0200
Subject: [PATCH 03/24] Add RELKIND_COLUMN_STORE to pg_class.h
Column stores are going to have separate pg_class (and pg_attribute)
entries, which are going to be marked with this relkind.
(Also, they are going to have nonzero pg_class.relam, pointing to the
column store access method in use)
---
src/include/catalog/pg_class.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index e526cd9..83bf8f3 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -158,6 +158,7 @@ DESCR("");
#define RELKIND_COMPOSITE_TYPE 'c' /* composite type */
#define RELKIND_FOREIGN_TABLE 'f' /* foreign table */
#define RELKIND_MATVIEW 'm' /* materialized view */
+#define RELKIND_COLUMN_STORE 'C' /* columnar store */
#define RELPERSISTENCE_PERMANENT 'p' /* regular table */
#define RELPERSISTENCE_UNLOGGED 'u' /* unlogged permanent table */
--
2.1.4
["0004-Add-PG_COLSTORE_NAMESPACE-to-pg_namespace.patch" (text/x-diff)]
From 873100928b45b784dbeb28f3f9a7767af3ac3ab0 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 7 Jul 2015 15:56:31 -0300
Subject: [PATCH 04/24] Add PG_COLSTORE_NAMESPACE to pg_namespace
Column stores are going to live in their own pg_colstore schema,
much like TOAST tables live in pg_toast.
---
src/include/catalog/pg_namespace.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/include/catalog/pg_namespace.h b/src/include/catalog/pg_namespace.h
index 2c8887c..d1c55d6 100644
--- a/src/include/catalog/pg_namespace.h
+++ b/src/include/catalog/pg_namespace.h
@@ -75,6 +75,9 @@ DESCR("reserved schema for TOAST tables");
DATA(insert OID = 2200 ( "public" PGUID _null_ ));
DESCR("standard public schema");
#define PG_PUBLIC_NAMESPACE 2200
+DATA(insert OID = 9 ( "pg_colstore" PGUID _null_ ));
+DESCR("reserved schema for column stores");
+#define PG_COLSTORE_NAMESPACE 9
/*
--
2.1.4
["0005-relcache-don-t-consider-nonzero-pg_class.relam-as-an.patch" (text/x-diff)]
From d12280d05fdebd2f35478badc4ce2ea804be8ffe Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Wed, 8 Jul 2015 18:27:30 -0300
Subject: [PATCH 05/24] relcache: don't consider nonzero pg_class.relam as an
index
Right now, only indexes have the "relam" field set to a value different
from zero. Once we introduce other kinds of access methods, that's no
longer the case. In the one test that currently only checks for relam,
add a test for relkind also. (Without this, creating a different kind
of relation with nonzero relam causes weird failures that the pg_index
row cannot be found).
---
src/backend/utils/cache/relcache.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 44e9509..5ca4e2f6 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1051,7 +1051,8 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
/*
* if it's an index, initialize index-related information
*/
- if (OidIsValid(relation->rd_rel->relam))
+ if (relation->rd_rel->relkind == RELKIND_INDEX &&
+ OidIsValid(relation->rd_rel->relam))
RelationInitIndexAccessInfo(relation);
/* extract reloptions if any */
--
2.1.4
["0006-Add-column-store-catalogs.patch" (text/x-diff)]
From 46f2b7e90d6aaad2a052b6cfa9e44221be6def14 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Mon, 23 Mar 2015 13:55:28 +0100
Subject: [PATCH 06/24] Add column store catalogs
This commit adds two new system catalogs:
pg_cstore_am
------------
- Set of functions implementing the column store ("access method")
pg_cstore
---------
- information about a particular column store (part of a table)
- lists columns that are part of each column store
- references pg_cstore_am
Implementation notes
--------------------
- pg_cstore currently has OIDs, but we should remove them as they are
not really necessary. (It's not just a matter of adding the
BKI_WITHOUT_OIDS token though, because the DROP code currently
requires it.)
- pg_cstore_am will go away as soon as we can use pg_am for the task.
---
src/backend/catalog/Makefile | 2 +-
src/include/catalog/indexing.h | 12 ++++++
src/include/catalog/pg_cstore.h | 60 ++++++++++++++++++++++++++++++
src/include/catalog/pg_cstore_am.h | 53 ++++++++++++++++++++++++++
src/test/regress/expected/sanity_check.out | 2 +
5 files changed, 128 insertions(+), 1 deletion(-)
create mode 100644 src/include/catalog/pg_cstore.h
create mode 100644 src/include/catalog/pg_cstore_am.h
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index 25130ec..bdf4e35 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -41,7 +41,7 @@ POSTGRES_BKI_SRCS = $(addprefix \
$(top_srcdir)/src/include/catalog/,\ pg_foreign_data_wrapper.h pg_foreign_server.h \
pg_user_mapping.h \ pg_foreign_table.h pg_policy.h pg_replication_origin.h \
pg_default_acl.h pg_seclabel.h pg_shseclabel.h \
- pg_collation.h pg_range.h pg_transform.h \
+ pg_collation.h pg_range.h pg_transform.h pg_cstore_am.h pg_cstore.h \
toasting.h indexing.h \
)
diff --git a/src/include/catalog/indexing.h b/src/include/catalog/indexing.h
index c38958d..ec6a5a2 100644
--- a/src/include/catalog/indexing.h
+++ b/src/include/catalog/indexing.h
@@ -130,6 +130,18 @@ DECLARE_UNIQUE_INDEX(pg_conversion_name_nsp_index, 2669, on \
pg_conversion using DECLARE_UNIQUE_INDEX(pg_conversion_oid_index, 2670, on \
pg_conversion using btree(oid oid_ops)); #define ConversionOidIndexId 2670
+DECLARE_UNIQUE_INDEX(pg_cstore_oid_index, 3398, on pg_cstore using btree(oid \
oid_ops)); +#define CStoreOidIndexId 3398
+DECLARE_INDEX(pg_cstore_cststoreid_index, 3399, on pg_cstore using btree(cststoreid \
oid_ops)); +#define CStoreStoreOidIndexId 3399
+DECLARE_UNIQUE_INDEX(pg_cstore_cstrelid_cstname_index, 3400, on pg_cstore using \
btree(cstrelid oid_ops, cstname name_ops)); +#define CStoreCstRelidCstnameIndexId \
3400 +
+DECLARE_UNIQUE_INDEX(pg_cstore_am_oid_index, 3394, on pg_cstore_am using btree(oid \
oid_ops)); +#define CStoreAmOidIndexId 3394
+DECLARE_UNIQUE_INDEX(pg_cstore_am_name_index, 3395, on pg_cstore_am using \
btree(cstamname name_ops)); +#define CStoreAmNameIndexId 3395
+
DECLARE_UNIQUE_INDEX(pg_database_datname_index, 2671, on pg_database using \
btree(datname name_ops)); #define DatabaseNameIndexId 2671
DECLARE_UNIQUE_INDEX(pg_database_oid_index, 2672, on pg_database using btree(oid \
oid_ops));
diff --git a/src/include/catalog/pg_cstore.h b/src/include/catalog/pg_cstore.h
new file mode 100644
index 0000000..4ef8d70
--- /dev/null
+++ b/src/include/catalog/pg_cstore.h
@@ -0,0 +1,60 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_cstore.h
+ * definition of column stores - groups of attributes stored in
+ * columnar orientation, along with the relation's initial contents.
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_cstore.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_CSTORE_H
+#define PG_CSTORE_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_cstore definition. cpp turns this into
+ * typedef struct FormData_pg_cstore
+ * ----------------
+ */
+#define CStoreRelationId 3397
+
+CATALOG(pg_cstore,3397)
+{
+ NameData cstname; /* name of the colstore */
+ Oid cstrelid; /* relation containing this cstore */
+ Oid cststoreid; /* pg_class OID of the cstore itself */
+ int16 cstnatts; /* number of attributes in the cstore */
+
+ /* variable-length fields start here, but we allow direct access to cstatts */
+ int2vector cstatts; /* column numbers of cols in this store */
+
+} FormData_pg_cstore;
+
+/* ----------------
+ * Form_pg_cstore corresponds to a pointer to a tuple with
+ * the format of pg_cstore relation.
+ * ----------------
+ */
+typedef FormData_pg_cstore *Form_pg_cstore;
+
+/* ----------------
+ * compiler constants for pg_cstore
+ * ----------------
+ */
+#define Natts_pg_cstore 5
+#define Anum_pg_cstore_cstname 1
+#define Anum_pg_cstore_cstrelid 2
+#define Anum_pg_cstore_cststoreid 3
+#define Anum_pg_cstore_cstnatts 4
+#define Anum_pg_cstore_cstatts 5
+
+#endif /* PG_CSTORE_H */
diff --git a/src/include/catalog/pg_cstore_am.h b/src/include/catalog/pg_cstore_am.h
new file mode 100644
index 0000000..232b0b2
--- /dev/null
+++ b/src/include/catalog/pg_cstore_am.h
@@ -0,0 +1,53 @@
+/*-------------------------------------------------------------------------
+ *
+ * pg_cstore_am.h
+ * definition of the system "cstore access method" relation
+ * (pg_cstore_am) along with the relation's initial contents.
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/catalog/pg_cstore_am.h
+ *
+ * NOTES
+ * the genbki.pl script reads this file and generates .bki
+ * information from the DATA() statements.
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef PG_CSTORE_AM_H
+#define PG_CSTORE_AM_H
+
+#include "catalog/genbki.h"
+
+/* ----------------
+ * pg_cstore_am definition. cpp turns this into
+ * typedef struct FormData_pg_cstore_am
+ * ----------------
+ */
+#define CStoreAmRelationId 3396
+
+CATALOG(pg_cstore_am,3396)
+{
+ NameData cstamname; /* column store am name */
+ Oid cstamhandler; /* handler function */
+
+} FormData_pg_cstore_am;
+
+/* ----------------
+ * Form_pg_cstore_am corresponds to a pointer to a tuple with
+ * the format of pg_cstore_am relation.
+ * ----------------
+ */
+typedef FormData_pg_cstore_am *Form_pg_cstore_am;
+
+/* ----------------
+ * compiler constants for pg_cstore_am
+ * ----------------
+ */
+#define Natts_pg_cstore_am 2
+#define Anum_pg_cstore_am_cstamname 1
+#define Anum_pg_cstore_am_cstamhandler 2
+
+#endif /* PG_CSTORE_AM_H */
diff --git a/src/test/regress/expected/sanity_check.out \
b/src/test/regress/expected/sanity_check.out index eb0bc88..b05061a 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -97,6 +97,8 @@ pg_class|t
pg_collation|t
pg_constraint|t
pg_conversion|t
+pg_cstore|t
+pg_cstore_am|t
pg_database|t
pg_db_role_setting|t
pg_default_acl|t
--
2.1.4
["0007-add-syscaches-for-column-store-catalogs.patch" (text/x-diff)]
From 2a0ef1a4ee7a05bda1e72e6851ae60a23503717d Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 7 Jul 2015 15:53:51 -0300
Subject: [PATCH 07/24] add syscaches for column store catalogs
---
src/backend/utils/cache/syscache.c | 35 +++++++++++++++++++++++++++++++++++
src/include/utils/syscache.h | 3 +++
2 files changed, 38 insertions(+)
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index efce7b9..1c39625 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -32,6 +32,8 @@
#include "catalog/pg_collation.h"
#include "catalog/pg_constraint.h"
#include "catalog/pg_conversion.h"
+#include "catalog/pg_cstore.h"
+#include "catalog/pg_cstore_am.h"
#include "catalog/pg_database.h"
#include "catalog/pg_db_role_setting.h"
#include "catalog/pg_default_acl.h"
@@ -347,6 +349,39 @@ static const struct cachedesc cacheinfo[] = {
},
8
},
+ {CStoreAmRelationId, /* CSTOREAMNAME */
+ CStoreAmNameIndexId,
+ 1,
+ {
+ Anum_pg_cstore_am_cstamname,
+ 0,
+ 0,
+ 0
+ },
+ 8
+ },
+ {CStoreAmRelationId, /* CSTOREAMOID */
+ CStoreAmOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 8
+ },
+ {CStoreRelationId, /* CSTOREOID */
+ CStoreOidIndexId,
+ 1,
+ {
+ ObjectIdAttributeNumber,
+ 0,
+ 0,
+ 0
+ },
+ 16
+ },
{DatabaseRelationId, /* DATABASEOID */
DatabaseOidIndexId,
1,
diff --git a/src/include/utils/syscache.h b/src/include/utils/syscache.h
index 18404e2..abaeb21 100644
--- a/src/include/utils/syscache.h
+++ b/src/include/utils/syscache.h
@@ -52,6 +52,9 @@ enum SysCacheIdentifier
CONNAMENSP,
CONSTROID,
CONVOID,
+ CSTOREAMNAME,
+ CSTOREAMOID,
+ CSTOREOID,
DATABASEOID,
DEFACLROLENSPOBJ,
ENUMOID,
--
2.1.4
["0008-Add-COLUMN-STORE-clause-to-CREATE-TABLE.patch" (text/x-diff)]
From 3602e8ba81b5e457af0487de33d2a7fc5357947f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 21 Apr 2015 15:28:51 -0300
Subject: [PATCH 08/24] Add COLUMN STORE clause to CREATE TABLE
This allows two ways to specify column stores when creating a table.
The first is using column constraint syntax:
CREATE TABLE test (
a INT COLUMN STORE x USING y [WITH (...)] [TABLESPACE t],
b INT COLUMN STORE y USING z [WITH (...)] [TABLESPACE t]
);
which maps each column into its own column store. Note the USING
clause, which specifies the column store access method used for that
particular store.
The alternative syntax piggy-backs on table-level constraints, and
allows column stores comprising more than one column:
CREATE TABLE test (
a INT,
b INT,
COLUMN STORE y USING z (a,b) WITH (...) [TABLESPACE t]
);
Implementation notes:
* Adds "STORE" as an unreserved keyword.
* Adds a new node type, ColumnStoreClause.
* The two forms of column stores are represented separately in the
CreateStmt parse node. The column-level form lives inside ColumnDef;
the table-level form lives in the list of constraints.
* We allow for a "WITH (options)", which is supposed to be passed to the
column store AM. Currently it is not yet used for anything.
---
doc/src/sgml/ref/create_table.sgml | 46 +++++++++++++++++++++++++++++--
src/backend/nodes/copyfuncs.c | 19 +++++++++++++
src/backend/nodes/equalfuncs.c | 17 ++++++++++++
src/backend/nodes/outfuncs.c | 17 ++++++++++++
src/backend/parser/gram.y | 55 ++++++++++++++++++++++++++++++++++++--
src/backend/parser/parse_utilcmd.c | 4 +++
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 36 +++++++++++++++++++++----
src/include/parser/kwlist.h | 1 +
9 files changed, 187 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index fac7e1e..116ba82 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -22,8 +22,9 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS \
] <replaceable class="PARAMETER">table_name</replaceable> ( [
- { <replaceable class="PARAMETER">column_name</replaceable> <replaceable \
class="PARAMETER">data_type</replaceable> [ COLLATE \
<replaceable>collation</replaceable> ] [ <replaceable \
class="PARAMETER">column_constraint</replaceable> [ ... ] ] + { <replaceable \
class="PARAMETER">column_name</replaceable> <replaceable \
class="PARAMETER">data_type</replaceable> [ COLLATE \
<replaceable>collation</replaceable> ] [ <replaceable \
class="PARAMETER">column_constraint</replaceable> [ ... ] ] [ column_storage ] | \
<replaceable>table_constraint</replaceable> + | \
<replaceable>table_level_column_storage</replaceable>
| LIKE <replaceable>source_table</replaceable> [ \
<replaceable>like_option</replaceable> ... ] } [, ... ]
] )
@@ -55,6 +56,14 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] \
TABLE [ IF NOT EXI
[ ON DELETE <replaceable class="parameter">action</replaceable> ] [ ON UPDATE \
<replaceable class="parameter">action</replaceable> ] } [ DEFERRABLE | NOT \
DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
+<phrase>and <replaceable class="PARAMETER">column_store</replaceable> is:</phrase>
+
+[ COLUMN STORE <replaceable class="PARAMETER">column_store_name</replaceable>
+ USING <replaceable class="PARAMETER">column_store_access_method</replaceable>
+ [ WITH ( <replaceable class="PARAMETER">storage_parameter</replaceable>
+ [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] ) ]
+ [ TABLESPACE <replaceable class="PARAMETER">tablespace_name</replaceable> ] ]
+
<phrase>and <replaceable class="PARAMETER">table_constraint</replaceable> \
is:</phrase>
[ CONSTRAINT <replaceable class="PARAMETER">constraint_name</replaceable> ]
@@ -66,6 +75,15 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] \
TABLE [ IF NOT EXI [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE \
<replaceable class="parameter">action</replaceable> ] [ ON UPDATE <replaceable \
class="parameter">action</replaceable> ] } [ DEFERRABLE | NOT DEFERRABLE ] [ \
INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
+<phrase>and <replaceable class="PARAMETER">table_level_column_store</replaceable> \
is:</phrase> +
+[ COLUMN STORE <replaceable class="PARAMETER">column_store_name</replaceable>
+ USING <replaceable class="PARAMETER">column_store_access_method</replaceable>
+ (<replaceable class="PARAMETER">column_name</replaceable> [, ...] )
+ [ WITH ( <replaceable class="PARAMETER">storage_parameter</replaceable>
+ [= <replaceable class="PARAMETER">value</replaceable>] [, ... ] ) ]
+ [ TABLESPACE <replaceable class="PARAMETER">tablespace_name</replaceable> ] ]
+
<phrase>and <replaceable class="PARAMETER">like_option</replaceable> is:</phrase>
{ INCLUDING | EXCLUDING } { DEFAULTS | CONSTRAINTS | INDEXES | STORAGE | COMMENTS | \
ALL } @@ -262,6 +280,28 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | \
UNLOGGED ] TABLE [ IF NOT EXI </varlistentry>
<varlistentry>
+ <term><literal>COLUMN STORE <replaceable>column store \
name</replaceable></literal></term> + <listitem>
+ <para>
+ The <literal>COLUMN STORE</> clause assigns a column to a named
+ column store with a column store access method defined by the
+ compulsory <literal>USING</> clause. Optional parameters may be set
+ in the <literal>WITH</> clause. A tablespace can be defined for each
+ column store by using the <literal>TABLESPACE</> clause, otherwise
+ the column store will use the same tablespace as the main table.
+ </para>
+ <para>
+ Column stores may contain multiple columns. Multi-column column stores
+ must be defined as table-level clauses.
+ </para>
+ <para>
+ Columns may be assigned to only one column store, otherwise they will
+ be stored as part of the main table.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><literal>INHERITS ( <replaceable>parent_table</replaceable> [, ... ] \
)</literal></term> <listitem>
<para>
@@ -307,7 +347,8 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] \
TABLE [ IF NOT EXI </para>
<para>
- Column <literal>STORAGE</> settings are also copied from parent tables.
+ Column <literal>STORAGE</> settings are also copied from parent tables,
+ but not column storage definitions.
</para>
</listitem>
@@ -320,6 +361,7 @@ CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] \
TABLE [ IF NOT EXI The <literal>LIKE</literal> clause specifies a table from which
the new table automatically copies all column names, their data types,
and their not-null constraints.
+ Column storage parameters are not copied by the LIKE clause.
</para>
<para>
Unlike <literal>INHERITS</literal>, the new table and original table
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index bd2e80e..ae7ca30 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -2554,6 +2554,21 @@ _copyCollateClause(const CollateClause *from)
return newnode;
}
+static ColumnStoreClause *
+_copyColumnStoreClause(const ColumnStoreClause *from)
+{
+ ColumnStoreClause *newnode = makeNode(ColumnStoreClause);
+
+ COPY_STRING_FIELD(name);
+ COPY_STRING_FIELD(storetype);
+ COPY_NODE_FIELD(columns);
+ COPY_NODE_FIELD(options);
+ COPY_LOCATION_FIELD(location);
+ COPY_STRING_FIELD(tablespacename);
+
+ return newnode;
+}
+
static IndexElem *
_copyIndexElem(const IndexElem *from)
{
@@ -2586,6 +2601,7 @@ _copyColumnDef(const ColumnDef *from)
COPY_NODE_FIELD(cooked_default);
COPY_NODE_FIELD(collClause);
COPY_SCALAR_FIELD(collOid);
+ COPY_NODE_FIELD(cstoreClause);
COPY_NODE_FIELD(constraints);
COPY_NODE_FIELD(fdwoptions);
COPY_LOCATION_FIELD(location);
@@ -4883,6 +4899,9 @@ copyObject(const void *from)
case T_CollateClause:
retval = _copyCollateClause(from);
break;
+ case T_ColumnStoreClause:
+ retval = _copyColumnStoreClause(from);
+ break;
case T_SortBy:
retval = _copySortBy(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 19412fe..18c2119 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2241,6 +2241,19 @@ _equalCollateClause(const CollateClause *a, const \
CollateClause *b) }
static bool
+_equalColumnStoreClause(const ColumnStoreClause *a, const ColumnStoreClause *b)
+{
+ COMPARE_STRING_FIELD(name);
+ COMPARE_STRING_FIELD(storetype);
+ COMPARE_NODE_FIELD(columns);
+ COMPARE_NODE_FIELD(options);
+ COMPARE_LOCATION_FIELD(location);
+ COMPARE_STRING_FIELD(tablespacename);
+
+ return true;
+}
+
+static bool
_equalSortBy(const SortBy *a, const SortBy *b)
{
COMPARE_NODE_FIELD(node);
@@ -2330,6 +2343,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b)
COMPARE_NODE_FIELD(cooked_default);
COMPARE_NODE_FIELD(collClause);
COMPARE_SCALAR_FIELD(collOid);
+ COMPARE_NODE_FIELD(cstoreClause);
COMPARE_NODE_FIELD(constraints);
COMPARE_NODE_FIELD(fdwoptions);
COMPARE_LOCATION_FIELD(location);
@@ -3240,6 +3254,9 @@ equal(const void *a, const void *b)
case T_CollateClause:
retval = _equalCollateClause(a, b);
break;
+ case T_ColumnStoreClause:
+ retval = _equalColumnStoreClause(a, b);
+ break;
case T_SortBy:
retval = _equalSortBy(a, b);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index a878498..6399d30 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -2279,6 +2279,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node)
WRITE_NODE_FIELD(collClause);
WRITE_OID_FIELD(collOid);
WRITE_NODE_FIELD(constraints);
+ WRITE_NODE_FIELD(cstoreClause);
WRITE_NODE_FIELD(fdwoptions);
WRITE_LOCATION_FIELD(location);
}
@@ -2319,6 +2320,19 @@ _outCollateClause(StringInfo str, const CollateClause *node)
}
static void
+_outColumnStoreClause(StringInfo str, const ColumnStoreClause *node)
+{
+ WRITE_NODE_TYPE("COLUMNSTORECLAUSE");
+
+ WRITE_STRING_FIELD(name);
+ WRITE_STRING_FIELD(storetype);
+ WRITE_NODE_FIELD(columns);
+ WRITE_NODE_FIELD(options);
+ WRITE_LOCATION_FIELD(location);
+ WRITE_STRING_FIELD(tablespacename);
+}
+
+static void
_outIndexElem(StringInfo str, const IndexElem *node)
{
WRITE_NODE_TYPE("INDEXELEM");
@@ -3363,6 +3377,9 @@ _outNode(StringInfo str, const void *obj)
case T_CollateClause:
_outCollateClause(str, obj);
break;
+ case T_ColumnStoreClause:
+ _outColumnStoreClause(str, obj);
+ break;
case T_IndexElem:
_outIndexElem(str, obj);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 1efc6d6..1ec1426 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -171,6 +171,7 @@ static TypeName *TableFuncTypeName(List *columns);
static RangeVar *makeRangeVarFromAnyName(List *names, int position, core_yyscan_t \
yyscanner); static void SplitColQualList(List *qualList,
List **constraintList, CollateClause **collClause,
+ ColumnStoreClause **cstoreClause,
core_yyscan_t yyscanner);
static void processCASbits(int cas_bits, int location, const char *constrType,
bool *deferrable, bool *initdeferred, bool *not_valid,
@@ -501,7 +502,7 @@ static Node *makeRecursiveViewSelect(char *relname, List \
*aliases, Node *query); %type <keyword> unreserved_keyword type_func_name_keyword
%type <keyword> col_name_keyword reserved_keyword
-%type <node> TableConstraint TableLikeClause
+%type <node> TableConstraint TableLikeClause ColStoreClause
%type <ival> TableLikeOptionList TableLikeOption
%type <list> ColQualList
%type <node> ColConstraint ColConstraintElem ConstraintAttr
@@ -627,7 +628,7 @@ static Node *makeRecursiveViewSelect(char *relname, List \
*aliases, Node *query); SAVEPOINT SCHEMA SCROLL SEARCH SECOND_P SECURITY SELECT \
SEQUENCE SEQUENCES SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE \
SHOW SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P START
- STATEMENT STATISTICS STDIN STDOUT STORAGE STRICT_P STRIP_P SUBSTRING
+ STATEMENT STATISTICS STDIN STDOUT STORAGE STORE STRICT_P STRIP_P SUBSTRING
SYMMETRIC SYSID SYSTEM_P
TABLE TABLES TABLESAMPLE TABLESPACE TEMP TEMPLATE TEMPORARY TEXT_P THEN
@@ -2934,11 +2935,13 @@ TableElement:
columnDef { $$ = $1; }
| TableLikeClause { $$ = $1; }
| TableConstraint { $$ = $1; }
+ | ColStoreClause { $$ = $1; }
;
TypedTableElement:
columnOptions { $$ = $1; }
| TableConstraint { $$ = $1; }
+ | ColStoreClause { $$ = $1; }
;
columnDef: ColId Typename create_generic_options ColQualList
@@ -2956,6 +2959,7 @@ columnDef: ColId Typename create_generic_options ColQualList
n->collOid = InvalidOid;
n->fdwoptions = $3;
SplitColQualList($4, &n->constraints, &n->collClause,
+ &n->cstoreClause,
yyscanner);
n->location = @1;
$$ = (Node *)n;
@@ -2976,6 +2980,7 @@ columnOptions: ColId WITH OPTIONS ColQualList
n->cooked_default = NULL;
n->collOid = InvalidOid;
SplitColQualList($4, &n->constraints, &n->collClause,
+ &n->cstoreClause,
yyscanner);
n->location = @1;
$$ = (Node *)n;
@@ -3011,6 +3016,20 @@ ColConstraint:
n->location = @1;
$$ = (Node *) n;
}
+ | COLUMN STORE ColId USING ColId opt_reloptions OptTableSpace
+ {
+ /*
+ * Note: as with COLLATE, this is here only temporarily.
+ */
+ ColumnStoreClause *n = makeNode(ColumnStoreClause);
+ n->name = $3;
+ n->storetype = $5;
+ n->columns = NIL;
+ n->options = $6;
+ n->tablespacename = $7;
+ n->location = @1;
+ $$ = (Node *) n;
+ }
;
/* DEFAULT NULL is already the default for Postgres.
@@ -3190,6 +3209,19 @@ TableConstraint:
| ConstraintElem { $$ = $1; }
;
+ColStoreClause:
+ COLUMN STORE ColId USING ColId '(' columnList ')' opt_reloptions OptTableSpace
+ {
+ ColumnStoreClause *n = makeNode(ColumnStoreClause);
+ n->name = $3;
+ n->storetype = $5;
+ n->columns = $7;
+ n->options = $9;
+ n->tablespacename = $10;
+ $$ = (Node *) n;
+ }
+ ;
+
ConstraintElem:
CHECK '(' a_expr ')' ConstraintAttributeSpec
{
@@ -8972,6 +9004,7 @@ CreateDomainStmt:
n->domainname = $3;
n->typeName = $5;
SplitColQualList($6, &n->constraints, &n->collClause,
+ NULL,
yyscanner);
$$ = (Node *)n;
}
@@ -13846,6 +13879,7 @@ unreserved_keyword:
| STDIN
| STDOUT
| STORAGE
+ | STORE
| STRICT_P
| STRIP_P
| SYSID
@@ -14706,6 +14740,7 @@ makeRangeVarFromAnyName(List *names, int position, \
core_yyscan_t yyscanner) static void
SplitColQualList(List *qualList,
List **constraintList, CollateClause **collClause,
+ ColumnStoreClause **cstoreClause,
core_yyscan_t yyscanner)
{
ListCell *cell;
@@ -14736,6 +14771,22 @@ SplitColQualList(List *qualList,
parser_errposition(c->location)));
*collClause = c;
}
+ else if (IsA(n, ColumnStoreClause))
+ {
+ ColumnStoreClause *c = (ColumnStoreClause *) n;
+
+ if (cstoreClause == NULL)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("COLUMN STORE clause not allowed here"),
+ parser_errposition(c->location)));
+ if (*cstoreClause)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("multiple COLUMN STORE clauses not allowed"),
+ parser_errposition(c->location)));
+ *cstoreClause = c;
+ }
else
elog(ERROR, "unexpected node type %d", (int) n->type);
/* remove non-Constraint nodes from qualList */
diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c
index 16d40c7..562eff9 100644
--- a/src/backend/parser/parse_utilcmd.c
+++ b/src/backend/parser/parse_utilcmd.c
@@ -262,6 +262,10 @@ transformCreateStmt(CreateStmt *stmt, const char *queryString)
transformTableLikeClause(&cxt, (TableLikeClause *) element);
break;
+ case T_ColumnStoreClause:
+ stmt->colstores = lappend(stmt->colstores, element);
+ break;
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(element));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 748e434..27b38b1 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -396,6 +396,7 @@ typedef enum NodeTag
T_MultiAssignRef,
T_TypeCast,
T_CollateClause,
+ T_ColumnStoreClause,
T_SortBy,
T_WindowDef,
T_RangeSubselect,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index f0dcd2f..302ba1f 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -558,6 +558,24 @@ typedef struct RangeTableSample
} RangeTableSample;
/*
+ * ColumnStoreClause - a COLUMN STORE declaration
+ *
+ * When this appears as a column constraint, the store includes only that
+ * column, and the column list is NIL. When it appears as a table constraint,
+ * the column list contains one or more column names.
+ */
+typedef struct ColumnStoreClause
+{
+ NodeTag type;
+ char *name; /* column store name */
+ char *storetype; /* column store type */
+ List *columns; /* list of String column names, or NIL */
+ List *options; /* list of DefElem options */
+ char *tablespacename; /* table space to use, or NULL */
+ int location;
+} ColumnStoreClause;
+
+/*
* ColumnDef - column definition (used in various creates)
*
* If the column has a default value, we may have the value expression
@@ -570,6 +588,10 @@ typedef struct RangeTableSample
* (represented as a CollateClause with arg==NULL) or cooked form
* (the collation's OID).
*
+ * Similarly, a COLUMN STORE specification may come in raw form (a
+ * ColumnStoreClause) or cooked (the OID of the corresponding store in the
+ * parent relation. Note we only reuse its name, not the store itself.)
+ *
* The constraints list may contain a CONSTR_DEFAULT item in a raw
* parsetree produced by gram.y, but transformCreateStmt will remove
* the item and set raw_default instead. CONSTR_DEFAULT items
@@ -589,6 +611,7 @@ typedef struct ColumnDef
Node *cooked_default; /* default value (transformed expr tree) */
CollateClause *collClause; /* untransformed COLLATE spec, if any */
Oid collOid; /* collation OID (InvalidOid if not set) */
+ ColumnStoreClause *cstoreClause; /* untransformed COL STORE, if any */
List *constraints; /* other constraints on column */
List *fdwoptions; /* per-column FDW options */
int location; /* parse location, or -1 if none/unknown */
@@ -1723,11 +1746,13 @@ typedef struct VariableShowStmt
/* ----------------------
* Create Table Statement
*
- * NOTE: in the raw gram.y output, ColumnDef and Constraint nodes are
- * intermixed in tableElts, and constraints is NIL. After parse analysis,
- * tableElts contains just ColumnDefs, and constraints contains just
- * Constraint nodes (in fact, only CONSTR_CHECK nodes, in the present
- * implementation).
+ * NOTE: in the raw gram.y output, ColumnDef, ColumnStoreClause and Constraint
+ * nodes are intermixed in tableElts, and constraints and colstores are NIL.
+ * After parse analysis, tableElts contains just ColumnDefs, constraints
+ * contains just Constraint nodes (in fact, only CONSTR_CHECK nodes, in the
+ * present implementation), and colstores contains only ColumnStoreClause
+ * nodes (further note that ColumnDef nodes might have ColumnStoreClause nodes
+ * within. Those are left alone during parse analysis.)
* ----------------------
*/
@@ -1740,6 +1765,7 @@ typedef struct CreateStmt
* inhRelation) */
TypeName *ofTypename; /* OF typename */
List *constraints; /* constraints (list of Constraint nodes) */
+ List *colstores; /* list of ColumnStoreClause nodes */
List *options; /* options from WITH clause */
OnCommitAction oncommit; /* what do we do at COMMIT? */
char *tablespacename; /* table space to use, or NULL */
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 2414069..3d499c1 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -364,6 +364,7 @@ PG_KEYWORD("statistics", STATISTICS, UNRESERVED_KEYWORD)
PG_KEYWORD("stdin", STDIN, UNRESERVED_KEYWORD)
PG_KEYWORD("stdout", STDOUT, UNRESERVED_KEYWORD)
PG_KEYWORD("storage", STORAGE, UNRESERVED_KEYWORD)
+PG_KEYWORD("store", STORE, UNRESERVED_KEYWORD)
PG_KEYWORD("strict", STRICT_P, UNRESERVED_KEYWORD)
PG_KEYWORD("strip", STRIP_P, UNRESERVED_KEYWORD)
PG_KEYWORD("substring", SUBSTRING, COL_NAME_KEYWORD)
--
2.1.4
["0009-add-pg_class.relhascstore.patch" (text/x-diff)]
From f220720dd85f8eeea45291f3c419662ed79754be Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 28 Aug 2015 15:06:34 -0300
Subject: [PATCH 09/24] add pg_class.relhascstore
This is a close parallel to relhasindex for indexes.
---
src/backend/catalog/heap.c | 1 +
src/backend/catalog/system_views.sql | 2 +
src/backend/rewrite/rewriteDefine.c | 7 +
src/backend/utils/cache/relcache.c | 5 +-
src/include/catalog/pg_class.h | 46 ++--
src/test/regress/expected/rules.out | 2 +
src/test/regress/expected/sanity_check.out | 326 ++++++++++++++---------------
src/test/regress/sql/sanity_check.sql | 2 +-
8 files changed, 204 insertions(+), 187 deletions(-)
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index d04e94d..2a6c576 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -792,6 +792,7 @@ InsertPgClassTuple(Relation pg_class_desc,
values[Anum_pg_class_relallvisible - 1] = Int32GetDatum(rd_rel->relallvisible);
values[Anum_pg_class_reltoastrelid - 1] = ObjectIdGetDatum(rd_rel->reltoastrelid);
values[Anum_pg_class_relhasindex - 1] = BoolGetDatum(rd_rel->relhasindex);
+ values[Anum_pg_class_relhascstore - 1] = BoolGetDatum(rd_rel->relhascstore);
values[Anum_pg_class_relisshared - 1] = BoolGetDatum(rd_rel->relisshared);
values[Anum_pg_class_relpersistence - 1] = CharGetDatum(rd_rel->relpersistence);
values[Anum_pg_class_relkind - 1] = CharGetDatum(rd_rel->relkind);
diff --git a/src/backend/catalog/system_views.sql \
b/src/backend/catalog/system_views.sql index ccc030f..4234f6f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -117,6 +117,7 @@ CREATE VIEW pg_tables AS
pg_get_userbyid(C.relowner) AS tableowner,
T.spcname AS tablespace,
C.relhasindex AS hasindexes,
+ C.relhascstore AS hascstores,
C.relhasrules AS hasrules,
C.relhastriggers AS hastriggers,
C.relrowsecurity AS rowsecurity
@@ -131,6 +132,7 @@ CREATE VIEW pg_matviews AS
pg_get_userbyid(C.relowner) AS matviewowner,
T.spcname AS tablespace,
C.relhasindex AS hasindexes,
+ C.relhascstore AS hascstores,
C.relispopulated AS ispopulated,
pg_get_viewdef(C.oid) AS definition
FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
diff --git a/src/backend/rewrite/rewriteDefine.c \
b/src/backend/rewrite/rewriteDefine.c index 39c83a6..3c5dd5e 100644
--- a/src/backend/rewrite/rewriteDefine.c
+++ b/src/backend/rewrite/rewriteDefine.c
@@ -447,6 +447,12 @@ DefineQueryRewrite(char *rulename,
errmsg("could not convert table \"%s\" to a view because it has indexes",
RelationGetRelationName(event_relation))));
+ if (event_relation->rd_rel->relhascstore)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("could not convert table \"%s\" to a view because it has column \
stores", + RelationGetRelationName(event_relation))));
+
if (event_relation->rd_rel->relhassubclass)
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -606,6 +612,7 @@ DefineQueryRewrite(char *rulename,
classForm->relallvisible = 0;
classForm->reltoastrelid = InvalidOid;
classForm->relhasindex = false;
+ classForm->relhascstore = false;
classForm->relkind = RELKIND_VIEW;
classForm->relhasoids = false;
classForm->relhaspkey = false;
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 5ca4e2f6..ddb1fc0 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -1691,7 +1691,7 @@ formrdesc(const char *relationName, Oid relationReltype,
*/
if (IsBootstrapProcessingMode())
{
- /* In bootstrap mode, we have no indexes */
+ /* In bootstrap mode, we have no indexes or column stores */
relation->rd_rel->relhasindex = false;
}
else
@@ -1700,6 +1700,9 @@ formrdesc(const char *relationName, Oid relationReltype,
relation->rd_rel->relhasindex = true;
}
+ /* this is used for catalogs only, so no column stores */
+ relation->rd_rel->relhascstore = false;
+
/*
* add new reldesc to relcache
*/
diff --git a/src/include/catalog/pg_class.h b/src/include/catalog/pg_class.h
index 83bf8f3..310b51b 100644
--- a/src/include/catalog/pg_class.h
+++ b/src/include/catalog/pg_class.h
@@ -49,6 +49,7 @@ CATALOG(pg_class,1259) BKI_BOOTSTRAP BKI_ROWTYPE_OID(83) \
BKI_SCHEMA_MACRO
* up-to-date) */
Oid reltoastrelid; /* OID of toast table; 0 if none */
bool relhasindex; /* T if has (or has had) any indexes */
+ bool relhascstore; /* T if has (or has had) any column stores */
bool relisshared; /* T if shared across databases */
char relpersistence; /* see RELPERSISTENCE_xxx constants below */
char relkind; /* see RELKIND_xxx constants below */
@@ -95,7 +96,7 @@ typedef FormData_pg_class *Form_pg_class;
* ----------------
*/
-#define Natts_pg_class 30
+#define Natts_pg_class 31
#define Anum_pg_class_relname 1
#define Anum_pg_class_relnamespace 2
#define Anum_pg_class_reltype 3
@@ -109,23 +110,24 @@ typedef FormData_pg_class *Form_pg_class;
#define Anum_pg_class_relallvisible 11
#define Anum_pg_class_reltoastrelid 12
#define Anum_pg_class_relhasindex 13
-#define Anum_pg_class_relisshared 14
-#define Anum_pg_class_relpersistence 15
-#define Anum_pg_class_relkind 16
-#define Anum_pg_class_relnatts 17
-#define Anum_pg_class_relchecks 18
-#define Anum_pg_class_relhasoids 19
-#define Anum_pg_class_relhaspkey 20
-#define Anum_pg_class_relhasrules 21
-#define Anum_pg_class_relhastriggers 22
-#define Anum_pg_class_relhassubclass 23
-#define Anum_pg_class_relrowsecurity 24
-#define Anum_pg_class_relispopulated 25
-#define Anum_pg_class_relreplident 26
-#define Anum_pg_class_relfrozenxid 27
-#define Anum_pg_class_relminmxid 28
-#define Anum_pg_class_relacl 29
-#define Anum_pg_class_reloptions 30
+#define Anum_pg_class_relhascstore 14
+#define Anum_pg_class_relisshared 15
+#define Anum_pg_class_relpersistence 16
+#define Anum_pg_class_relkind 17
+#define Anum_pg_class_relnatts 18
+#define Anum_pg_class_relchecks 19
+#define Anum_pg_class_relhasoids 20
+#define Anum_pg_class_relhaspkey 21
+#define Anum_pg_class_relhasrules 22
+#define Anum_pg_class_relhastriggers 23
+#define Anum_pg_class_relhassubclass 24
+#define Anum_pg_class_relrowsecurity 25
+#define Anum_pg_class_relispopulated 26
+#define Anum_pg_class_relreplident 27
+#define Anum_pg_class_relfrozenxid 28
+#define Anum_pg_class_relminmxid 29
+#define Anum_pg_class_relacl 30
+#define Anum_pg_class_reloptions 31
/* ----------------
* initial contents of pg_class
@@ -140,13 +142,13 @@ typedef FormData_pg_class *Form_pg_class;
* Note: "3" in the relfrozenxid column stands for FirstNormalTransactionId;
* similarly, "1" in relminmxid stands for FirstMultiXactId
*/
-DATA(insert OID = 1247 ( pg_type PGNSP 71 0 PGUID 0 0 0 0 0 0 0 f f p r 30 0 t f f \
f f f t n 3 1 _null_ _null_ )); +DATA(insert OID = 1247 ( pg_type PGNSP 71 0 PGUID \
0 0 0 0 0 0 0 f f f p r 30 0 t f f f f f t n 3 1 _null_ _null_ )); DESCR("");
-DATA(insert OID = 1249 ( pg_attribute PGNSP 75 0 PGUID 0 0 0 0 0 0 0 f f p r 21 0 f \
f f f f f t n 3 1 _null_ _null_ )); +DATA(insert OID = 1249 ( pg_attribute PGNSP 75 \
0 PGUID 0 0 0 0 0 0 0 f f f p r 21 0 f f f f f f t n 3 1 _null_ _null_ )); \
DESCR("");
-DATA(insert OID = 1255 ( pg_proc PGNSP 81 0 PGUID 0 0 0 0 0 0 0 f f p r 28 0 t f f \
f f f t n 3 1 _null_ _null_ )); +DATA(insert OID = 1255 ( pg_proc PGNSP 81 0 PGUID \
0 0 0 0 0 0 0 f f f p r 28 0 t f f f f f t n 3 1 _null_ _null_ )); DESCR("");
-DATA(insert OID = 1259 ( pg_class PGNSP 83 0 PGUID 0 0 0 0 0 0 0 f f p r 30 0 t f \
f f f f t n 3 1 _null_ _null_ )); +DATA(insert OID = 1259 ( pg_class PGNSP 83 0 \
PGUID 0 0 0 0 0 0 0 f f f p r 31 0 t f f f f f t n 3 1 _null_ _null_ )); DESCR("");
diff --git a/src/test/regress/expected/rules.out \
b/src/test/regress/expected/rules.out index 44c6740..8a1c43a 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1359,6 +1359,7 @@ pg_matviews| SELECT n.nspname AS schemaname,
pg_get_userbyid(c.relowner) AS matviewowner,
t.spcname AS tablespace,
c.relhasindex AS hasindexes,
+ c.relhascstore AS hascstores,
c.relispopulated AS ispopulated,
pg_get_viewdef(c.oid) AS definition
FROM ((pg_class c
@@ -2068,6 +2069,7 @@ pg_tables| SELECT n.nspname AS schemaname,
pg_get_userbyid(c.relowner) AS tableowner,
t.spcname AS tablespace,
c.relhasindex AS hasindexes,
+ c.relhascstore AS hascstores,
c.relhasrules AS hasrules,
c.relhastriggers AS hastriggers,
c.relrowsecurity AS rowsecurity
diff --git a/src/test/regress/expected/sanity_check.out \
b/src/test/regress/expected/sanity_check.out index b05061a..6abdab7 100644
--- a/src/test/regress/expected/sanity_check.out
+++ b/src/test/regress/expected/sanity_check.out
@@ -7,172 +7,172 @@ VACUUM;
--
-- temporarily disable fancy output, so catalog changes create less diff noise
\a\t
-SELECT relname, relhasindex
+SELECT relname, relhasindex, relhascstore
FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = relnamespace
WHERE relkind = 'r' AND (nspname ~ '^pg_temp_') IS NOT TRUE
ORDER BY relname;
-a|f
-a_star|f
-abstime_tbl|f
-aggtest|f
-array_index_op_test|t
-array_op_test|f
-b|f
-b_star|f
-box_tbl|f
-bprime|f
-bt_f8_heap|t
-bt_i4_heap|t
-bt_name_heap|t
-bt_txt_heap|t
-c|f
-c_star|f
-char_tbl|f
-check2_tbl|f
-check_tbl|f
-circle_tbl|t
-city|f
-copy_tbl|f
-d|f
-d_star|f
-date_tbl|f
-default_tbl|f
-defaultexpr_tbl|f
-dept|f
-dupindexcols|t
-e_star|f
-emp|f
-equipment_r|f
-f_star|f
-fast_emp4000|t
-float4_tbl|f
-float8_tbl|f
-func_index_heap|t
-hash_f8_heap|t
-hash_i4_heap|t
-hash_name_heap|t
-hash_txt_heap|t
-hobbies_r|f
-ihighway|t
-inet_tbl|f
-inhf|f
-inhx|t
-insert_tbl|f
-int2_tbl|f
-int4_tbl|f
-int8_tbl|f
-interval_tbl|f
-iportaltest|f
-kd_point_tbl|t
-line_tbl|f
-log_table|f
-lseg_tbl|f
-main_table|f
-money_data|f
-num_data|f
-num_exp_add|t
-num_exp_div|t
-num_exp_ln|t
-num_exp_log10|t
-num_exp_mul|t
-num_exp_power_10_ln|t
-num_exp_sqrt|t
-num_exp_sub|t
-num_input_test|f
-num_result|f
-onek|t
-onek2|t
-path_tbl|f
-person|f
-pg_aggregate|t
-pg_am|t
-pg_amop|t
-pg_amproc|t
-pg_attrdef|t
-pg_attribute|t
-pg_auth_members|t
-pg_authid|t
-pg_cast|t
-pg_class|t
-pg_collation|t
-pg_constraint|t
-pg_conversion|t
-pg_cstore|t
-pg_cstore_am|t
-pg_database|t
-pg_db_role_setting|t
-pg_default_acl|t
-pg_depend|t
-pg_description|t
-pg_enum|t
-pg_event_trigger|t
-pg_extension|t
-pg_foreign_data_wrapper|t
-pg_foreign_server|t
-pg_foreign_table|t
-pg_index|t
-pg_inherits|t
-pg_language|t
-pg_largeobject|t
-pg_largeobject_metadata|t
-pg_namespace|t
-pg_opclass|t
-pg_operator|t
-pg_opfamily|t
-pg_pltemplate|t
-pg_policy|t
-pg_proc|t
-pg_range|t
-pg_replication_origin|t
-pg_rewrite|t
-pg_seclabel|t
-pg_shdepend|t
-pg_shdescription|t
-pg_shseclabel|t
-pg_statistic|t
-pg_tablespace|t
-pg_transform|t
-pg_trigger|t
-pg_ts_config|t
-pg_ts_config_map|t
-pg_ts_dict|t
-pg_ts_parser|t
-pg_ts_template|t
-pg_type|t
-pg_user_mapping|t
-point_tbl|t
-polygon_tbl|t
-quad_point_tbl|t
-radix_text_tbl|t
-ramp|f
-real_city|f
-reltime_tbl|f
-road|t
-shighway|t
-slow_emp4000|f
-sql_features|f
-sql_implementation_info|f
-sql_languages|f
-sql_packages|f
-sql_parts|f
-sql_sizing|f
-sql_sizing_profiles|f
-stud_emp|f
-student|f
-tenk1|t
-tenk2|t
-test_range_excl|t
-test_range_gist|t
-test_range_spgist|t
-test_tsvector|f
-testjsonb|f
-text_tbl|f
-time_tbl|f
-timestamp_tbl|f
-timestamptz_tbl|f
-timetz_tbl|f
-tinterval_tbl|f
-varchar_tbl|f
+a|f|f
+a_star|f|f
+abstime_tbl|f|f
+aggtest|f|f
+array_index_op_test|t|f
+array_op_test|f|f
+b|f|f
+b_star|f|f
+box_tbl|f|f
+bprime|f|f
+bt_f8_heap|t|f
+bt_i4_heap|t|f
+bt_name_heap|t|f
+bt_txt_heap|t|f
+c|f|f
+c_star|f|f
+char_tbl|f|f
+check2_tbl|f|f
+check_tbl|f|f
+circle_tbl|t|f
+city|f|f
+copy_tbl|f|f
+d|f|f
+d_star|f|f
+date_tbl|f|f
+default_tbl|f|f
+defaultexpr_tbl|f|f
+dept|f|f
+dupindexcols|t|f
+e_star|f|f
+emp|f|f
+equipment_r|f|f
+f_star|f|f
+fast_emp4000|t|f
+float4_tbl|f|f
+float8_tbl|f|f
+func_index_heap|t|f
+hash_f8_heap|t|f
+hash_i4_heap|t|f
+hash_name_heap|t|f
+hash_txt_heap|t|f
+hobbies_r|f|f
+ihighway|t|f
+inet_tbl|f|f
+inhf|f|f
+inhx|t|f
+insert_tbl|f|f
+int2_tbl|f|f
+int4_tbl|f|f
+int8_tbl|f|f
+interval_tbl|f|f
+iportaltest|f|f
+kd_point_tbl|t|f
+line_tbl|f|f
+log_table|f|f
+lseg_tbl|f|f
+main_table|f|f
+money_data|f|f
+num_data|f|f
+num_exp_add|t|f
+num_exp_div|t|f
+num_exp_ln|t|f
+num_exp_log10|t|f
+num_exp_mul|t|f
+num_exp_power_10_ln|t|f
+num_exp_sqrt|t|f
+num_exp_sub|t|f
+num_input_test|f|f
+num_result|f|f
+onek|t|f
+onek2|t|f
+path_tbl|f|f
+person|f|f
+pg_aggregate|t|f
+pg_am|t|f
+pg_amop|t|f
+pg_amproc|t|f
+pg_attrdef|t|f
+pg_attribute|t|f
+pg_auth_members|t|f
+pg_authid|t|f
+pg_cast|t|f
+pg_class|t|f
+pg_collation|t|f
+pg_constraint|t|f
+pg_conversion|t|f
+pg_cstore|t|f
+pg_cstore_am|t|f
+pg_database|t|f
+pg_db_role_setting|t|f
+pg_default_acl|t|f
+pg_depend|t|f
+pg_description|t|f
+pg_enum|t|f
+pg_event_trigger|t|f
+pg_extension|t|f
+pg_foreign_data_wrapper|t|f
+pg_foreign_server|t|f
+pg_foreign_table|t|f
+pg_index|t|f
+pg_inherits|t|f
+pg_language|t|f
+pg_largeobject|t|f
+pg_largeobject_metadata|t|f
+pg_namespace|t|f
+pg_opclass|t|f
+pg_operator|t|f
+pg_opfamily|t|f
+pg_pltemplate|t|f
+pg_policy|t|f
+pg_proc|t|f
+pg_range|t|f
+pg_replication_origin|t|f
+pg_rewrite|t|f
+pg_seclabel|t|f
+pg_shdepend|t|f
+pg_shdescription|t|f
+pg_shseclabel|t|f
+pg_statistic|t|f
+pg_tablespace|t|f
+pg_transform|t|f
+pg_trigger|t|f
+pg_ts_config|t|f
+pg_ts_config_map|t|f
+pg_ts_dict|t|f
+pg_ts_parser|t|f
+pg_ts_template|t|f
+pg_type|t|f
+pg_user_mapping|t|f
+point_tbl|t|f
+polygon_tbl|t|f
+quad_point_tbl|t|f
+radix_text_tbl|t|f
+ramp|f|f
+real_city|f|f
+reltime_tbl|f|f
+road|t|f
+shighway|t|f
+slow_emp4000|f|f
+sql_features|f|f
+sql_implementation_info|f|f
+sql_languages|f|f
+sql_packages|f|f
+sql_parts|f|f
+sql_sizing|f|f
+sql_sizing_profiles|f|f
+stud_emp|f|f
+student|f|f
+tenk1|t|f
+tenk2|t|f
+test_range_excl|t|f
+test_range_gist|t|f
+test_range_spgist|t|f
+test_tsvector|f|f
+testjsonb|f|f
+text_tbl|f|f
+time_tbl|f|f
+timestamp_tbl|f|f
+timestamptz_tbl|f|f
+timetz_tbl|f|f
+tinterval_tbl|f|f
+varchar_tbl|f|f
-- restore normal output mode
\a\t
--
diff --git a/src/test/regress/sql/sanity_check.sql \
b/src/test/regress/sql/sanity_check.sql index 0da838e..ca87cbc 100644
--- a/src/test/regress/sql/sanity_check.sql
+++ b/src/test/regress/sql/sanity_check.sql
@@ -10,7 +10,7 @@ VACUUM;
-- temporarily disable fancy output, so catalog changes create less diff noise
\a\t
-SELECT relname, relhasindex
+SELECT relname, relhasindex, relhascstore
FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = relnamespace
WHERE relkind = 'r' AND (nspname ~ '^pg_temp_') IS NOT TRUE
ORDER BY relname;
--
2.1.4
["0010-Add-ColumnStoreOptInfo-to-RelationData.patch" (text/x-diff)]
From fb2d5bc74e244fabf8c951c14730672f8cc2ee92 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 23 Jun 2015 19:53:05 +0200
Subject: [PATCH 10/24] Add ColumnStoreOptInfo to RelationData
This commit adds new member rd_cstore which caches data from the
pg_cstore row for each RELKIND_COLUMN_STORE relation for easy access --
very similar to what rd_index does for indexes.
Implementation notes
- Adds a new node ColumnStoreOptInfo, parallel to IndexOptInfo
---
src/backend/nodes/outfuncs.c | 20 ++++++
src/backend/optimizer/util/plancat.c | 88 ++++++++++++++++++++++++
src/backend/utils/cache/relcache.c | 125 +++++++++++++++++++++++++++++++++++
src/include/nodes/nodes.h | 1 +
src/include/nodes/relation.h | 30 +++++++++
src/include/utils/rel.h | 10 +++
src/include/utils/relcache.h | 1 +
7 files changed, 275 insertions(+)
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 6399d30..235b4e3 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1904,6 +1904,23 @@ _outIndexOptInfo(StringInfo str, const IndexOptInfo *node)
}
static void
+_outColumnStoreOptInfo(StringInfo str, const ColumnStoreOptInfo *node)
+{
+ WRITE_NODE_TYPE("COLUMNSTOREOPTINFO");
+
+ /* NB: this isn't a complete set of fields */
+ WRITE_OID_FIELD(colstoreoid);
+
+ /* Do NOT print rel field, else infinite recursion */
+ WRITE_UINT_FIELD(pages);
+ WRITE_FLOAT_FIELD(tuples, "%.0f");
+ WRITE_INT_FIELD(ncolumns);
+ /* array fields aren't really worth the trouble to print */
+ WRITE_OID_FIELD(cstam);
+ /* we don't bother with fields copied from the pg_am entry */
+}
+
+static void
_outEquivalenceClass(StringInfo str, const EquivalenceClass *node)
{
/*
@@ -3343,6 +3360,9 @@ _outNode(StringInfo str, const void *obj)
case T_PlannerParamItem:
_outPlannerParamItem(str, obj);
break;
+ case T_ColumnStoreOptInfo:
+ _outColumnStoreOptInfo(str, obj);
+ break;
case T_CreateStmt:
_outCreateStmt(str, obj);
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 9442e5f..276c052 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -27,6 +27,7 @@
#include "catalog/catalog.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
+#include "catalog/pg_cstore.h"
#include "foreign/fdwapi.h"
#include "miscadmin.h"
#include "nodes/makefuncs.h"
@@ -71,6 +72,7 @@ static List *build_index_tlist(PlannerInfo *root, IndexOptInfo *index,
* min_attr lowest valid AttrNumber
* max_attr highest valid AttrNumber
* indexlist list of IndexOptInfos for relation's indexes
+ * cstlist list of ColumnStoreOptInfos for relation's colstores
* serverid if it's a foreign table, the server OID
* fdwroutine if it's a foreign table, the FDW function pointers
* pages number of pages
@@ -93,6 +95,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
Relation relation;
bool hasindex;
List *indexinfos = NIL;
+ List *colstoreinfos = NIL;
/*
* We need not lock the relation since it was already locked, either by
@@ -381,6 +384,91 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
rel->indexlist = indexinfos;
+ /* Grab column store info using the relcache, while we have it */
+ if (relation->rd_rel->relhascstore)
+ {
+ List *colstoreoidlist;
+ ListCell *l;
+ LOCKMODE lmode;
+
+ colstoreoidlist = RelationGetColStoreList(relation);
+
+ /*
+ * For each column store, we get the same type of lock that the
+ * executor will need, and do not release it. This saves a couple
+ * of trips to the shared lock manager while not creating any real
+ * loss of concurrency, because no schema changes could be
+ * happening on the colstore while we hold lock on the parent rel,
+ * and neither lock type blocks any other kind of colstore operation.
+ */
+ if (rel->relid == root->parse->resultRelation)
+ lmode = RowExclusiveLock;
+ else
+ lmode = AccessShareLock;
+
+ foreach(l, colstoreoidlist)
+ {
+ Oid colstoreoid = lfirst_oid(l);
+ Relation colstoreRelation;
+ Form_pg_cstore colstore;
+ ColumnStoreOptInfo *info;
+ int ncolumns;
+ int i;
+
+ /*
+ * Extract info from the relation descriptor for the colstore.
+ *
+ * FIXME There's no 'rd_cstore' in RelationData at the moment,
+ * so it needs to be added and integrated into relcache
+ * (just a bit of copy'n'paste programming using the
+ * rd_index logic).
+ *
+ * TODO Define colstore_open(), similar to index_open().
+ */
+ colstoreRelation = relation_open(colstoreoid, lmode);
+ colstore = colstoreRelation->rd_cstore;
+
+ /* XXX Invalid and not-yet-usable colstores would be handled here. */
+
+ info = makeNode(ColumnStoreOptInfo);
+
+ info->colstoreoid = colstore->cststoreid;
+ info->reltablespace =
+ RelationGetForm(colstoreRelation)->reltablespace;
+ info->rel = rel;
+ info->ncolumns = ncolumns = colstore->cstnatts;
+ info->cstkeys = (int *) palloc(sizeof(int) * ncolumns);
+
+ for (i = 0; i < ncolumns; i++)
+ info->cstkeys[i] = colstore->cstatts.values[i];
+
+ /*
+ * TODO This is where to fetch AM for the colstore (see how
+ * the index are handled above.
+ */
+
+ /*
+ * XXX this is where indexes build targetlists. Colstores don't
+ * have these (at least not ATM).
+ */
+
+ /*
+ * Estimate the colstore size. We don't support partial
+ * colstores, we just use the same reltuples as the parent.
+ */
+ info->pages = RelationGetNumberOfBlocks(colstoreRelation);
+ info->tuples = rel->tuples;
+
+ relation_close(colstoreRelation, NoLock);
+
+ colstoreinfos = lcons(info, colstoreinfos);
+ }
+
+ list_free(colstoreoidlist);
+ }
+
+ rel->cstlist = colstoreinfos;
+
/* Grab foreign-table info using the relcache, while we have it */
if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE)
{
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index ddb1fc0..8dcf42a 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -273,6 +273,7 @@ static void IndexSupportInitialize(oidvector *indclass,
AttrNumber maxAttributeNumber);
static OpClassCacheEnt *LookupOpclassInfo(Oid operatorClassOid,
StrategyNumber numSupport);
+static void RelationInitColumnStoreInfo(Relation rel);
static void RelationCacheInitFileRemoveInDir(const char *tblspcpath);
static void unlink_initfile(const char *initfilename);
@@ -432,6 +433,7 @@ RelationParseRelOptions(Relation relation, HeapTuple tuple)
case RELKIND_INDEX:
case RELKIND_VIEW:
case RELKIND_MATVIEW:
+ case RELKIND_COLUMN_STORE:
break;
default:
return;
@@ -1055,6 +1057,12 @@ RelationBuildDesc(Oid targetRelId, bool insertIt)
OidIsValid(relation->rd_rel->relam))
RelationInitIndexAccessInfo(relation);
+ /*
+ * if it's a column store, initialize pg_cstore data
+ */
+ if (relation->rd_rel->relkind == RELKIND_COLUMN_STORE)
+ RelationInitColumnStoreInfo(relation);
+
/* extract reloptions if any */
RelationParseRelOptions(relation, pg_class_tuple);
@@ -1528,6 +1536,27 @@ LookupOpclassInfo(Oid operatorClassOid,
return opcentry;
}
+/*
+ * For a column store relation, initialize rd_cstore
+ */
+static void
+RelationInitColumnStoreInfo(Relation rel)
+{
+ HeapTuple tuple;
+ MemoryContext oldcontext;
+
+ tuple = SearchSysCache1(CSTOREOID,
+ ObjectIdGetDatum(RelationGetRelid(rel)));
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for column store %u",
+ RelationGetRelid(rel));
+ oldcontext = MemoryContextSwitchTo(CacheMemoryContext);
+ rel->rd_cstoretuple = heap_copytuple(tuple);
+ rel->rd_cstore = (Form_pg_cstore) GETSTRUCT(rel->rd_cstoretuple);
+ MemoryContextSwitchTo(oldcontext);
+
+ ReleaseSysCache(tuple);
+}
/*
* formrdesc
@@ -3900,6 +3929,100 @@ RelationGetIndexList(Relation relation)
}
/*
+ * RelationGetColStoreList -- get a list of OIDs of colstores on this relation
+ *
+ * XXX Copy'n'paste of RelationGetIndexList(), simplified for colstores.
+ *
+ * The colstore list is created only if someone requests it. We scan pg_cstore
+ * to find relevant colstores, and add the list to the relcache entry so that
+ * we won't have to compute it again. Note that shared cache inval of a
+ * relcache entry will delete the old list and set rd_cstvalid to 0,
+ * so that we must recompute the colstore list on next request. This handles
+ * creation or deletion of a colstore.
+ *
+ * XXX At the moment we don't need the invalidation, because all the colstores
+ * are defined at table creation time, but if we ever decide to implement
+ * ALTER TABLE ... [ADD|DROP] COLUMN STORE, this will be handy. Also, this
+ * is just copy'n'paste programming using RelationGetIndexList().
+ *
+ * XXX Currently there are no 'islive' or 'isvalid' flags for colstores.
+ *
+ * The returned list is guaranteed to be sorted in order by OID. This is
+ * needed by the executor, since for colstore types that we obtain exclusive
+ * locks on when updating the colstore, all backends must lock the colstores
+ * in the same order or we will get deadlocks (see ExecOpenColstores()). Any
+ * consistent ordering would do, but ordering by OID is easy.
+ *
+ * Since shared cache inval causes the relcache's copy of the list to go away,
+ * we return a copy of the list palloc'd in the caller's context. The caller
+ * may list_free() the returned list after scanning it. This is necessary
+ * since the caller will typically be doing syscache lookups on the relevant
+ * colstores, and syscache lookup could cause SI messages to be processed!
+ */
+List *
+RelationGetColStoreList(Relation relation)
+{
+ Relation cstrel;
+ SysScanDesc cstscan;
+ ScanKeyData skey;
+ HeapTuple htup;
+ List *result;
+ List *oldlist;
+ MemoryContext oldcxt;
+
+ /* Quick exit if we already computed the list. */
+ if (relation->rd_cstvalid)
+ return list_copy(relation->rd_cstlist);
+
+ /*
+ * We build the list we intend to return (in the caller's context) while
+ * doing the scan. After successfully completing the scan, we copy that
+ * list into the relcache entry. This avoids cache-context memory leakage
+ * if we get some sort of error partway through.
+ */
+ result = NIL;
+
+ /* Prepare to scan pg_cstore for entries having cstrelid = this rel. */
+ ScanKeyInit(&skey,
+ Anum_pg_cstore_cstrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(RelationGetRelid(relation)));
+
+ cstrel = heap_open(CStoreRelationId, AccessShareLock);
+ cstscan = systable_beginscan(cstrel, CStoreCstRelidCstnameIndexId, true,
+ NULL, 1, &skey);
+
+ while (HeapTupleIsValid(htup = systable_getnext(cstscan)))
+ {
+ Form_pg_cstore cstore = (Form_pg_cstore) GETSTRUCT(htup);
+
+ /*
+ * TODO Ignore column stores that are being dropped (ALTER TABLE
+ * drop COLUMN STORE) here.
+ */
+
+ /* Add index's OID to result list in the proper order */
+ result = insert_ordered_oid(result, cstore->cststoreid);
+ }
+
+ systable_endscan(cstscan);
+
+ heap_close(cstrel, AccessShareLock);
+
+ /* Now save a copy of the completed list in the relcache entry. */
+ oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
+ oldlist = relation->rd_cstlist;
+ relation->rd_cstlist = list_copy(result);
+ relation->rd_cstvalid = true;
+ MemoryContextSwitchTo(oldcxt);
+
+ /* Don't leak the old list, if there is one */
+ list_free(oldlist);
+
+ return result;
+}
+
+/*
* insert_ordered_oid
* Insert a new Oid into a sorted list of Oids, preserving ordering
*
@@ -4871,6 +4994,8 @@ load_relcache_init_file(bool shared)
rel->rd_createSubid = InvalidSubTransactionId;
rel->rd_newRelfilenodeSubid = InvalidSubTransactionId;
rel->rd_amcache = NULL;
+ rel->rd_cstvalid = 0;
+ rel->rd_cstlist = NIL;
MemSet(&rel->pgstat_info, 0, sizeof(rel->pgstat_info));
/*
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 27b38b1..53e0a0b 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -249,6 +249,7 @@ typedef enum NodeTag
T_PlaceHolderInfo,
T_MinMaxAggInfo,
T_PlannerParamItem,
+ T_ColumnStoreOptInfo,
/*
* TAGS FOR MEMORY NODES (memnodes.h)
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 5dc23d9..18a8891 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -468,6 +468,7 @@ typedef struct RelOptInfo
Relids lateral_relids; /* minimum parameterization of rel */
Relids lateral_referencers; /* rels that reference me laterally */
List *indexlist; /* list of IndexOptInfo */
+ List *cstlist; /* list of ColumnStoreOptInfo */
BlockNumber pages; /* size estimates derived from pg_class */
double tuples;
double allvisfrac;
@@ -562,6 +563,35 @@ typedef struct IndexOptInfo
bool amhasgetbitmap; /* does AM have amgetbitmap interface? */
} IndexOptInfo;
+/*
+ * ColumnStoreOptInfo
+ * Per-colstore information for planning/optimization
+ *
+ * cstkeys[] has ncolumns entries, we don't allow expression in colstores
+ *
+ * TODO maybe should have a bunch of capability flags like IndexOptInfo
+ */
+typedef struct ColumnStoreOptInfo
+{
+ NodeTag type;
+
+ Oid colstoreoid; /* OID of the column store relation */
+ Oid reltablespace; /* tablespace of colstore (not table) */
+ RelOptInfo *rel; /* back-link to colstore's table */
+
+ /* colstore-size statistics (from pg_class and elsewhere) */
+ BlockNumber pages; /* number of disk pages in colstore */
+ double tuples; /* number of index tuples in colstore */
+
+ /* colstore descriptor information (from pg_cstore) */
+ int ncolumns; /* number of columns in index */
+ int *cstkeys; /* column numbers of colstore's keys */
+ Oid cstam; /* OID of the access method (in pg_cstore_am) */
+
+ RegProcedure cstcostestimate; /* OID of the colstore method's cost fcn */
+
+ List *csttlist; /* targetlist representing colstore's columns */
+} ColumnStoreOptInfo;
/*
* EquivalenceClasses
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 8a55a09..99c7ef6 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -17,6 +17,7 @@
#include "access/tupdesc.h"
#include "catalog/pg_am.h"
#include "catalog/pg_class.h"
+#include "catalog/pg_cstore.h"
#include "catalog/pg_index.h"
#include "fmgr.h"
#include "nodes/bitmapset.h"
@@ -79,6 +80,7 @@ typedef struct RelationData
bool rd_isvalid; /* relcache entry is valid */
char rd_indexvalid; /* state of rd_indexlist: 0 = not valid, 1 =
* valid, 2 = temporarily forced */
+ bool rd_cstvalid; /* rd_cstlist is valid */
/*
* rd_createSubid is the ID of the highest subtransaction the rel has
@@ -117,6 +119,9 @@ typedef struct RelationData
Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */
Bitmapset *rd_idattr; /* included in replica identity index */
+ /* data managed by RelationGetColStoreList: */
+ List *rd_cstlist; /* list of OIDs of colstores on relation */
+
/*
* rd_options is set whenever rd_rel is loaded into the relcache entry.
* Note that you can NOT look into rd_rel for this data. NULL means "use
@@ -171,6 +176,11 @@ typedef struct RelationData
/* use "struct" here to avoid needing to include fdwapi.h: */
struct FdwRoutine *rd_fdwroutine; /* cached function pointers, or NULL */
+ /* These are non-NULL only for a column store relation: */
+ Form_pg_cstore rd_cstore; /* pg_cstore tuple describing this colstore */
+ /* use "struct" here to avoid needing to include htup.h: */
+ struct HeapTupleData *rd_cstoretuple; /* all of pg_cstore tuple */
+
/*
* Hack for CLUSTER, rewriting ALTER TABLE, etc: when writing a new
* version of a table, we need to make any toast pointers inserted into it
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h
index 6953281..0a8d52c 100644
--- a/src/include/utils/relcache.h
+++ b/src/include/utils/relcache.h
@@ -38,6 +38,7 @@ extern void RelationClose(Relation relation);
* Routines to compute/retrieve additional cached information
*/
extern List *RelationGetIndexList(Relation relation);
+extern List *RelationGetColStoreList(Relation relation);
extern Oid RelationGetOidIndex(Relation relation);
extern Oid RelationGetReplicaIndex(Relation relation);
extern List *RelationGetIndexExpressions(Relation relation);
--
2.1.4
["0011-Infrastructure-to-create-column-stores.patch" (text/x-diff)]
From 993eea51a2b73f7ee02454b290e0e40288e86a03 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 24 Apr 2015 19:15:02 -0300
Subject: [PATCH 11/24] Infrastructure to create column stores
This introduces enough infrastructure to actually create the catalog
entries when the grammar has received COLUMN STORE clauses.
This creates pg_class entries (and relfilenodes) for the colstores, with
the corresponding pg_attribute entries; pg_cstore entries are also
created. Add pg_depend entries too, so that everything is dropped when
the table is.
We create one relfilenode for each column store, which is referenced in
pg_class.relfilenode. The column store AM is free to use this or not,
as well as create other relfilenodes by itself. If we get a
pg_relfilenode catalog as discussed, this may be improved.
Implementation notes:
- Column store creation has two steps: DetermineColumnStores (called
before the heap itself is created) examines table-level and
column-level clauses, and merges both with column stores coming from
inherited relations. These are transformed into a list of
ColumnStoreElem struct.
The second step occurs after the heap itself is created, and records
everything into pg_class, pg_cstore and pg_attribute entries,
one relfilenodes for each column store, assigns OIDs to everything,
etc.
- This patch creates new ObjectClass and ObjectType enum members for column
stores, and associated fallout.
- Need to get rid of OIDs in pg_cstore, which requires some changes in
the pg_depend entries and the deletion code introduced here.
- FIXME CloneColumnStores is a stub. This function is used to recreate
column stores when a table's heap is rewritten (cluster, vacuum full, etc).
- FIXME The event trigger system hasn't been updated to cope with the
new object types.
---
src/backend/access/common/tupdesc.c | 20 +-
src/backend/bootstrap/bootparse.y | 1 +
src/backend/catalog/Makefile | 6 +-
src/backend/catalog/colstore.c | 544 +++++++++++++++++++++++++++++++++++
src/backend/catalog/dependency.c | 9 +
src/backend/catalog/heap.c | 22 +-
src/backend/catalog/objectaddress.c | 80 ++++++
src/backend/catalog/toasting.c | 1 +
src/backend/commands/cluster.c | 2 +
src/backend/commands/event_trigger.c | 4 +
src/backend/commands/tablecmds.c | 79 ++++-
src/backend/commands/view.c | 2 +-
src/include/access/tupdesc.h | 6 +-
src/include/catalog/colstore.h | 41 +++
src/include/catalog/dependency.h | 1 +
src/include/catalog/heap.h | 7 +
src/include/nodes/parsenodes.h | 1 +
17 files changed, 811 insertions(+), 15 deletions(-)
create mode 100644 src/backend/catalog/colstore.c
create mode 100644 src/include/catalog/colstore.h
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 41d71c8..cb622a7 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -20,6 +20,7 @@
#include "postgres.h"
#include "access/htup_details.h"
+#include "catalog/colstore.h"
#include "catalog/pg_type.h"
#include "miscadmin.h"
#include "parser/parse_type.h"
@@ -580,12 +581,17 @@ TupleDescInitEntryCollation(TupleDesc desc,
*
* Given a relation schema (list of ColumnDef nodes), build a TupleDesc.
*
+ * Also append ColumnStoreClauseInfo structs in *colstores, with one element for
+ * each attribute that contains a COLUMN STORE clause. colstores may be NULL if
+ * the caller is certain that there are no colstore clauses (e.g. when defining a
+ * view.)
+ *
* Note: the default assumption is no OIDs; caller may modify the returned
* TupleDesc if it wants OIDs. Also, tdtypeid will need to be filled in
* later on.
*/
TupleDesc
-BuildDescForRelation(List *schema)
+BuildDescForRelation(List *schema, List **colstores)
{
int natts;
AttrNumber attnum;
@@ -648,6 +654,18 @@ BuildDescForRelation(List *schema)
has_not_null |= entry->is_not_null;
desc->attrs[attnum - 1]->attislocal = entry->is_local;
desc->attrs[attnum - 1]->attinhcount = entry->inhcount;
+
+ /* Fill in the column store info, if this column requires it */
+ if (entry->cstoreClause)
+ {
+ ColumnStoreClauseInfo *store = palloc(sizeof(ColumnStoreClauseInfo));
+
+ store->attnum = attnum;
+ store->cstoreClause = entry->cstoreClause;
+ store->attnums = NIL;
+ store->cstoreOid = InvalidOid;
+ *colstores = lappend(*colstores, store);
+ }
}
if (has_not_null)
diff --git a/src/backend/bootstrap/bootparse.y b/src/backend/bootstrap/bootparse.y
index d8d1b06..10d38d2 100644
--- a/src/backend/bootstrap/bootparse.y
+++ b/src/backend/bootstrap/bootparse.y
@@ -240,6 +240,7 @@ Boot_CreateStmt:
BOOTSTRAP_SUPERUSERID,
tupdesc,
NIL,
+ NIL,
RELKIND_RELATION,
RELPERSISTENCE_PERMANENT,
shared_relation,
diff --git a/src/backend/catalog/Makefile b/src/backend/catalog/Makefile
index bdf4e35..f28dd8c 100644
--- a/src/backend/catalog/Makefile
+++ b/src/backend/catalog/Makefile
@@ -10,9 +10,9 @@ subdir = src/backend/catalog
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-OBJS = catalog.o dependency.o heap.o index.o indexing.o namespace.o aclchk.o \
- objectaccess.o objectaddress.o pg_aggregate.o pg_collation.o \
- pg_constraint.o pg_conversion.o \
+OBJS = aclchk.o catalog.o colstore.o dependency.o heap.o index.o indexing.o \
+ namespace.o objectaccess.o objectaddress.o \
+ pg_aggregate.o pg_collation.o pg_constraint.o pg_conversion.o \
pg_depend.o pg_enum.o pg_inherits.o pg_largeobject.o pg_namespace.o \
pg_operator.o pg_proc.o pg_range.o pg_db_role_setting.o pg_shdepend.o \
pg_type.o storage.o toasting.o
diff --git a/src/backend/catalog/colstore.c b/src/backend/catalog/colstore.c
new file mode 100644
index 0000000..553ab68
--- /dev/null
+++ b/src/backend/catalog/colstore.c
@@ -0,0 +1,544 @@
+/*-------------------------------------------------------------------------
+ *
+ * colstore.c
+ * POSTGRES column store support code
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/catalog/colstore.c
+ */
+#include "postgres.h"
+
+#include "access/genam.h"
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/sysattr.h"
+#include "catalog/catalog.h"
+#include "catalog/colstore.h"
+#include "catalog/dependency.h"
+#include "catalog/heap.h"
+#include "catalog/indexing.h"
+#include "catalog/pg_cstore.h"
+#include "catalog/pg_namespace.h"
+#include "catalog/pg_tablespace.h" /* GLOBALTABLESPACE_OID */
+#include "commands/tablespace.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "nodes/parsenodes.h"
+#include "utils/builtins.h"
+#include "utils/fmgroids.h"
+#include "utils/lsyscache.h"
+#include "utils/syscache.h"
+#include "utils/rel.h"
+
+typedef struct ColumnStoreElem
+{
+ char *name;
+ Oid cst_am_oid;
+ Oid tablespaceId;
+ List *columns; /* of AttrNumber */
+} ColumnStoreElem;
+
+static TupleDesc ColumnStoreBuildDesc(ColumnStoreElem *elem, Relation parent);
+static void AddNewColstoreTuple(Relation pg_cstore, ColumnStoreElem *entry,
+ Oid storeId, Oid relid);
+
+/*
+ * Figure out the set of column stores to create.
+ *
+ * decl_cstores is a list of column stores directly declared for the relation.
+ * inh_cstores is a list of column stores inherited from parent relations.
+ * We need this distinction because multiple uses of a column in declared ones
+ * is an error, but we ignore duplicates for the inherited ones.
+ *
+ * tablespaceId is the tablespace of the owning relation; it is used as the
+ * default tablespace for column stores that do not have a tablespace
+ * specification.
+ *
+ * Return value is a list of ColumnStoreElem.
+ */
+List *
+DetermineColumnStores(TupleDesc tupdesc, List *decl_cstores,
+ List *inh_cstores, Oid tablespaceId)
+{
+ List *newstores = NIL;
+ Bitmapset *used;
+ ListCell *cell,
+ *cell2;
+
+ /*
+ * There is no provision (not here, nor in the grammar) for a column that's
+ * part of a column store in the parent, but in the heap for the child
+ * table. Do we need a fix for that?
+ */
+
+ /*
+ * We use this bitmapset to keep track of columns which have already been
+ * assigned to a store.
+ */
+ used = NULL;
+
+ /*
+ * First scan the list of declared column stores. Using columns here more
+ * than once causes an error to be raised.
+ */
+ foreach(cell, decl_cstores)
+ {
+ ColumnStoreClauseInfo *info = (ColumnStoreClauseInfo *) lfirst(cell);
+ ColumnStoreClause *clause = info->cstoreClause;
+ ColumnStoreElem *newstore;
+
+ Assert(clause != NULL);
+ Assert((info->attnum != InvalidAttrNumber) ||
+ (clause->columns != NIL));
+ Assert(info->attnums == NIL && info->cstoreOid == InvalidOid);
+
+ /*
+ * Verify that the name has not already been taken
+ */
+ foreach(cell2, newstores)
+ {
+ ColumnStoreElem *elem = (ColumnStoreElem *) lfirst(cell2);
+
+ if (strcmp(elem->name, clause->name) == 0)
+ /* XXX ereport */
+ elog(ERROR, "duplicate column store name \"%s\"", clause->name);
+ }
+
+ newstore = (ColumnStoreElem *) palloc(sizeof(ColumnStoreElem));
+ newstore->name = clause->name;
+ newstore->cst_am_oid = GetColumnStoreAMByName(clause->storetype, false);
+
+ /*
+ * Select tablespace to use. If not specified, use the tablespace
+ * of the parent relation.
+ *
+ * These are effectively the same checks as in DefineRelation.
+ */
+ if (clause->tablespacename)
+ {
+ newstore->tablespaceId =
+ get_tablespace_oid(clause->tablespacename, false);
+ }
+ else
+ /* use tablespace of the relation */
+ newstore->tablespaceId = tablespaceId;
+
+ /* Check permissions except when using database's default */
+ if (OidIsValid(newstore->tablespaceId) &&
+ newstore->tablespaceId != MyDatabaseTableSpace)
+ {
+ AclResult aclresult;
+
+ aclresult = pg_tablespace_aclcheck(newstore->tablespaceId,
+ GetUserId(), ACL_CREATE);
+ if (aclresult != ACLCHECK_OK)
+ aclcheck_error(aclresult, ACL_KIND_TABLESPACE,
+ get_tablespace_name(newstore->tablespaceId));
+ }
+
+ /* In all cases disallow placing user objects in pg_global */
+ if (newstore->tablespaceId == GLOBALTABLESPACE_OID)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("only shared relations can be placed in pg_global tablespace")));
+
+ /*
+ * Fill in the attnum list: if it's a column store declared as column
+ * constraint, the only attnum was already determined by
+ * BuildDescForRelation. Otherwise we need to resolve column names to
+ * attnums using the tuple descriptor.
+ */
+ if (info->attnum != InvalidAttrNumber)
+ newstore->columns = list_make1_int(info->attnum);
+ else
+ {
+ newstore->columns = NIL;
+
+ foreach(cell2, clause->columns)
+ {
+ char *colname = strVal(lfirst(cell2));
+ AttrNumber attnum,
+ i;
+
+ attnum = InvalidAttrNumber;
+ for (i = 1; i <= tupdesc->natts; i++)
+ {
+ if (namestrcmp(&(tupdesc->attrs[i - 1]->attname), colname) == 0)
+ attnum = i;
+ }
+ /* XXX ereport */
+ if (attnum == InvalidAttrNumber)
+ elog(ERROR, "no column \"%s\" in the table", colname);
+
+ newstore->columns = lappend_int(newstore->columns, attnum);
+ }
+ }
+
+ /* Make sure there are no columns specified in multiple stores */
+ foreach(cell2, newstore->columns)
+ {
+ AttrNumber attno = lfirst_int(cell2);
+
+ /* XXX ereport */
+ if (bms_is_member(attno, used))
+ elog(ERROR, "column already in a store");
+
+ used = bms_add_member(used, attno);
+ }
+
+ newstores = lappend(newstores, newstore);
+ }
+
+ /*
+ * Now process the list of column stores coming from parent relations.
+ * Columns that are already used by previous stores are silently ignored.
+ * (In particular, this means that some parent stores might not exist at
+ * all in the child).
+ */
+ foreach (cell, inh_cstores)
+ {
+ ColumnStoreClauseInfo *info = (ColumnStoreClauseInfo *) lfirst(cell);
+ Relation parentstore;
+ List *attnums = NIL;
+
+ Assert((info->attnum == InvalidAttrNumber) &&
+ (info->cstoreClause == NULL));
+ Assert((info->attnums != NIL) &&
+ (info->cstoreOid != InvalidOid));
+
+ parentstore = relation_open(info->cstoreOid, AccessShareLock);
+
+ /*
+ * Examine the column list first. If all columns are used in
+ * previously defined column stores, we can ignore this one.
+ */
+ foreach(cell2, info->attnums)
+ {
+ AttrNumber attnum = lfirst_int(cell2);
+
+ if (bms_is_member(attnum, used))
+ continue;
+ attnums = lappend_int(attnums, attnum);
+ used = bms_add_member(used, attnum);
+ }
+
+ /* If we ended up with a nonempty list, add this store to the list */
+ if (attnums != NIL)
+ {
+ ColumnStoreElem *newstore;
+ newstore = (ColumnStoreElem *) palloc(sizeof(ColumnStoreElem));
+
+ newstore->name = pstrdup(NameStr(parentstore->rd_cstore->cstname));
+ newstore->cst_am_oid = parentstore->rd_cstore->cststoreid;
+ newstore->tablespaceId = parentstore->rd_rel->reltablespace;
+ newstore->columns = attnums;
+
+ newstores = lappend(newstores, newstore);
+ }
+
+ relation_close(parentstore, AccessShareLock);
+ }
+
+ /*
+ * Check that the names of the column stores are unique, so that we can
+ * print a nice error message instead of a confusing unique violation
+ * later. This is O(N^2), but that should not be a problem.
+ *
+ * XXX We don't have relname here, so we can't put it to the message.
+ */
+ foreach (cell, newstores)
+ {
+ ListCell *cell2;
+ ColumnStoreElem *elem1 = (ColumnStoreElem *) lfirst(cell);
+
+ foreach (cell2, newstores)
+ {
+ ColumnStoreElem *elem2 = (ColumnStoreElem *) lfirst(cell2);
+
+ if (elem1 == elem2) /* skip the same element */
+ continue;
+
+ /* same names */
+ if (strcmp(elem1->name, elem2->name) == 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("column store \"%s\" already exists", elem1->name)));
+
+ }
+ }
+
+ /*
+ * Return the info we collected.
+ */
+ return newstores;
+}
+
+/*
+ * Create the column stores for the given table. This creates the files
+ * assigned as relfilenode for each column store; also, the pg_class and
+ * pg_cstore catalog entries are created.
+ */
+void
+CreateColumnStores(Relation rel, List *colstores)
+{
+ Relation pg_class;
+ Relation pg_cstore;
+ ListCell *cell;
+ int storenum = 1;
+ ObjectAddress parentrel;
+
+ if (colstores == NIL)
+ return;
+
+ pg_class = heap_open(RelationRelationId, RowExclusiveLock);
+ pg_cstore = heap_open(CStoreRelationId, RowExclusiveLock);
+
+ ObjectAddressSet(parentrel, RelationRelationId,
+ RelationGetRelid(rel));
+
+ foreach(cell, colstores)
+ {
+ ColumnStoreElem *elem = (ColumnStoreElem *) lfirst(cell);
+ Relation store;
+ Oid newStoreId;
+ TupleDesc storedesc = ColumnStoreBuildDesc(elem, rel);
+ char *classname;
+ ObjectAddress myself;
+
+ /*
+ * Get the OID for the new column store.
+ */
+ newStoreId = GetNewRelFileNode(elem->tablespaceId, pg_class,
+ rel->rd_rel->relpersistence);
+
+ classname = psprintf("pg_colstore_%u_%d",
+ RelationGetRelid(rel), storenum++);
+
+ /*
+ * Create the relcache entry for the store. This also creates the
+ * underlying storage; it's smgr's responsibility to remove the file if
+ * we fail later on.
+ */
+ store =
+ heap_create(classname,
+ PG_COLSTORE_NAMESPACE,
+ elem->tablespaceId,
+ newStoreId,
+ InvalidOid,
+ storedesc,
+ RELKIND_COLUMN_STORE,
+ rel->rd_rel->relpersistence,
+ false,
+ false,
+ allowSystemTableMods);
+
+ /* insert pg_attribute tuples */
+ AddNewAttributeTuples(newStoreId,
+ storedesc,
+ RELKIND_COLUMN_STORE,
+ false,
+ 0);
+
+ /*
+ * Insert the pg_class tuple
+ */
+ store->rd_rel->relam = elem->cst_am_oid;
+ InsertPgClassTuple(pg_class, store, newStoreId,
+ (Datum) 0, (Datum) 0);
+
+ /* And finally insert the pg_cstore tuple */
+ AddNewColstoreTuple(pg_cstore, elem, newStoreId,
+ RelationGetRelid(rel));
+
+ /*
+ * This is a good place to record dependencies. We choose to have all the
+ * subsidiary entries (both pg_class and pg_cstore entries) depend on the
+ * pg_class entry for the main relation.
+ */
+ ObjectAddressSet(myself, CStoreRelationId, newStoreId);
+ recordDependencyOn(&myself, &parentrel, DEPENDENCY_INTERNAL);
+ ObjectAddressSet(myself, RelationRelationId, newStoreId);
+ recordDependencyOn(&myself, &parentrel, DEPENDENCY_INTERNAL);
+
+ heap_close(store, NoLock);
+ }
+
+ heap_close(pg_class, RowExclusiveLock);
+ heap_close(pg_cstore, RowExclusiveLock);
+}
+
+/*
+ * Build a tuple descriptor for a not-yet-catalogued column store.
+ */
+static TupleDesc
+ColumnStoreBuildDesc(ColumnStoreElem *elem, Relation parent)
+{
+ TupleDesc tupdesc;
+ TupleDesc parentdesc;
+ ListCell *cell;
+ AttrNumber attnum = 1;
+
+ parentdesc = RelationGetDescr(parent);
+
+ tupdesc = CreateTemplateTupleDesc(list_length(elem->columns), false);
+
+ foreach(cell, elem->columns)
+ {
+ AttrNumber parentattnum = lfirst_int(cell);
+
+ TupleDescInitEntry(tupdesc,
+ attnum++,
+ NameStr(parentdesc->attrs[parentattnum - 1]->attname),
+ parentdesc->attrs[parentattnum - 1]->atttypid,
+ parentdesc->attrs[parentattnum - 1]->atttypmod,
+ parentdesc->attrs[parentattnum - 1]->attndims);
+ }
+
+ return tupdesc;
+}
+
+/*
+ * Return a list of column store definitions for an existing table
+ */
+List *
+CloneColumnStores(Relation rel)
+{
+ /* FIXME fill this in */
+ return NIL;
+}
+
+/*
+ * Add a new pg_cstore tuple for a column store
+ */
+static void
+AddNewColstoreTuple(Relation pg_cstore, ColumnStoreElem *entry,
+ Oid storeId, Oid relid)
+{
+ HeapTuple newtup;
+ ListCell *cell;
+ Datum values[Natts_pg_cstore];
+ bool nulls[Natts_pg_cstore];
+ NameData cstname;
+ int natts;
+ int16 *attrarr;
+ int2vector *attrs;
+ int i = 0;
+
+ /* build the int2vector of attribute numbers */
+ natts = list_length(entry->columns);
+ Assert(natts > 0);
+ attrarr = palloc(sizeof(int16 *) * natts);
+ foreach(cell, entry->columns)
+ attrarr[i++] = (AttrNumber) lfirst_int(cell);
+ attrs = buildint2vector(attrarr, natts);
+
+ /* build the pg_cstore tuple */
+ namestrcpy(&cstname, entry->name);
+ values[Anum_pg_cstore_cstname - 1] = NameGetDatum(&cstname);
+ values[Anum_pg_cstore_cstrelid - 1] = ObjectIdGetDatum(relid);
+ values[Anum_pg_cstore_cststoreid - 1] = ObjectIdGetDatum(storeId);
+ values[Anum_pg_cstore_cstnatts - 1] = Int32GetDatum(list_length(entry->columns));
+ values[Anum_pg_cstore_cstatts - 1] = PointerGetDatum(attrs);
+ memset(nulls, 0, sizeof(nulls));
+ newtup = heap_form_tuple(RelationGetDescr(pg_cstore), values, nulls);
+
+ HeapTupleSetOid(newtup, storeId);
+
+ /* insert it into pg_cstore */
+ simple_heap_insert(pg_cstore, newtup);
+
+ /* keep indexes current */
+ CatalogUpdateIndexes(pg_cstore, newtup);
+
+ heap_freetuple(newtup);
+}
+
+Oid
+get_relation_cstore_oid(Oid relid, const char *cstore_name, bool missing_ok)
+{
+ Relation pg_cstore_rel;
+ ScanKeyData skey[2];
+ SysScanDesc sscan;
+ HeapTuple cstore_tuple;
+ Oid cstore_oid;
+
+ pg_cstore_rel = heap_open(CStoreRelationId, AccessShareLock);
+
+ ScanKeyInit(&skey[0],
+ Anum_pg_cstore_cstrelid,
+ BTEqualStrategyNumber, F_OIDEQ,
+ ObjectIdGetDatum(relid));
+ ScanKeyInit(&skey[1],
+ Anum_pg_cstore_cstname,
+ BTEqualStrategyNumber, F_NAMEEQ,
+ CStringGetDatum(cstore_name));
+
+ sscan = systable_beginscan(pg_cstore_rel,
+ CStoreCstRelidCstnameIndexId, true, NULL, 2,
+ skey);
+
+ cstore_tuple = systable_getnext(sscan);
+
+ if (!HeapTupleIsValid(cstore_tuple))
+ {
+ if (!missing_ok)
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("column store \"%s\" for table \"%s\" does not exist",
+ cstore_name, get_rel_name(relid))));
+
+ cstore_oid = InvalidOid;
+ }
+ else
+ cstore_oid = HeapTupleGetOid(cstore_tuple);
+
+ /* Clean up. */
+ systable_endscan(sscan);
+ heap_close(pg_cstore_rel, AccessShareLock);
+
+ return cstore_oid;
+}
+
+void
+RemoveColstoreById(Oid cstoreOid)
+{
+ Relation pg_cstore;
+ HeapTuple tuple;
+
+ pg_cstore = heap_open(CStoreRelationId, RowExclusiveLock);
+
+ tuple = SearchSysCache1(CSTOREOID, ObjectIdGetDatum(cstoreOid));
+
+ if (!HeapTupleIsValid(tuple))
+ elog(ERROR, "cache lookup failed for column store %u", cstoreOid);
+
+ simple_heap_delete(pg_cstore, &tuple->t_self);
+
+ ReleaseSysCache(tuple);
+
+ heap_close(pg_cstore, RowExclusiveLock);
+}
+
+Oid
+GetColumnStoreAMByName(char *cstamname, bool missing_ok)
+{
+ Oid oid;
+
+ oid = GetSysCacheOid1(CSTOREAMNAME,
+ CStringGetDatum(cstamname));
+ if (!OidIsValid(oid))
+ {
+ if (missing_ok)
+ return InvalidOid;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("column store access method \"%s\" does not exist",
+ cstamname)));
+ }
+
+ return oid;
+}
diff --git a/src/backend/catalog/dependency.c b/src/backend/catalog/dependency.c
index 90b1cd8..8584890 100644
--- a/src/backend/catalog/dependency.c
+++ b/src/backend/catalog/dependency.c
@@ -16,6 +16,7 @@
#include "access/htup_details.h"
#include "access/xact.h"
+#include "catalog/colstore.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/index.h"
@@ -144,6 +145,7 @@ static const Oid object_classes[] = {
AccessMethodProcedureRelationId, /* OCLASS_AMPROC */
RewriteRelationId, /* OCLASS_REWRITE */
TriggerRelationId, /* OCLASS_TRIGGER */
+ CStoreRelationId, /* OCLASS_COLSTORE */
NamespaceRelationId, /* OCLASS_SCHEMA */
TSParserRelationId, /* OCLASS_TSPARSER */
TSDictionaryRelationId, /* OCLASS_TSDICT */
@@ -1214,6 +1216,10 @@ doDeletion(const ObjectAddress *object, int flags)
RemoveTriggerById(object->objectId);
break;
+ case OCLASS_COLSTORE:
+ RemoveColstoreById(object->objectId);
+ break;
+
case OCLASS_SCHEMA:
RemoveSchemaById(object->objectId);
break;
@@ -2412,6 +2418,9 @@ getObjectClass(const ObjectAddress *object)
case PolicyRelationId:
return OCLASS_POLICY;
+ case CStoreRelationId:
+ return OCLASS_COLSTORE;
+
case TransformRelationId:
return OCLASS_TRANSFORM;
}
diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c
index 2a6c576..a9f34d5 100644
--- a/src/backend/catalog/heap.c
+++ b/src/backend/catalog/heap.c
@@ -37,6 +37,7 @@
#include "access/xlog.h"
#include "catalog/binary_upgrade.h"
#include "catalog/catalog.h"
+#include "catalog/colstore.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/index.h"
@@ -653,7 +654,7 @@ InsertPgAttributeTuple(Relation pg_attribute_rel,
* tuples to pg_attribute.
* --------------------------------
*/
-static void
+void
AddNewAttributeTuples(Oid new_rel_oid,
TupleDesc tupdesc,
char relkind,
@@ -712,10 +713,11 @@ AddNewAttributeTuples(Oid new_rel_oid,
/*
* Next we add the system attributes. Skip OID if rel has no OIDs. Skip
- * all for a view or type relation. We don't bother with making datatype
- * dependencies here, since presumably all these types are pinned.
+ * all for a colstore, view or type relation. We don't bother with making
+ * datatype dependencies here, since presumably all these types are pinned.
*/
- if (relkind != RELKIND_VIEW && relkind != RELKIND_COMPOSITE_TYPE)
+ if (relkind != RELKIND_VIEW && relkind != RELKIND_COMPOSITE_TYPE &&
+ relkind != RELKIND_COLUMN_STORE)
{
for (i = 0; i < (int) lengthof(SysAtt); i++)
{
@@ -994,6 +996,7 @@ AddNewRelationType(const char *typeName,
* ownerid: OID of new rel's owner
* tupdesc: tuple descriptor (source of column definitions)
* cooked_constraints: list of precooked check constraints and defaults
+ * colstores: list (of ColumnStoreElem) of column stores for this rel
* relkind: relkind for new rel
* relpersistence: rel's persistence status (permanent, temp, or unlogged)
* shared_relation: TRUE if it's to be a shared relation
@@ -1023,6 +1026,7 @@ heap_create_with_catalog(const char *relname,
Oid ownerid,
TupleDesc tupdesc,
List *cooked_constraints,
+ List *colstores,
char relkind,
char relpersistence,
bool shared_relation,
@@ -1248,6 +1252,9 @@ heap_create_with_catalog(const char *relname,
pfree(relarrayname);
}
+ /* Set relhascstore correctly */
+ new_rel_desc->rd_rel->relhascstore = colstores != NIL;
+
/*
* now create an entry in pg_class for the relation.
*
@@ -1266,6 +1273,13 @@ heap_create_with_catalog(const char *relname,
reloptions);
/*
+ * If the new relation has any column stores, create them now. This
+ * assigns their OIDs and creates the files on disk (it's smgr's
+ * responsibility to remove these files if we fail below.)
+ */
+ CreateColumnStores(new_rel_desc, colstores);
+
+ /*
* now add tuples to pg_attribute for the attributes in our new relation.
*/
AddNewAttributeTuples(relid, new_rel_desc->rd_att, relkind,
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index 052aab1..51b5b6f 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -18,6 +18,7 @@
#include "access/htup_details.h"
#include "access/sysattr.h"
#include "catalog/catalog.h"
+#include "catalog/colstore.h"
#include "catalog/indexing.h"
#include "catalog/objectaddress.h"
#include "catalog/pg_amop.h"
@@ -132,6 +133,18 @@ static const ObjectPropertyType ObjectProperty[] =
true
},
{
+ CStoreRelationId,
+ CStoreOidIndexId,
+ CSTOREOID,
+ -1,
+ Anum_pg_cstore_cstname,
+ InvalidAttrNumber,
+ InvalidAttrNumber,
+ InvalidAttrNumber,
+ -1,
+ false
+ },
+ {
ConstraintRelationId,
ConstraintOidIndexId,
CONSTROID,
@@ -575,6 +588,10 @@ static const struct object_type_map
{
"trigger", OBJECT_TRIGGER
},
+ /* OCLASS_COLSTORE */
+ {
+ "column store", OBJECT_COLSTORE
+ },
/* OCLASS_SCHEMA */
{
"schema", OBJECT_SCHEMA
@@ -765,6 +782,7 @@ get_object_address(ObjectType objtype, List *objname, List *objargs,
case OBJECT_TRIGGER:
case OBJECT_TABCONSTRAINT:
case OBJECT_POLICY:
+ case OBJECT_COLSTORE:
address = get_object_address_relobject(objtype, objname,
&relation, missing_ok);
break;
@@ -1278,6 +1296,13 @@ get_object_address_relobject(ObjectType objtype, List *objname,
InvalidOid;
address.objectSubId = 0;
break;
+ case OBJECT_COLSTORE:
+ address.classId = CStoreRelationId;
+ address.objectId = relation ?
+ get_relation_cstore_oid(reloid, depname, missing_ok) :
+ InvalidOid;
+ address.objectSubId = 0;
+ break;
default:
elog(ERROR, "unrecognized objtype: %d", (int) objtype);
/* placate compiler, which doesn't know elog won't return */
@@ -2028,6 +2053,7 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
case OBJECT_TRIGGER:
case OBJECT_POLICY:
case OBJECT_TABCONSTRAINT:
+ case OBJECT_COLSTORE:
if (!pg_class_ownercheck(RelationGetRelid(relation), roleid))
aclcheck_error(ACLCHECK_NOT_OWNER, ACL_KIND_CLASS,
RelationGetRelationName(relation));
@@ -2791,6 +2817,30 @@ getObjectDescription(const ObjectAddress *object)
break;
}
+ case OCLASS_COLSTORE:
+ {
+ Relation cstDesc;
+ HeapTuple tup;
+ Form_pg_cstore store;
+
+ cstDesc = heap_open(CStoreRelationId, AccessShareLock);
+
+ tup = get_catalog_object_by_oid(cstDesc, object->objectId);
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "could not find tuple for column store %u",
+ object->objectId);
+
+ store = (Form_pg_cstore) GETSTRUCT(tup);
+
+ appendStringInfo(&buffer, _("column store %s on "),
+ NameStr(store->cstname));
+ getRelationDescription(&buffer, store->cstrelid);
+
+ heap_close(cstDesc, AccessShareLock);
+ break;
+ }
+
case OCLASS_TRANSFORM:
{
HeapTuple trfTup;
@@ -3544,6 +3594,10 @@ getObjectTypeDescription(const ObjectAddress *object)
appendStringInfoString(&buffer, "trigger");
break;
+ case OCLASS_COLSTORE:
+ appendStringInfoString(&buffer, "column store");
+ break;
+
case OCLASS_SCHEMA:
appendStringInfoString(&buffer, "schema");
break;
@@ -4185,6 +4239,32 @@ getObjectIdentityParts(const ObjectAddress *object,
break;
}
+ case OCLASS_COLSTORE:
+ {
+ Relation cstDesc;
+ HeapTuple tup;
+ Form_pg_cstore store;
+
+ cstDesc = heap_open(CStoreRelationId, AccessShareLock);
+
+ tup = get_catalog_object_by_oid(cstDesc, object->objectId);
+
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "could not find tuple for column store %u",
+ object->objectId);
+
+ store = (Form_pg_cstore) GETSTRUCT(tup);
+
+ appendStringInfo(&buffer, "%s on ",
+ quote_identifier(NameStr(store->cstname)));
+ getRelationIdentity(&buffer, store->cstrelid, objname);
+ if (objname)
+ *objname = lappend(*objname, pstrdup(NameStr(store->cstname)));
+
+ heap_close(cstDesc, AccessShareLock);
+ break;
+ }
+
case OCLASS_POLICY:
{
Relation polDesc;
diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c
index 3652d7b..33ba582 100644
--- a/src/backend/catalog/toasting.c
+++ b/src/backend/catalog/toasting.c
@@ -279,6 +279,7 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid,
rel->rd_rel->relowner,
tupdesc,
NIL,
+ NIL,
RELKIND_TOASTVALUE,
rel->rd_rel->relpersistence,
shared_relation,
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 7ab4874..c40a575 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -25,6 +25,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "catalog/catalog.h"
+#include "catalog/colstore.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/index.h"
@@ -667,6 +668,7 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, char relpersistence,
OldHeap->rd_rel->relowner,
OldHeapDesc,
NIL,
+ CloneColumnStores(OldHeap),
RELKIND_RELATION,
relpersistence,
false,
diff --git a/src/backend/commands/event_trigger.c b/src/backend/commands/event_trigger.c
index 3d1cb0b..d323932 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -1118,6 +1118,8 @@ EventTriggerSupportsObjectType(ObjectType obtype)
case OBJECT_USER_MAPPING:
case OBJECT_VIEW:
return true;
+ case OBJECT_COLSTORE:
+ elog(ERROR, "unsupported --- XXX fill this in");
}
return true;
}
@@ -1168,6 +1170,8 @@ EventTriggerSupportsObjectClass(ObjectClass objclass)
case OCLASS_EXTENSION:
case OCLASS_POLICY:
return true;
+ case OCLASS_COLSTORE:
+ elog(ERROR, "unsupported --- XXX fill this in");
}
return true;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 126b119..854177b 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -23,6 +23,7 @@
#include "access/xact.h"
#include "access/xlog.h"
#include "catalog/catalog.h"
+#include "catalog/colstore.h"
#include "catalog/dependency.h"
#include "catalog/heap.h"
#include "catalog/index.h"
@@ -269,7 +270,8 @@ struct DropRelationCallbackState
static void truncate_check_rel(Relation rel);
static List *MergeAttributes(List *schema, List *supers, char relpersistence,
- List **supOids, List **supconstr, int *supOidCount);
+ List **supOids, List **supconstr, int *supOidCount,
+ List **colstores);
static bool MergeCheckConstraint(List *constraints, char *name, Node *expr);
static void MergeAttributesIntoExisting(Relation child_rel, Relation parent_rel);
static void MergeConstraintsIntoExisting(Relation child_rel, Relation parent_rel);
@@ -459,6 +461,9 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
Oid tablespaceId;
Relation rel;
TupleDesc descriptor;
+ List *decl_cstores = NIL,
+ *inh_cstores = NIL,
+ *colstores;
List *inheritOids;
List *old_constraints;
bool localHasOids;
@@ -568,19 +573,43 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
ofTypeId = InvalidOid;
/*
+ * Initialize the list of column stores with the ones provided in
+ * table constraint form.
+ */
+ foreach(listptr, stmt->colstores)
+ {
+ ColumnStoreClause *clause = (ColumnStoreClause *) lfirst(listptr);
+ ColumnStoreClauseInfo *store = palloc(sizeof(ColumnStoreClauseInfo));
+
+ store->cstoreClause = clause;
+ store->attnum = InvalidAttrNumber;
+ store->attnums = NIL;
+ store->cstoreOid = InvalidOid;
+
+ decl_cstores = lappend(decl_cstores, store);
+ }
+
+ /*
* Look up inheritance ancestors and generate relation schema, including
- * inherited attributes.
+ * inherited attributes. Add column stores coming from parent rels.
*/
schema = MergeAttributes(schema, stmt->inhRelations,
stmt->relation->relpersistence,
- &inheritOids, &old_constraints, &parentOidCount);
+ &inheritOids, &old_constraints, &parentOidCount,
+ &inh_cstores);
/*
* Create a tuple descriptor from the relation schema. Note that this
* deals with column names, types, and NOT NULL constraints, but not
* default values or CHECK constraints; we handle those below.
*/
- descriptor = BuildDescForRelation(schema);
+ descriptor = BuildDescForRelation(schema, &decl_cstores);
+
+ /*
+ * Determine the column stores we need.
+ */
+ colstores = DetermineColumnStores(descriptor, decl_cstores, inh_cstores,
+ tablespaceId);
/*
* Notice that we allow OIDs here only for plain tables, even though some
@@ -662,6 +691,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId,
descriptor,
list_concat(cookedDefaults,
old_constraints),
+ colstores,
relkind,
stmt->relation->relpersistence,
false,
@@ -1360,6 +1390,7 @@ storage_name(char c)
* 'supconstr' receives a list of constraints belonging to the parents,
* updated as necessary to be valid for the child.
* 'supOidCount' is set to the number of parents that have OID columns.
+ * 'colstores' is appended ColumnStoreClauseInfo structs from parent rels.
*
* Return value:
* Completed schema list.
@@ -1405,7 +1436,8 @@ storage_name(char c)
*/
static List *
MergeAttributes(List *schema, List *supers, char relpersistence,
- List **supOids, List **supconstr, int *supOidCount)
+ List **supOids, List **supconstr, int *supOidCount,
+ List **colstores)
{
ListCell *entry;
List *inhSchema = NIL;
@@ -1503,6 +1535,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
TupleConstr *constr;
AttrNumber *newattno;
AttrNumber parent_attno;
+ List *pstores;
+ ListCell *cell;
/*
* A self-exclusive lock is needed here. If two backends attempt to
@@ -1659,6 +1693,7 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
def->collClause = NULL;
def->collOid = attribute->attcollation;
def->constraints = NIL;
+ def->cstoreClause = NULL;
def->location = -1;
inhSchema = lappend(inhSchema, def);
newattno[parent_attno - 1] = ++child_attno;
@@ -1767,6 +1802,38 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
}
}
+ /*
+ * Process column stores in the parent, using the completed
+ * newattno map.
+ */
+ pstores = RelationGetColStoreList(relation);
+ foreach(cell, pstores)
+ {
+ Oid cstoreid = lfirst_oid(cell);
+ Relation storerel;
+ Form_pg_cstore cst;
+ ColumnStoreClauseInfo *cstinfo;
+ int i;
+
+ /* AccessShare should be sufficient, since we hold lock on rel */
+ storerel = relation_open(cstoreid, AccessShareLock);
+ cst = storerel->rd_cstore;
+
+ cstinfo = palloc(sizeof(ColumnStoreClauseInfo));
+ cstinfo->attnum = InvalidAttrNumber;
+ cstinfo->cstoreClause = NULL;
+ cstinfo->cstoreOid = RelationGetRelid(storerel);
+ cstinfo->attnums = NIL;
+ for (i = 0; i < cst->cstnatts; i++)
+ cstinfo->attnums = lappend_int(cstinfo->attnums,
+ newattno[cst->cstatts.values[i]-1]);
+
+ relation_close(storerel, AccessShareLock);
+
+ *colstores = lappend(*colstores, cstinfo);
+ }
+ list_free(pstores);
+
pfree(newattno);
/*
@@ -1854,6 +1921,8 @@ MergeAttributes(List *schema, List *supers, char relpersistence,
storage_name(def->storage),
storage_name(newdef->storage))));
+ /* FIXME see about merging cstore decl here */
+
/* Mark the column as locally defined */
def->is_local = true;
/* Merge of NOT NULL constraints = OR 'em together */
diff --git a/src/backend/commands/view.c b/src/backend/commands/view.c
index efa4be1..d7b702b 100644
--- a/src/backend/commands/view.c
+++ b/src/backend/commands/view.c
@@ -170,7 +170,7 @@ DefineVirtualRelation(RangeVar *relation, List *tlist, bool replace,
* verify that the old column list is an initial prefix of the new
* column list.
*/
- descriptor = BuildDescForRelation(attrList);
+ descriptor = BuildDescForRelation(attrList, NULL);
checkViewTupleDesc(descriptor, rel->rd_att);
/*
diff --git a/src/include/access/tupdesc.h b/src/include/access/tupdesc.h
index 91b0034..e1e96bc 100644
--- a/src/include/access/tupdesc.h
+++ b/src/include/access/tupdesc.h
@@ -123,7 +123,11 @@ extern void TupleDescInitEntryCollation(TupleDesc desc,
AttrNumber attributeNumber,
Oid collationid);
-extern TupleDesc BuildDescForRelation(List *schema);
+extern void TupleDescInitEntryColStore(TupleDesc desc,
+ AttrNumber attributeNumber,
+ Oid colstore);
+
+extern TupleDesc BuildDescForRelation(List *schema, List **colstores);
extern TupleDesc BuildDescFromLists(List *names, List *types, List *typmods, List *collations);
diff --git a/src/include/catalog/colstore.h b/src/include/catalog/colstore.h
new file mode 100644
index 0000000..5dcc91a
--- /dev/null
+++ b/src/include/catalog/colstore.h
@@ -0,0 +1,41 @@
+#ifndef COLSTORE_H
+#define COLSTORE_H
+
+#include "nodes/nodes.h"
+#include "nodes/parsenodes.h"
+#include "nodes/pg_list.h"
+#include "utils/relcache.h"
+
+/*
+ * When creating a table with column store declarations, this struct
+ * carries the necessary info.
+ *
+ * We're catering for several different cases here: (a) column store
+ * declarations as column constraints (attnum and cstoreClause are both set,
+ * attnums and cstoreOid are invalid), (b) declarations as table constraint
+ * (attnum is invalid but cstoreClause is set; attnums and cstoreOid are
+ * invalid), and (c) store definitions inherited from parent relations (attnum
+ * and cstoreClause are both invalid, attnums list the attribute numbers and
+ * cstoreOid is the OID of the pg_cstore entry of the column store for the
+ * parent relation. Note that the cstatts data from the parent's entry must be
+ * ignored in favor of the attnum list given here.)
+ */
+typedef struct ColumnStoreClauseInfo
+{
+ AttrNumber attnum;
+ ColumnStoreClause *cstoreClause;
+ List *attnums;
+ Oid cstoreOid;
+} ColumnStoreClauseInfo;
+
+
+extern List *DetermineColumnStores(TupleDesc tupdesc, List *decl_cstores,
+ List *inh_cstores, Oid tablespaceId);
+extern void CreateColumnStores(Relation rel, List *colstores);
+extern List *CloneColumnStores(Relation rel);
+extern Oid get_relation_cstore_oid(Oid relid, const char *cstore_name,
+ bool missing_ok);
+extern void RemoveColstoreById(Oid cstoreOid);
+extern Oid GetColumnStoreAMByName(char *cstamname, bool missing_ok);
+
+#endif /* COLSTORE_H */
diff --git a/src/include/catalog/dependency.h b/src/include/catalog/dependency.h
index fbcf904..71a28c3 100644
--- a/src/include/catalog/dependency.h
+++ b/src/include/catalog/dependency.h
@@ -138,6 +138,7 @@ typedef enum ObjectClass
OCLASS_AMPROC, /* pg_amproc */
OCLASS_REWRITE, /* pg_rewrite */
OCLASS_TRIGGER, /* pg_trigger */
+ OCLASS_COLSTORE, /* pg_cstore */
OCLASS_SCHEMA, /* pg_namespace */
OCLASS_TSPARSER, /* pg_ts_parser */
OCLASS_TSDICT, /* pg_ts_dict */
diff --git a/src/include/catalog/heap.h b/src/include/catalog/heap.h
index e6ac394..27a065c 100644
--- a/src/include/catalog/heap.h
+++ b/src/include/catalog/heap.h
@@ -60,6 +60,7 @@ extern Oid heap_create_with_catalog(const char *relname,
Oid ownerid,
TupleDesc tupdesc,
List *cooked_constraints,
+ List *colstores,
char relkind,
char relpersistence,
bool shared_relation,
@@ -95,6 +96,12 @@ extern void InsertPgClassTuple(Relation pg_class_desc,
Datum relacl,
Datum reloptions);
+extern void AddNewAttributeTuples(Oid new_rel_oid,
+ TupleDesc tupdesc,
+ char relkind,
+ bool oidislocal,
+ int oidinhcount);
+
extern List *AddRelationNewConstraints(Relation rel,
List *newColDefaults,
List *newConstraints,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 302ba1f..7c790bf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -1401,6 +1401,7 @@ typedef enum ObjectType
OBJECT_CAST,
OBJECT_COLUMN,
OBJECT_COLLATION,
+ OBJECT_COLSTORE,
OBJECT_CONVERSION,
OBJECT_DATABASE,
OBJECT_DEFAULT,
--
2.1.4
["0012-add-psql-d-support-for-column-stores.patch" (text/x-diff)]
From 6d3fdbc4d2efde4099f2c120b85bd2996a99f463 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Tue, 18 Aug 2015 15:24:15 -0300
Subject: [PATCH 12/24] add psql \d support for column stores
---
src/bin/psql/describe.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 898f8b3..7e8da72 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1523,6 +1523,10 @@ describeOneTableDetails(const char *schemaname,
printfPQExpBuffer(&title, _("Composite type \"%s.%s\""),
schemaname, relationname);
break;
+ case 'C':
+ printfPQExpBuffer(&title, _("Column store \"%s.%s\""),
+ schemaname, relationname);
+ break;
case 'f':
printfPQExpBuffer(&title, _("Foreign table \"%s.%s\""),
schemaname, relationname);
@@ -2526,6 +2530,41 @@ describeOneTableDetails(const char *schemaname,
/* Tablespace info */
add_tablespace_footer(&cont, tableinfo.relkind, tableinfo.tablespace,
true);
+
+ /* Print column stores FIXME -- add version dependency */
+ {
+ const char *cs = _("Column stores");
+ int csw = pg_wcswidth(cs, strlen(cs), pset.encoding);
+
+ /* FIXME get rid of the braces, use JOIN ON? */
+ printfPQExpBuffer(&buf, "SELECT cstamname, cstname, ARRAY(SELECT attname FROM \
pg_attribute WHERE attrelid = cstrelid AND attnum = ANY(cstatts)) FROM pg_cstore AS \
cst, pg_class AS cl, pg_cstore_am AS am WHERE cl.oid = cst.cststoreid AND cl.relam = \
am.oid AND cstrelid = '%s'::regclass ORDER BY cstatts[0];", oid); + result = \
PSQLexec(buf.data); + if (!result)
+ goto error_return;
+ else
+ tuples = PQntuples(result);
+
+ for (i = 0; i < tuples; i++)
+ {
+ if (i == 0)
+ printfPQExpBuffer(&buf, "%s: %s USING %s (%s)",
+ cs,
+ PQgetvalue(result, i, 1),
+ PQgetvalue(result, i, 0),
+ PQgetvalue(result, i, 2));
+ else
+ printfPQExpBuffer(&buf, "%*s %s USING %s (%s)",
+ csw, "",
+ PQgetvalue(result, i, 1),
+ PQgetvalue(result, i, 0),
+ PQgetvalue(result, i, 2));
+ if (i < tuples - 1)
+ appendPQExpBufferChar(&buf, ',');
+
+ printTableAddFooter(&cont, buf.data);
+ }
+ PQclear(result);
+ }
}
/* reloptions, if verbose */
--
2.1.4
["0013-add-colstore-function-to-dbsize.patch" (text/x-diff)]
From 4a08bb604d5deddddcf1f45963dd10438b28027c Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 28 Aug 2015 15:10:20 -0300
Subject: [PATCH 13/24] add colstore function to dbsize
---
src/backend/utils/adt/dbsize.c | 64 ++++++++++++++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 2 ++
src/include/utils/builtins.h | 1 +
3 files changed, 67 insertions(+)
diff --git a/src/backend/utils/adt/dbsize.c b/src/backend/utils/adt/dbsize.c
index 82311b4..6dd8155 100644
--- a/src/backend/utils/adt/dbsize.c
+++ b/src/backend/utils/adt/dbsize.c
@@ -443,6 +443,46 @@ calculate_indexes_size(Relation rel)
return size;
}
+/*
+ * Calculate total on-disk size of all column stores attached to the given table.
+ *
+ * Can be applied safely to a column store, but you'll just get zero.
+ */
+static int64
+calculate_column_stores_size(Relation rel)
+{
+ int64 size = 0;
+
+ /*
+ * Aggregate all indexes on the given relation
+ */
+ if (rel->rd_rel->relhascstore)
+ {
+ List *cstore_oids = RelationGetColStoreList(rel);
+ ListCell *cell;
+
+ foreach(cell, cstore_oids)
+ {
+ Oid cstoreOid = lfirst_oid(cell);
+ Relation cstoreRel;
+ ForkNumber forkNum;
+
+ cstoreRel = relation_open(cstoreOid, AccessShareLock);
+
+ for (forkNum = 0; forkNum <= MAX_FORKNUM; forkNum++)
+ size += calculate_relation_size(&(cstoreRel->rd_node),
+ cstoreRel->rd_backend,
+ forkNum);
+
+ relation_close(cstoreRel, AccessShareLock);
+ }
+
+ list_free(cstore_oids);
+ }
+
+ return size;
+}
+
Datum
pg_table_size(PG_FUNCTION_ARGS)
{
@@ -481,6 +521,25 @@ pg_indexes_size(PG_FUNCTION_ARGS)
PG_RETURN_INT64(size);
}
+Datum
+pg_column_stores_size(PG_FUNCTION_ARGS)
+{
+ Oid relOid = PG_GETARG_OID(0);
+ Relation rel;
+ int64 size;
+
+ rel = try_relation_open(relOid, AccessShareLock);
+
+ if (rel == NULL)
+ PG_RETURN_NULL();
+
+ size = calculate_column_stores_size(rel);
+
+ relation_close(rel, AccessShareLock);
+
+ PG_RETURN_INT64(size);
+}
+
/*
* Compute the on-disk size of all files for the relation,
* including heap data, index data, toast data, FSM, VM.
@@ -501,6 +560,11 @@ calculate_total_relation_size(Relation rel)
*/
size += calculate_indexes_size(rel);
+ /*
+ * Add size of all attached column stores as well
+ */
+ size += calculate_column_stores_size(rel);
+
return size;
}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index ddf7c67..f0481bd 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3664,6 +3664,8 @@ DATA(insert OID = 2997 ( pg_table_size PGNSP PGUID 12 1 0 0 0 \
f f f f t f v 1 DESCR("disk space usage for the specified table, including TOAST, \
free space and visibility map"); DATA(insert OID = 2998 ( pg_indexes_size PGNSP \
PGUID 12 1 0 0 0 f f f f t f v 1 0 20 "2205" _null_ _null_ _null_ _null_ _null_ \
pg_indexes_size _null_ _null_ _null_ )); DESCR("disk space usage for all indexes \
attached to the specified table"); +DATA(insert OID = 3350 ( \
pg_column_stores_size PGNSP PGUID 12 1 0 0 0 f f f f t f v 1 0 20 "2205" _null_ \
_null_ _null_ _null_ _null_ pg_column_stores_size _null_ _null_ _null_ )); \
+DESCR("disk space usage for all column stores attached to the specified table"); \
DATA(insert OID = 2999 ( pg_relation_filenode PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 \
0 26 "2205" _null_ _null_ _null_ _null_ _null_ pg_relation_filenode _null_ _null_ \
_null_ )); DESCR("filenode identifier of relation");
DATA(insert OID = 3454 ( pg_filenode_relation PGNSP PGUID 12 1 0 0 0 f f f f t f s 2 \
0 2205 "26 26" _null_ _null_ _null_ _null_ _null_ pg_filenode_relation _null_ _null_ \
_null_ ));
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index fc1679e..aa79730 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -464,6 +464,7 @@ extern Datum pg_size_pretty(PG_FUNCTION_ARGS);
extern Datum pg_size_pretty_numeric(PG_FUNCTION_ARGS);
extern Datum pg_table_size(PG_FUNCTION_ARGS);
extern Datum pg_indexes_size(PG_FUNCTION_ARGS);
+extern Datum pg_column_stores_size(PG_FUNCTION_ARGS);
extern Datum pg_relation_filenode(PG_FUNCTION_ARGS);
extern Datum pg_filenode_relation(PG_FUNCTION_ARGS);
extern Datum pg_relation_filepath(PG_FUNCTION_ARGS);
--
2.1.4
["0014-Add-a-generic-API-for-column-stores-to-implement.patch" (text/x-diff)]
From 9ffed6aecfd6d6aa672d9cfaf9329dd68b8b10f5 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 28 Aug 2015 12:53:36 -0300
Subject: [PATCH 14/24] Add a generic API for column stores to implement
This patch introduces struct ColumnStoreRoutine, similar to FdwRoutine,
which contains function pointers that core code uses to operate on
column stores. The actual API for column stores is defined in
src/include/colstore/colstoreapi.h, and includes functions to insert and
fetch (in retail and batch modes), as well as "discard" and "prune"
elements from the store.
Implementation notes:
- There's a new execnode, ColumnStoreInfo, which is used by the executor
to access the provided routines.
---
src/backend/catalog/colstore.c | 156 +++++++++++++++++++++++++++++++++++++
src/backend/utils/cache/relcache.c | 3 +
src/include/catalog/colstore.h | 3 +
src/include/colstore/colstoreapi.h | 85 ++++++++++++++++++++
src/include/nodes/execnodes.h | 19 +++++
src/include/nodes/nodes.h | 4 +-
src/include/utils/rel.h | 3 +
7 files changed, 272 insertions(+), 1 deletion(-)
create mode 100644 src/include/colstore/colstoreapi.h
diff --git a/src/backend/catalog/colstore.c b/src/backend/catalog/colstore.c
index 553ab68..3954e1d 100644
--- a/src/backend/catalog/colstore.c
+++ b/src/backend/catalog/colstore.c
@@ -21,8 +21,10 @@
#include "catalog/heap.h"
#include "catalog/indexing.h"
#include "catalog/pg_cstore.h"
+#include "catalog/pg_cstore_am.h"
#include "catalog/pg_namespace.h"
#include "catalog/pg_tablespace.h" /* GLOBALTABLESPACE_OID */
+#include "colstore/colstoreapi.h"
#include "commands/tablespace.h"
#include "miscadmin.h"
#include "nodes/bitmapset.h"
@@ -30,6 +32,7 @@
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
#include "utils/syscache.h"
#include "utils/rel.h"
@@ -542,3 +545,156 @@ GetColumnStoreAMByName(char *cstamname, bool missing_ok)
return oid;
}
+
+/* ----------------
+ * BuildColumnStoreInfo
+ * Construct an ColumnStoreInfo record for an open column store
+ *
+ * ColumnStoreInfo stores the information about the column store that's needed
+ * by FormColumnStoreDatum, which is used for insertion of individual column
+ * store tuples. Normally we build a ColumnStoreInfo for a column store just
+ * once per command, and then use it for (potentially) many tuples.
+ * ----------------
+ */
+ColumnStoreInfo *
+BuildColumnStoreInfo(Relation cstore)
+{
+ ColumnStoreInfo *csi = makeNode(ColumnStoreInfo);
+ Form_pg_cstore cstoreStruct = cstore->rd_cstore;
+ int i;
+ int numKeys;
+
+ /* check the number of keys, and copy attr numbers into the IndexInfo */
+ numKeys = cstoreStruct->cstnatts;
+ if (numKeys < 1 || numKeys > INDEX_MAX_KEYS)
+ elog(ERROR, "invalid cstnatts %d for column store %u",
+ numKeys, RelationGetRelid(cstore));
+ csi->csi_NumColumnStoreAttrs = numKeys;
+ for (i = 0; i < numKeys; i++)
+ csi->csi_KeyAttrNumbers[i] = cstoreStruct->cstatts.values[i];
+
+ csi->csi_ColumnStoreRoutine =
+ GetColumnStoreRoutineForRelation(cstore, true);
+
+ return csi;
+}
+
+/*
+ * GetColumnStoreRoutine - call the specified column store handler routine
+ * to get its ColumnStoreRoutine struct.
+ */
+ColumnStoreRoutine *
+GetColumnStoreRoutine(Oid cstamhandler)
+{
+ Datum datum;
+ ColumnStoreRoutine *routine;
+
+ datum = OidFunctionCall0(cstamhandler);
+ routine = (ColumnStoreRoutine *) DatumGetPointer(datum);
+
+ if (routine == NULL || !IsA(routine, ColumnStoreRoutine))
+ elog(ERROR, "column store handler function %u did not return an ColumnStoreRoutine struct",
+ cstamhandler);
+
+ return routine;
+}
+
+/*
+ * GetColumnStoreHandlerByRelId - look up the handler of the column store handler
+ * for the given column store relation
+ */
+Oid
+GetColumnStoreHandlerByRelId(Oid relid)
+{
+ HeapTuple tp;
+ Form_pg_cstore_am cstform;
+ Oid cstamhandler;
+ Relation rel;
+ Oid amoid;
+
+ rel = relation_open(relid, AccessShareLock);
+ amoid = rel->rd_rel->relam;
+
+ Assert(amoid != InvalidOid);
+ Assert(rel->rd_rel->relkind == RELKIND_COLUMN_STORE);
+
+ relation_close(rel, AccessShareLock);
+
+ /* Get server OID for the foreign table. */
+ tp = SearchSysCache1(CSTOREAMOID, ObjectIdGetDatum(amoid));
+ if (!HeapTupleIsValid(tp))
+ elog(ERROR, "cache lookup failed for foreign table %u", amoid);
+ cstform = (Form_pg_cstore_am) GETSTRUCT(tp);
+ cstamhandler = cstform->cstamhandler;
+
+ /* Complain if column store has been set to NO HANDLER. */
+ if (!OidIsValid(cstamhandler))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("column store \"%s\" has no handler",
+ NameStr(cstform->cstamname))));
+
+ ReleaseSysCache(tp);
+
+ return cstamhandler;
+}
+
+/*
+ * GetColumnStoreRoutineByRelId - look up the handler of the column store, and
+ * retrieve its ColumnStoreRoutine struct.
+ */
+ColumnStoreRoutine *
+GetColumnStoreRoutineByRelId(Oid relid)
+{
+ Oid cstamhandler = GetColumnStoreHandlerByRelId(relid);
+
+ return GetColumnStoreRoutine(cstamhandler);
+}
+
+/*
+ * GetColumnStoreRoutineForRelation - look up the handler of the given column
+ * store, and retrieve its ColumnStoreRoutine struct.
+ *
+ * This function is preferred over GetColumnStoreRoutineByRelId because it
+ * caches the data in the relcache entry, saving a number of catalog lookups.
+ *
+ * If makecopy is true then the returned data is freshly palloc'd in the
+ * caller's memory context. Otherwise, it's a pointer to the relcache data,
+ * which will be lost in any relcache reset --- so don't rely on it long.
+ */
+ColumnStoreRoutine *
+GetColumnStoreRoutineForRelation(Relation relation, bool makecopy)
+{
+ ColumnStoreRoutine *cstroutine;
+ ColumnStoreRoutine *ccstroutine;
+
+ Assert(relation->rd_rel->relkind == RELKIND_COLUMN_STORE);
+ Assert(relation->rd_rel->relam != 0);
+
+ if (relation->rd_colstoreroutine == NULL)
+ {
+ /* Get the info by consulting the catalogs */
+ cstroutine = GetColumnStoreRoutineByRelId(RelationGetRelid(relation));
+
+ /* Save the data for later reuse in CacheMemoryContext */
+ ccstroutine =
+ (ColumnStoreRoutine *) MemoryContextAlloc(CacheMemoryContext,
+ sizeof(ColumnStoreRoutine));
+ memcpy(ccstroutine, cstroutine, sizeof(ColumnStoreRoutine));
+ relation->rd_colstoreroutine = ccstroutine;
+
+ /* Give back the locally palloc'd copy regardless of makecopy */
+ return cstroutine;
+ }
+
+ /* We have valid cached data --- does the caller want a copy? */
+ if (makecopy)
+ {
+ ccstroutine = (ColumnStoreRoutine *) palloc(sizeof(ColumnStoreRoutine));
+ memcpy(ccstroutine, relation->rd_colstoreroutine, sizeof(ColumnStoreRoutine));
+ return ccstroutine;
+ }
+
+ /* Only a short-lived reference is needed, so just hand back cached copy */
+ return relation->rd_colstoreroutine;
+}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 8dcf42a..3e82223 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -2054,6 +2054,8 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
MemoryContextDelete(relation->rd_rsdesc->rscxt);
if (relation->rd_fdwroutine)
pfree(relation->rd_fdwroutine);
+ if (relation->rd_colstoreroutine)
+ pfree(relation->rd_colstoreroutine);
pfree(relation);
}
@@ -4975,6 +4977,7 @@ load_relcache_init_file(bool shared)
rel->rd_exclprocs = NULL;
rel->rd_exclstrats = NULL;
rel->rd_fdwroutine = NULL;
+ rel->rd_colstoreroutine = NULL;
/*
* Reset transient-state fields in the relcache entry
diff --git a/src/include/catalog/colstore.h b/src/include/catalog/colstore.h
index 5dcc91a..638c0e5 100644
--- a/src/include/catalog/colstore.h
+++ b/src/include/catalog/colstore.h
@@ -1,6 +1,7 @@
#ifndef COLSTORE_H
#define COLSTORE_H
+#include "nodes/execnodes.h"
#include "nodes/nodes.h"
#include "nodes/parsenodes.h"
#include "nodes/pg_list.h"
@@ -38,4 +39,6 @@ extern Oid get_relation_cstore_oid(Oid relid, const char *cstore_name,
extern void RemoveColstoreById(Oid cstoreOid);
extern Oid GetColumnStoreAMByName(char *cstamname, bool missing_ok);
+extern ColumnStoreInfo *BuildColumnStoreInfo(Relation cstore);
+
#endif /* COLSTORE_H */
diff --git a/src/include/colstore/colstoreapi.h b/src/include/colstore/colstoreapi.h
new file mode 100644
index 0000000..c3625dc
--- /dev/null
+++ b/src/include/colstore/colstoreapi.h
@@ -0,0 +1,85 @@
+/*-------------------------------------------------------------------------
+ *
+ * colstoreapi.h
+ * API for column store implementations
+ *
+ * Copyright (c) 2010-2015, PostgreSQL Global Development Group
+ *
+ * src/include/colstore/colstoreapi.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COLSTOREAPI_H
+#define COLSTOREAPI_H
+
+#include "nodes/execnodes.h"
+#include "nodes/relation.h"
+
+typedef void (*ExecColumnStoreInsert_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int natts, Datum *values, bool *nulls,
+ ItemPointer tupleid);
+
+typedef void (*ExecColumnStoreBatchInsert_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int nrows, int natts, Datum **values, bool **nulls,
+ ItemPointer *tupleids);
+
+typedef void (*ExecColumnStoreFetch_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info, ItemPointer tid);
+
+typedef void (*ExecColumnStoreBatchFetch_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int ntids, ItemPointer *tids);
+
+typedef void (*ExecColumnStoreDiscard_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int ntids, ItemPointer *tids);
+
+typedef void (*ExecColumnStorePrune_function) (Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int ntids, ItemPointer *tids);
+
+/*
+ * ColumnStoreRoutine is the struct returned by a column store's handler
+ * function. It provides pointers to the callback functions needed by the
+ * planner and executor.
+ *
+ * More function pointers are likely to be added in the future. Therefore
+ * it's recommended that the handler initialize the struct with
+ * makeNode(ColumnStoreRoutine) so that all fields are set to NULL. This will
+ * ensure that no fields are accidentally left undefined.
+ *
+ * XXX Mostly a copy of FdwRoutine.
+ */
+typedef struct ColumnStoreRoutine
+{
+ NodeTag type;
+
+ /* insert a single row into the column store */
+ ExecColumnStoreInsert_function ExecColumnStoreInsert;
+
+ /* insert a batch of rows into the column store */
+ ExecColumnStoreBatchInsert_function ExecColumnStoreBatchInsert;
+
+ /* fetch values for a single row */
+ ExecColumnStoreFetch_function ExecColumnStoreFetch;
+
+ /* fetch a batch of values for a single row */
+ ExecColumnStoreBatchFetch_function ExecColumnStoreBatchFetch;
+
+ /* discard a batch of deleted rows from the column store */
+ ExecColumnStoreDiscard_function ExecColumnStoreDiscard;
+
+ /* prune the store - keep only the valid rows */
+ ExecColumnStorePrune_function ExecColumnStorePrune;
+
+} ColumnStoreRoutine;
+
+extern Oid GetColumnStoreHandlerByRelId(Oid relid);
+extern ColumnStoreRoutine *GetColumnStoreRoutine(Oid csthandler);
+extern ColumnStoreRoutine *GetColumnStoreRoutineByRelId(Oid relid);
+extern ColumnStoreRoutine *GetColumnStoreRoutineForRelation(Relation relation,
+ bool makecopy);
+
+#endif /* COLSTOREAPI_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 5796de8..5816a37 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -77,6 +77,25 @@ typedef struct IndexInfo
} IndexInfo;
/* ----------------
+ * ColumnStoreInfo information
+ *
+ * this struct holds the information needed to construct new column store
+ * entries for a particular column store.
+ *
+ * NumColumnStoreAttrs number of columns in this column store
+ * KeyAttrNumbers underlying-rel attribute numbers used as keys
+ * ColumnStoreRoutine ColumnStore callback functions
+ * ----------------
+ */
+typedef struct ColumnStoreInfo
+{
+ NodeTag type;
+ int csi_NumColumnStoreAttrs;
+ AttrNumber csi_KeyAttrNumbers[INDEX_MAX_KEYS];
+ struct ColumnStoreRoutine *csi_ColumnStoreRoutine;
+} ColumnStoreInfo;
+
+/* ----------------
* ExprContext_CB
*
* List of callbacks to be called at ExprContext shutdown.
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 53e0a0b..6950e87 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -37,6 +37,7 @@ typedef enum NodeTag
T_ResultRelInfo,
T_EState,
T_TupleTableSlot,
+ T_ColumnStoreInfo,
/*
* TAGS FOR PLAN NODES (plannodes.h)
@@ -455,7 +456,8 @@ typedef enum NodeTag
T_TIDBitmap, /* in nodes/tidbitmap.h */
T_InlineCodeBlock, /* in nodes/parsenodes.h */
T_FdwRoutine, /* in foreign/fdwapi.h */
- T_TsmRoutine /* in access/tsmapi.h */
+ T_TsmRoutine, /* in access/tsmapi.h */
+ T_ColumnStoreRoutine /* in colstore/colstoreapi.h */
} NodeTag;
/*
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h
index 99c7ef6..29a24b0 100644
--- a/src/include/utils/rel.h
+++ b/src/include/utils/rel.h
@@ -181,6 +181,9 @@ typedef struct RelationData
/* use "struct" here to avoid needing to include htup.h: */
struct HeapTupleData *rd_cstoretuple; /* all of pg_cstore tuple */
+ /* use "struct" here to avoid needing to include colstoreapi.h: */
+ struct ColumnStoreRoutine *rd_colstoreroutine; /* cached function pointers, or NULL */
+
/*
* Hack for CLUSTER, rewriting ALTER TABLE, etc: when writing a new
* version of a table, we need to make any toast pointers inserted into it
--
2.1.4
["0015-New-command-CREATE-COLUMN-STORE-ACCESS-METHOD.patch" (text/x-diff)]
From b53ff348bfbd0f6baa26dcb15847a59867f6988b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Fri, 28 Aug 2015 22:21:31 -0300
Subject: [PATCH 15/24] New command: CREATE COLUMN STORE ACCESS METHOD
This allows access methods to be created using ad-hoc DDL.
Whether to use this exact wording is not clear. Perhaps we want CREATE
ACCESS METHOD (i.e. make it more generic), so that this can be used by
sequence access methods too? There would be a clause to indicate what
type of access method we're creating.
A further question is whether we actually need this to be DDL at all.
We've discussed that it's not trivial to add further access methods of
any kind given the lack of support for extensions to write and replay
WAL. That makes the accessibility to create access methods pretty much
useless, until we fix that problem.
Implementation notes:
- This makes METHOD an unreserved keyword.
- Creates a new pseudo type cstore_handler
- Creates a new parse node CreateColumnStoreAMStmt.
- There's no ALTER/DROP support yet.
- ProcessUtilitySlow is not properly capturing data for event triggers.
---
src/backend/commands/Makefile | 2 +-
src/backend/commands/colstorecmds.c | 177 +++++++++++++++++++++++++++++++++++
src/backend/commands/event_trigger.c | 1 +
src/backend/nodes/copyfuncs.c | 14 +++
src/backend/nodes/equalfuncs.c | 12 +++
src/backend/parser/gram.y | 41 +++++++-
src/backend/tcop/utility.c | 9 ++
src/backend/utils/adt/pseudotypes.c | 27 ++++++
src/include/catalog/pg_proc.h | 4 +
src/include/catalog/pg_type.h | 2 +
src/include/commands/defrem.h | 1 +
src/include/nodes/nodes.h | 1 +
src/include/nodes/parsenodes.h | 13 +++
src/include/parser/kwlist.h | 1 +
src/include/utils/builtins.h | 2 +
15 files changed, 304 insertions(+), 3 deletions(-)
create mode 100644 src/backend/commands/colstorecmds.c
diff --git a/src/backend/commands/Makefile b/src/backend/commands/Makefile
index b1ac704..d2bb636 100644
--- a/src/backend/commands/Makefile
+++ b/src/backend/commands/Makefile
@@ -14,7 +14,7 @@ include $(top_builddir)/src/Makefile.global
OBJS = aggregatecmds.o alter.o analyze.o async.o cluster.o comment.o \
collationcmds.o constraint.o conversioncmds.o copy.o createas.o \
- dbcommands.o define.o discard.o dropcmds.o \
+ colstorecmds.o dbcommands.o define.o discard.o dropcmds.o \
event_trigger.o explain.o extension.o foreigncmds.o functioncmds.o \
indexcmds.o lockcmds.o matview.o operatorcmds.o opclasscmds.o \
policy.o portalcmds.o prepare.o proclang.o \
diff --git a/src/backend/commands/colstorecmds.c \
b/src/backend/commands/colstorecmds.c new file mode 100644
index 0000000..eabcced
--- /dev/null
+++ b/src/backend/commands/colstorecmds.c
@@ -0,0 +1,177 @@
+/*-------------------------------------------------------------------------
+ *
+ * colstorecmds.c
+ * column store creation/manipulation commands
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/commands/colstorecmds.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/heapam.h"
+#include "access/htup_details.h"
+#include "access/reloptions.h"
+#include "access/xact.h"
+#include "catalog/colstore.h"
+#include "catalog/dependency.h"
+#include "catalog/indexing.h"
+#include "catalog/objectaccess.h"
+#include "catalog/pg_cstore_am.h"
+#include "catalog/pg_proc.h"
+#include "catalog/pg_type.h"
+#include "commands/defrem.h"
+#include "colstore/colstoreapi.h"
+#include "miscadmin.h"
+#include "parser/parse_func.h"
+#include "tcop/utility.h"
+#include "utils/acl.h"
+#include "utils/builtins.h"
+#include "utils/lsyscache.h"
+#include "utils/rel.h"
+#include "utils/syscache.h"
+
+
+/*
+ * Convert a handler function name passed from the parser to an Oid.
+ */
+static Oid
+lookup_cstore_handler_func(DefElem *handler)
+{
+ Oid handlerOid;
+ Oid funcargtypes[1];
+
+ if (handler == NULL || handler->arg == NULL)
+ return InvalidOid;
+
+ /* handlers have no arguments */
+ handlerOid = LookupFuncName((List *) handler->arg, 0, funcargtypes, false);
+
+ /* check that handler has correct return type */
+ if (get_func_rettype(handlerOid) != CSTORE_HANDLEROID)
+ ereport(ERROR,
+ (errcode(ERRCODE_WRONG_OBJECT_TYPE),
+ errmsg("function %s must return type \"cstore_handler\"",
+ NameListToString((List *) handler->arg))));
+
+ return handlerOid;
+}
+
+/*
+ * Process function options of CREATE COLUMN STORE ACCESS METHOD
+ */
+static void
+parse_func_options(List *func_options,
+ bool *handler_given, Oid *csthandler)
+{
+ ListCell *cell;
+
+ *handler_given = false;
+ *csthandler = InvalidOid;
+
+ foreach(cell, func_options)
+ {
+ DefElem *def = (DefElem *) lfirst(cell);
+
+ if (strcmp(def->defname, "handler") == 0)
+ {
+ if (*handler_given)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("conflicting or redundant options")));
+ *handler_given = true;
+ *csthandler = lookup_cstore_handler_func(def);
+ }
+ else
+ elog(ERROR, "option \"%s\" not recognized",
+ def->defname);
+ }
+}
+
+/*
+ * Create a column store access method
+ */
+ObjectAddress
+CreateColumnStoreAM(CreateColumnStoreAMStmt *stmt)
+{
+ Relation rel;
+ Datum values[Natts_pg_cstore_am];
+ bool nulls[Natts_pg_cstore_am];
+ HeapTuple tuple;
+ Oid cstamId;
+ bool handler_given;
+ Oid csthandler;
+ ObjectAddress myself;
+ ObjectAddress referenced;
+
+ rel = heap_open(CStoreAmRelationId, RowExclusiveLock);
+
+ /* Must be super user */
+ if (!superuser())
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied to create column store access method \"%s\"",
+ stmt->cstamname),
+ errhint("Must be superuser to create a column store access method.")));
+
+ /*
+ * Check that there is no other column store AM by this name.
+ */
+ if (GetColumnStoreAMByName(stmt->cstamname, true) != InvalidOid)
+ ereport(ERROR,
+ (errcode(ERRCODE_DUPLICATE_OBJECT),
+ errmsg("column store access method \"%s\" already exists",
+ stmt->cstamname)));
+
+ /*
+ * Insert tuple into pg_cstore_am.
+ */
+ memset(values, 0, sizeof(values));
+ memset(nulls, false, sizeof(nulls));
+
+ values[Anum_pg_cstore_am_cstamname - 1] =
+ DirectFunctionCall1(namein, CStringGetDatum(stmt->cstamname));
+
+ /* Lookup handler and validator functions, if given */
+ parse_func_options(stmt->func_options, &handler_given, &csthandler);
+
+ /* XXX ereport */
+ if (!handler_given)
+ elog(ERROR, "column store access method requires HANDLER option");
+
+ values[Anum_pg_cstore_am_cstamhandler - 1] = ObjectIdGetDatum(csthandler);
+
+ tuple = heap_form_tuple(rel->rd_att, values, nulls);
+
+ cstamId = simple_heap_insert(rel, tuple);
+ CatalogUpdateIndexes(rel, tuple);
+
+ heap_freetuple(tuple);
+
+ /* record dependencies */
+ myself.classId = CStoreAmRelationId;
+ myself.objectId = cstamId;
+ myself.objectSubId = 0;
+
+ if (OidIsValid(csthandler))
+ {
+ referenced.classId = ProcedureRelationId;
+ referenced.objectId = csthandler;
+ referenced.objectSubId = 0;
+ recordDependencyOn(&myself, &referenced, DEPENDENCY_NORMAL);
+ }
+
+ /* dependency on extension */
+ recordDependencyOnCurrentExtension(&myself, false);
+
+ /* Post creation hook for new column store access method */
+ InvokeObjectPostCreateHook(CStoreAmRelationId, cstamId, 0);
+
+ heap_close(rel, RowExclusiveLock);
+
+ return myself;
+}
diff --git a/src/backend/commands/event_trigger.c \
b/src/backend/commands/event_trigger.c index d323932..0488ea9 100644
--- a/src/backend/commands/event_trigger.c
+++ b/src/backend/commands/event_trigger.c
@@ -90,6 +90,7 @@ static event_trigger_support_data event_trigger_support[] = {
{"CAST", true},
{"CONSTRAINT", true},
{"COLLATION", true},
+ {"COLUMN STORE ACCESS METHOD", true},
{"CONVERSION", true},
{"DATABASE", false},
{"DOMAIN", true},
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index ae7ca30..3aaf221 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -3811,6 +3811,17 @@ _copyImportForeignSchemaStmt(const ImportForeignSchemaStmt \
*from) return newnode;
}
+static CreateColumnStoreAMStmt *
+_copyCreateColumnStoreAMStmt(const CreateColumnStoreAMStmt *from)
+{
+ CreateColumnStoreAMStmt *newnode = makeNode(CreateColumnStoreAMStmt);
+
+ COPY_STRING_FIELD(cstamname);
+ COPY_NODE_FIELD(func_options);
+
+ return newnode;
+}
+
static CreateTransformStmt *
_copyCreateTransformStmt(const CreateTransformStmt *from)
{
@@ -4785,6 +4796,9 @@ copyObject(const void *from)
case T_ImportForeignSchemaStmt:
retval = _copyImportForeignSchemaStmt(from);
break;
+ case T_CreateColumnStoreAMStmt:
+ retval = _copyCreateColumnStoreAMStmt(from);
+ break;
case T_CreateTransformStmt:
retval = _copyCreateTransformStmt(from);
break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 18c2119..6db8fef 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -1832,6 +1832,15 @@ _equalImportForeignSchemaStmt(const ImportForeignSchemaStmt \
*a, const ImportFore }
static bool
+_equalCreateColumnStoreAMStmt(const CreateColumnStoreAMStmt *a, const \
CreateColumnStoreAMStmt *b) +{
+ COMPARE_STRING_FIELD(cstamname);
+ COMPARE_NODE_FIELD(func_options);
+
+ return true;
+}
+
+static bool
_equalCreateTransformStmt(const CreateTransformStmt *a, const CreateTransformStmt \
*b) {
COMPARE_SCALAR_FIELD(replace);
@@ -3140,6 +3149,9 @@ equal(const void *a, const void *b)
case T_ImportForeignSchemaStmt:
retval = _equalImportForeignSchemaStmt(a, b);
break;
+ case T_CreateColumnStoreAMStmt:
+ retval = _equalCreateColumnStoreAMStmt(a, b);
+ break;
case T_CreateTransformStmt:
retval = _equalCreateTransformStmt(a, b);
break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 1ec1426..cd308fb 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -264,7 +264,7 @@ static Node *makeRecursiveViewSelect(char *relname, List \
*aliases, Node *query); DeallocateStmt PrepareStmt ExecuteStmt
DropOwnedStmt ReassignOwnedStmt
AlterTSConfigurationStmt AlterTSDictionaryStmt
- CreateMatViewStmt RefreshMatViewStmt
+ CreateMatViewStmt RefreshMatViewStmt CreateColumnStoreAMStmt
%type <node> select_no_parens select_with_parens select_clause
simple_select values_clause
@@ -379,6 +379,9 @@ static Node *makeRecursiveViewSelect(char *relname, List \
*aliases, Node *query); %type <list> opt_fdw_options fdw_options
%type <defelt> fdw_option
+%type <list> opt_cstam_options cstam_options
+%type <defelt> cstam_option
+
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
@@ -605,7 +608,7 @@ static Node *makeRecursiveViewSelect(char *relname, List \
*aliases, Node *query); LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD \
LOCAL LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
- MAPPING MATCH MATERIALIZED MAXVALUE MINUTE_P MINVALUE MODE MONTH_P MOVE
+ MAPPING MATCH MATERIALIZED MAXVALUE METHOD MINUTE_P MINVALUE MODE MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEXT NO NONE
NOT NOTHING NOTIFY NOTNULL NOWAIT NULL_P NULLIF
@@ -793,6 +796,7 @@ stmt :
| CreateAsStmt
| CreateAssertStmt
| CreateCastStmt
+ | CreateColumnStoreAMStmt
| CreateConversionStmt
| CreateDomainStmt
| CreateExtensionStmt
@@ -4212,6 +4216,38 @@ opt_fdw_options:
| /*EMPTY*/ { $$ = NIL; }
;
+
+/*****************************************************************************
+ *
+ * QUERY:
+ * CREATE COLUMN STORE ACCESS METHOD name options
+ *
+ *****************************************************************************/
+
+CreateColumnStoreAMStmt: CREATE COLUMN STORE ACCESS METHOD name opt_cstam_options
+ {
+ CreateColumnStoreAMStmt *n = makeNode(CreateColumnStoreAMStmt);
+ n->cstamname = $6;
+ n->func_options = $7;
+ $$ = (Node *) n;
+ }
+ ;
+
+cstam_option:
+ HANDLER handler_name { $$ = makeDefElem("handler", (Node *)$2); }
+ | NO HANDLER { $$ = makeDefElem("handler", NULL); }
+ ;
+
+cstam_options:
+ cstam_option { $$ = list_make1($1); }
+ | cstam_options cstam_option { $$ = lappend($1, $2); }
+ ;
+
+opt_cstam_options:
+ cstam_options { $$ = $1; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
/*****************************************************************************
*
* QUERY :
@@ -13787,6 +13823,7 @@ unreserved_keyword:
| MATCH
| MATERIALIZED
| MAXVALUE
+ | METHOD
| MINUTE_P
| MINVALUE
| MODE
diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c
index e81bbc6..711c64f 100644
--- a/src/backend/tcop/utility.c
+++ b/src/backend/tcop/utility.c
@@ -209,6 +209,7 @@ check_xact_readonly(Node *parsetree)
case T_CreateForeignTableStmt:
case T_ImportForeignSchemaStmt:
case T_SecLabelStmt:
+ case T_CreateColumnStoreAMStmt:
PreventCommandIfReadOnly(CreateCommandTag(parsetree));
PreventCommandIfParallelMode(CreateCommandTag(parsetree));
break;
@@ -1520,6 +1521,10 @@ ProcessUtilitySlow(Node *parsetree,
address = ExecSecLabelStmt((SecLabelStmt *) parsetree);
break;
+ case T_CreateColumnStoreAMStmt:
+ CreateColumnStoreAM((CreateColumnStoreAMStmt *) parsetree);
+ break;
+
default:
elog(ERROR, "unrecognized node type: %d",
(int) nodeTag(parsetree));
@@ -2665,6 +2670,10 @@ CreateCommandTag(Node *parsetree)
}
break;
+ case T_CreateColumnStoreAMStmt:
+ tag = "CREATE COLUMN STORE ACCESS METHOD";
+ break;
+
default:
elog(WARNING, "unrecognized node type: %d",
(int) nodeTag(parsetree));
diff --git a/src/backend/utils/adt/pseudotypes.c \
b/src/backend/utils/adt/pseudotypes.c index 5b809aa..7680777 100644
--- a/src/backend/utils/adt/pseudotypes.c
+++ b/src/backend/utils/adt/pseudotypes.c
@@ -401,6 +401,33 @@ tsm_handler_out(PG_FUNCTION_ARGS)
/*
+ * cstore_handler_in - input routine for pseudo-type CSTORE_HANDLER.
+ */
+Datum
+cstore_handler_in(PG_FUNCTION_ARGS)
+{
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot accept a value of type cstore_handler")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+/*
+ * cstore_handler_out - output routine for pseudo-type CSTORE_HANDLER.
+ */
+Datum
+cstore_handler_out(PG_FUNCTION_ARGS)
+{
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("cannot display a value of type cstore_handler")));
+
+ PG_RETURN_VOID(); /* keep compiler quiet */
+}
+
+
+/*
* internal_in - input routine for pseudo-type INTERNAL.
*/
Datum
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index f0481bd..c5befcb 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3740,6 +3740,10 @@ DATA(insert OID = 3311 ( tsm_handler_in PGNSP PGUID 12 1 0 0 \
0 f f f f f f i 1 DESCR("I/O");
DATA(insert OID = 3312 ( tsm_handler_out PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 \
2275 "3310" _null_ _null_ _null_ _null_ _null_ tsm_handler_out _null_ _null_ _null_ \
)); DESCR("I/O");
+DATA(insert OID = 3352 ( cstore_handler_in PGNSP PGUID 12 1 0 0 0 f f f f f f i 1 0 \
3351 "2275" _null_ _null_ _null_ _null_ _null_ cstore_handler_in _null_ _null_ _null_ \
)); +DESCR("I/O");
+DATA(insert OID = 3353 ( cstore_handler_out PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 \
0 2275 "3351" _null_ _null_ _null_ _null_ _null_ cstore_handler_out _null_ _null_ \
_null_ )); +DESCR("I/O");
/* tablesample method handlers */
DATA(insert OID = 3313 ( bernoulli PGNSP PGUID 12 1 0 0 0 f f f f t f v 1 0 3310 \
"2281" _null_ _null_ _null_ _null_ _null_ tsm_bernoulli_handler _null_ _null_ _null_ \
));
diff --git a/src/include/catalog/pg_type.h b/src/include/catalog/pg_type.h
index 7dc95c8..716d487 100644
--- a/src/include/catalog/pg_type.h
+++ b/src/include/catalog/pg_type.h
@@ -696,6 +696,8 @@ DATA(insert OID = 3115 ( fdw_handler PGNSP PGUID 4 t p P f t \
\054 0 0 0 fdw_han #define FDW_HANDLEROID 3115
DATA(insert OID = 3310 ( tsm_handler PGNSP PGUID 4 t p P f t \054 0 0 0 \
tsm_handler_in tsm_handler_out - - - - - i p f 0 -1 0 0 _null_ _null_ _null_ )); \
#define TSM_HANDLEROID 3310 +DATA(insert OID = 3351 ( cstore_handler PGNSP PGUID 4 t \
p P f t \054 0 0 0 cstore_handler_in cstore_handler_out - - - - - i p f 0 -1 0 0 \
_null_ _null_ _null_ )); +#define CSTORE_HANDLEROID 3351
DATA(insert OID = 3831 ( anyrange PGNSP PGUID -1 f p P f t \054 0 0 0 anyrange_in \
anyrange_out - - - - - d x f 0 -1 0 0 _null_ _null_ _null_ )); #define \
ANYRANGEOID 3831
diff --git a/src/include/commands/defrem.h b/src/include/commands/defrem.h
index adae296..cd4bed3 100644
--- a/src/include/commands/defrem.h
+++ b/src/include/commands/defrem.h
@@ -132,6 +132,7 @@ extern Oid RemoveUserMapping(DropUserMappingStmt *stmt);
extern void RemoveUserMappingById(Oid umId);
extern void CreateForeignTable(CreateForeignTableStmt *stmt, Oid relid);
extern void ImportForeignSchema(ImportForeignSchemaStmt *stmt);
+extern ObjectAddress CreateColumnStoreAM(CreateColumnStoreAMStmt *stmt);
extern Datum transformGenericOptions(Oid catalogId,
Datum oldOptions,
List *options,
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 6950e87..97efd24 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -381,6 +381,7 @@ typedef enum NodeTag
T_CreatePolicyStmt,
T_AlterPolicyStmt,
T_CreateTransformStmt,
+ T_CreateColumnStoreAMStmt,
/*
* TAGS FOR PARSE TREE NODES (parsenodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 7c790bf..cbf7c90 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2057,6 +2057,19 @@ typedef struct ImportForeignSchemaStmt
List *options; /* list of options to pass to FDW */
} ImportForeignSchemaStmt;
+
+/* ----------------------
+ * Create COLUMN STORE ACCESS METHOD Statements
+ * ----------------------
+ */
+
+typedef struct CreateColumnStoreAMStmt
+{
+ NodeTag type;
+ char *cstamname; /* column store AM name */
+ List *func_options; /* HANDLER option */
+} CreateColumnStoreAMStmt;
+
/*----------------------
* Create POLICY Statement
*----------------------
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 3d499c1..59e2996 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -239,6 +239,7 @@ PG_KEYWORD("mapping", MAPPING, UNRESERVED_KEYWORD)
PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD)
PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD)
PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD)
+PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD)
PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD)
PG_KEYWORD("minvalue", MINVALUE, UNRESERVED_KEYWORD)
PG_KEYWORD("mode", MODE, UNRESERVED_KEYWORD)
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index aa79730..75709d0 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -569,6 +569,8 @@ extern Datum fdw_handler_in(PG_FUNCTION_ARGS);
extern Datum fdw_handler_out(PG_FUNCTION_ARGS);
extern Datum tsm_handler_in(PG_FUNCTION_ARGS);
extern Datum tsm_handler_out(PG_FUNCTION_ARGS);
+extern Datum cstore_handler_in(PG_FUNCTION_ARGS);
+extern Datum cstore_handler_out(PG_FUNCTION_ARGS);
extern Datum internal_in(PG_FUNCTION_ARGS);
extern Datum internal_out(PG_FUNCTION_ARGS);
extern Datum opaque_in(PG_FUNCTION_ARGS);
--
2.1.4
["0016-First-column-store-implementation-colstore_dummy.patch" (text/x-diff)]
From c399dfd15682a69a787833acd68930892f66fe88 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Sun, 30 Aug 2015 01:21:19 -0300
Subject: [PATCH 16/24] First column store implementation, colstore_dummy
This is a very simple module implementing the column store API.
Implementation notes:
- New subdir src/backend/colstore/
- The whole module is contained within a single file, colstore_dummy.c.
We might want to create subdirs so that larger modules (>1 file) don't
have their files mixed with other modules.
---
src/backend/Makefile | 2 +-
src/backend/colstore/Makefile | 17 ++
src/backend/colstore/colstore_dummy.c | 326 ++++++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.h | 4 +
src/include/colstore/colstore_dummy.h | 161 +++++++++++++++++
5 files changed, 509 insertions(+), 1 deletion(-)
create mode 100644 src/backend/colstore/Makefile
create mode 100644 src/backend/colstore/colstore_dummy.c
create mode 100644 src/include/colstore/colstore_dummy.h
diff --git a/src/backend/Makefile b/src/backend/Makefile
index 98b978f..66adc59 100644
--- a/src/backend/Makefile
+++ b/src/backend/Makefile
@@ -19,7 +19,7 @@ include $(top_builddir)/src/Makefile.global
SUBDIRS = access bootstrap catalog parser commands executor foreign lib libpq \
main nodes optimizer port postmaster regex replication rewrite \
- storage tcop tsearch utils $(top_builddir)/src/timezone
+ colstore storage tcop tsearch utils $(top_builddir)/src/timezone
include $(srcdir)/common.mk
diff --git a/src/backend/colstore/Makefile b/src/backend/colstore/Makefile
new file mode 100644
index 0000000..9f2fbd4
--- /dev/null
+++ b/src/backend/colstore/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for colstore
+#
+# IDENTIFICATION
+# src/backend/colstore/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/colstore
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = colstore_dummy.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/colstore/colstore_dummy.c \
b/src/backend/colstore/colstore_dummy.c new file mode 100644
index 0000000..03238e0
--- /dev/null
+++ b/src/backend/colstore/colstore_dummy.c
@@ -0,0 +1,326 @@
+/*------------------------------------------------------------------------
+ * colstore_dummy.c
+ * Simple column store implementation for POSTGRES
+ *
+ * Copyright (c) 2015, PostgreSQL Global Development Group
+ *
+ * src/backend/colstore/colstore_dummy.c
+ *
+ *------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "colstore/colstoreapi.h"
+#include "colstore/colstore_dummy.h"
+#include "storage/bufmgr.h"
+#include "storage/lmgr.h"
+#include "storage/smgr.h"
+#include "utils/rel.h"
+
+PG_FUNCTION_INFO_V1(cstore_dummy_handler);
+
+static void cstore_dummy_insert(Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int natts, Datum *values, bool *nulls,
+ ItemPointer tupleid);
+static void cstore_dummy_batch_insert(Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int nrows, int natts, Datum **values, bool **nulls,
+ ItemPointer *tupleids);
+static Buffer get_colstore_buffer(Relation rel, Relation colstore);
+static int ColumnarPageGetFreeItems(ColumnarPage page);
+
+
+Datum
+cstore_dummy_handler(PG_FUNCTION_ARGS)
+{
+ ColumnStoreRoutine *routine = makeNode(ColumnStoreRoutine);
+
+ routine->ExecColumnStoreInsert = cstore_dummy_insert;
+ routine->ExecColumnStoreBatchInsert = cstore_dummy_batch_insert;
+
+ PG_RETURN_POINTER(routine);
+}
+
+static void
+cstore_dummy_insert(Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int natts, Datum *values, bool *nulls,
+ ItemPointer tupleid)
+{
+ int i;
+ Buffer buffer = get_colstore_buffer(rel, colstorerel);
+ ColumnarPage page = BufferGetColumnarPage(buffer);
+ ColumnarPageHeader header = (ColumnarPageHeader)page;
+
+ /* how many free item slots are on the current page? */
+ int nitems = ColumnarPageGetFreeItems(page);
+
+ Assert(nitems > 0);
+
+ for (i = 0; i < header->pd_ncolumns; i++)
+ {
+ int byteIdx = (header->pd_nitems) / 8;
+ int bitIdx = (header->pd_nitems) % 8;
+
+ /* copy the data in place */
+ memcpy(PageGetColumnDataNext(page, i),
+ &values[i], PageGetColumnAttlen(page, i));
+
+ PageGetColumnDataAddBytes(page, i, PageGetColumnAttlen(page,i));
+
+ /* set the NULL bitmap */
+ *(PageGetColumnNulls(page, i) + byteIdx) &= (0x01 << bitIdx);
+ PageGetColumnNullsSetBytes(page, i, (byteIdx+1));
+ }
+
+ /* now set tuple ID */
+ memcpy(PageGetNextTupleId(page), tupleid, sizeof(ItemPointerData));
+
+ /* FIXME update min/max TID */
+
+ /* update number of items on the page */
+ header->pd_nitems += 1;
+
+ Assert(header->pd_nitems <= header->pd_maxitems);
+
+ PageSetChecksumInplace((Page)page, BufferGetBlockNumber(buffer));
+
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buffer);
+}
+
+static void
+cstore_dummy_batch_insert(Relation rel,
+ Relation colstorerel, ColumnStoreInfo *info,
+ int nrows, int natts, Datum **values, bool **nulls,
+ ItemPointer *tupleids)
+{
+ int i,
+ j;
+ int first = 0;
+
+ while (first < nrows)
+ {
+ Buffer buffer = get_colstore_buffer(rel, colstorerel);
+ ColumnarPage page = BufferGetColumnarPage(buffer);
+ ColumnarPageHeader header = (ColumnarPageHeader)page;
+
+ /* how many free item slots are on the current page? */
+ int nitems = ColumnarPageGetFreeItems(page);
+
+ Assert(nitems > 0);
+
+ nitems = (nitems < (nrows - first)) ? nitems : (nrows - first);
+
+ for (i = 0; i < header->pd_ncolumns; i++)
+ {
+ for (j = 0; j < nitems; j++)
+ {
+ int byteIdx = (header->pd_nitems + j) / 8;
+ int bitIdx = (header->pd_nitems + j) % 8;
+
+ /* copy the data in place */
+ memcpy(PageGetColumnDataNext(page, i),
+ &values[i][first+j], PageGetColumnAttlen(page, i));
+
+ PageGetColumnDataAddBytes(page, i, PageGetColumnAttlen(page,i));
+
+ /* set the NULL bitmap */
+ *(PageGetColumnNulls(page, i) + byteIdx) &= (0x01 << bitIdx);
+ PageGetColumnNullsSetBytes(page, i, (byteIdx+1));
+ }
+ }
+
+ /* now set tuple IDs */
+ for (i = 0; i < nitems; i++)
+ memcpy(PageGetNextTupleId(page) + i * sizeof(ItemPointerData),
+ &tupleids[i], sizeof(ItemPointerData));
+
+ /* FIXME update min/max TID */
+
+ /* update number of items on the page */
+ header->pd_nitems += nitems;
+ first += nitems;
+
+ Assert(header->pd_nitems <= header->pd_maxitems);
+
+ PageSetChecksumInplace((Page)page, BufferGetBlockNumber(buffer));
+
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buffer);
+ }
+}
+
+void
+ColumnarPageInit(ColumnarPage page, Size pageSize, Relation rel)
+{
+ int i;
+ ColumnarPageHeader header;
+ TupleDesc tupdesc;
+ Size itemsize;
+ Size freespace;
+ int maxtuples = 0;
+ int natts;
+ int nnulls = 0;
+
+ /* zero the page first */
+ memset(page, 0, pageSize);
+
+ tupdesc = RelationGetDescr(rel);
+ natts = tupdesc->natts;
+
+ header = (ColumnarPageHeader)page;
+
+ header->pd_ncolumns = natts;
+ header->pd_flags = 0;
+
+ /*
+ * Set the pd_lower/upper/special in a sensible way - we don't use special
+ * space, so we'll set pd_special to pageSize. And we'll set both pd_lower
+ * and pd_upper right after the column info array, So the page seems to be
+ * entirely full (pd_upper-pd_lower==0).
+ *
+ * XXX An alternative might be to store the column info structs in the
+ * special section, not sure if that's better.
+ */
+ header->pd_lower = offsetof(ColumnarPageHeaderData, pd_columns)
+ - natts * sizeof(ColumnInfoData);
+ header->pd_upper = header->pd_lower;
+ header->pd_special = pageSize; /* no special */
+
+ PageSetPageSizeAndVersion(page, pageSize, PG_PAGE_LAYOUT_VERSION);
+
+ /* we need an item pointer for each 'row' */
+ itemsize = sizeof(ItemPointerData);
+
+ /* compute size of a single 'row' added to the page */
+ for (i = 0; i < natts; i++)
+ {
+ if (tupdesc->attrs[i]->attlen < 0)
+ elog(ERROR, "variable-length data types not supported yet");
+
+ itemsize += tupdesc->attrs[i]->attlen;
+
+ header->pd_columns[i].attnum = tupdesc->attrs[i]->attnum;
+ header->pd_columns[i].attlen = tupdesc->attrs[i]->attlen;
+ header->pd_columns[i].atttypid = tupdesc->attrs[i]->atttypid;
+ header->pd_columns[i].attnotnull = tupdesc->attrs[i]->attnotnull;
+
+ nnulls += (header->pd_columns[i].attnotnull) ? 0 : 1;
+ }
+
+ freespace = pageSize - offsetof(ColumnarPageHeaderData, pd_columns)
+ - natts * sizeof(ColumnInfoData);
+
+ /*
+ * We'll do a bit arithmetics magic, because we need to include NULLs,
+ * because 8 rows needs 1 byte in NULL bitmap
+ */
+ maxtuples = 8 * freespace / (itemsize * 8 + nnulls);
+
+ /*
+ * We haven't considered alignment yet, so let's see if we fit on the page
+ * (and if not, decrement the number of items until we do).
+ */
+ while (true)
+ {
+ Size offset = offsetof(ColumnarPageHeaderData, pd_columns)
+ + natts * sizeof(ColumnInfoData);
+
+ for (i = 0; i < natts; i++)
+ {
+ offset = MAXALIGN(offset);
+ header->pd_columns[i].data_start = offset;
+
+ /* space for data */
+ offset += maxtuples * tupdesc->attrs[i]->attlen;
+
+ offset = MAXALIGN(offset);
+ header->pd_columns[i].null_start = offset;
+
+ /* NULL bitmap size */
+ offset += (maxtuples + 7) / 8;
+ }
+
+ /* and finally one item pointer for each row */
+ offset = MAXALIGN(offset);
+
+ header->pd_tupleids = offset;
+ offset += maxtuples * sizeof(ItemPointerData);
+
+ /* if we fit onto a page, terminate, otherwise decrement maxtuples */
+ if (offset <= pageSize)
+ break;
+
+ maxtuples--;
+ }
+
+ /* remember the max number of tuples */
+ header->pd_maxitems = maxtuples;
+
+ return;
+}
+
+static Buffer
+get_colstore_buffer(Relation rel, Relation colstore)
+{
+ Buffer buffer;
+ ColumnarPage page;
+ BlockNumber targetBlock = InvalidBlockNumber;
+ bool needLock = !RELATION_IS_LOCAL(rel); /* check the parent */
+ BlockNumber nblocks = RelationGetNumberOfBlocks(colstore);
+
+ /* we'll always try the last block first, and then possibly extend */
+ if (nblocks > 0)
+ targetBlock = nblocks - 1;
+
+ /* get the last block (if the relation is empty, just do the extension) */
+ if (targetBlock != InvalidBlockNumber)
+ {
+ buffer = ReadBuffer(colstore, targetBlock);
+
+ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
+
+ page = BufferGetColumnarPage(buffer);
+
+ /* if there's enough space for another item, we're done */
+ if (ColumnarPageGetFreeItems(page) > 0)
+ return buffer;
+
+ /* otherwise, let's allocate a new page at the end */
+ LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
+ ReleaseBuffer(buffer);
+ }
+
+ if (needLock)
+ LockRelationForExtension(colstore, ExclusiveLock);
+
+ buffer = ReadBuffer(colstore, P_NEW);
+
+ /*
+ * Now acquire lock on the new page.
+ */
+ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE);
+
+ if (needLock)
+ UnlockRelationForExtension(colstore, ExclusiveLock);
+
+ page = BufferGetColumnarPage(buffer);
+
+ ColumnarPageInit(page, BufferGetPageSize(buffer), colstore);
+
+ RelationSetTargetBlock(colstore, BufferGetBlockNumber(buffer));
+
+ MarkBufferDirty(buffer);
+
+ return buffer;
+}
+
+static int
+ColumnarPageGetFreeItems(ColumnarPage page)
+{
+ ColumnarPageHeader header = (ColumnarPageHeader) page;
+
+ return (header->pd_maxitems - header->pd_nitems);
+}
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index c5befcb..72c98ad 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -3751,6 +3751,10 @@ DESCR("BERNOULLI tablesample method handler");
DATA(insert OID = 3314 ( system PGNSP PGUID 12 1 0 0 0 f f f f t f v 1 0 3310 \
"2281" _null_ _null_ _null_ _null_ _null_ tsm_system_handler _null_ _null_ _null_ )); \
DESCR("SYSTEM tablesample method handler");
+/* column stores */
+DATA(insert OID = 3354 ( cstore_dummy_handler PGNSP PGUID 12 1 0 0 0 f f f f t f i \
0 0 3351 "" _null_ _null_ _null_ _null_ _null_ cstore_dummy_handler _null_ _null_ \
_null_ )); +DESCR("dummy column store method");
+
/* cryptographic */
DATA(insert OID = 2311 ( md5 PGNSP PGUID 12 1 0 0 0 f f f f t f i 1 0 25 "25" \
_null_ _null_ _null_ _null_ _null_ md5_text _null_ _null_ _null_ )); DESCR("MD5 \
hash");
diff --git a/src/include/colstore/colstore_dummy.h \
b/src/include/colstore/colstore_dummy.h new file mode 100644
index 0000000..372eeff
--- /dev/null
+++ b/src/include/colstore/colstore_dummy.h
@@ -0,0 +1,161 @@
+/*-------------------------------------------------------------------------
+ *
+ * colstore_dummy.h
+ * API for column store implementations
+ *
+ * Copyright (c) 2010-2015, PostgreSQL Global Development Group
+ *
+ * src/include/colstore/colstore_dummy.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef COLSTOREDUMMY_H
+#define COLSTOREDUMMY_H
+
+#include "access/attnum.h"
+#include "access/xlogdefs.h"
+#include "storage/block.h"
+#include "storage/bufpage.h"
+#include "storage/item.h"
+#include "storage/itemptr.h"
+#include "storage/off.h"
+#include "utils/rel.h"
+
+/*
+ * A columnar disk page is an abstraction layered on top of a postgres
+ * disk block (which is simply a unit of i/o, see block.h).
+ *
+ * specifically, while a disk block can be unformatted, a columnar disk
+ * page format depends on the particular column store implementation.
+ * For the 'dummy' implementation, it is a slotted page of the form:
+ *
+ * +----------------+-------+-----------------+-----------------+
+ * | ColumnarPageHeaderData | ColumnInfoData1 | ColumnInfoData2 |
+ * +-------+----------------+-+---------------+-----------------+
+ * | ... ColumnInfoDataN | tuple IDs |
+ * +---------------------+----+----------------+----------------+
+ * | column 1 data | column 2 data | .... |
+ * +---------------------+---------------------+----------------+
+ * | |
+ * | |
+ * +---------------------------------+--------------------------+
+ * | ... | column N data |
+ * +---------------------------------+--------------------------+
+ *
+ * a page is full when a new tuple can't be added (even after moving
+ * the data around, compressing etc.)
+ *
+ * all blocks written out by an access method must be disk pages.
+ *
+ * EXCEPTIONS:
+ *
+ * obviously, a page is not formatted before it is initialized by
+ * a call to ColumnarPageInit.
+ *
+ * NOTES:
+ *
+ * The tuple IDs contain tuple IDs for all tuples stored on this page,
+ * providing a mapping to the heap part. It's good to keep this array
+ * sorted, as that makes lookup faster. It's also possible to encode
+ * this array using RLE, for example (again, that works better for
+ * sorted data). There's also a min/max TID in the page header.
+ *
+ * The 'column data' combine all the data for a column, i.e. the actual
+ * values and NULL bitmap. The data may be partially compressed, etc.
+ *
+ * Some of the page fields may seem too big (e.g. 32 bits for nitems seems
+ * a bit over the top, but (a) 16 bits is just on the border for 64kB pages
+ * (and larger pages may get supported in the future), (b) we do expect
+ * efficient storage of some data types (e.g. bool type in 1 bit). That makes
+ * the 16bit data type inadequate.
+ *
+ * We must however keep the beginning of the header exactly the same as for
+ * regular pages, so that the checksum / validation stuff works.
+ */
+
+typedef Pointer ColumnarPage;
+
+typedef struct ColumnInfoData
+{
+ AttrNumber attnum;
+ int attlen;
+ Oid atttypid;
+ bool attnotnull;
+ LocationIndex data_start;
+ LocationIndex data_bytes;
+ LocationIndex null_start;
+ LocationIndex null_bytes;
+} ColumnInfoData;
+
+typedef struct ColumnarPageHeaderData
+{
+ /* XXX LSN is member of *any* block, not only page-organized ones */
+ PageXLogRecPtr pd_lsn; /* LSN: next byte after last byte of xlog
+ * record for last change to this page */
+ uint16 pd_checksum; /* checksum */
+ uint16 pd_flags; /* flag bits, see below */
+ LocationIndex pd_lower; /* offset to start of free space */
+ LocationIndex pd_upper; /* offset to end of free space */
+ LocationIndex pd_special; /* offset to start of special space */
+ uint16 pd_pagesize_version;
+
+ /* our fields start here */
+ LocationIndex pd_tupleids; /* offset of tuple IDs */
+ uint16 pd_ncolumns; /* number of columns on the page */
+ uint32 pd_nitems; /* number of items on the page */
+ uint32 pd_maxitems; /* max number of items on the page */
+ ItemPointerData pd_min_tid; /* mininum TID placed on page */
+ ItemPointerData pd_max_tid; /* maximum TID placed on page */
+ ColumnInfoData pd_columns[FLEXIBLE_ARRAY_MEMBER]; /* column info array */
+} ColumnarPageHeaderData;
+
+typedef ColumnarPageHeaderData *ColumnarPageHeader;
+
+#define BufferGetColumnarPage(buffer) ((ColumnarPage)BufferGetBlock(buffer))
+
+#define PageGetNumOfItems(page) \
+ (((ColumnarPageHeader)page)->pd_nitems)
+
+#define PageGetColumnInfo(page, column) \
+ (((ColumnarPageHeader)page)->pd_columns[column])
+
+#define PageGetColumnAttlen(page, column) \
+ (PageGetColumnInfo(page,column).attlen)
+
+#define PageGetColumnDataOffset(page, column) \
+ (PageGetColumnInfo(page,column).data_start)
+
+#define PageGetColumnDataBytes(page, column) \
+ (PageGetColumnInfo(page,column).data_bytes)
+
+#define PageGetColumnDataAddBytes(page, column, len) \
+ (PageGetColumnInfo(page,column).data_bytes += len)
+
+#define PageGetColumnDataOffsetNext(page, column) \
+ (PageGetColumnDataOffset(page, column) + \
+ PageGetColumnDataBytes(page, column))
+
+#define PageGetColumnNullsOffset(page, column) \
+ (((ColumnarPageHeader)page)->pd_columns[column].null_start)
+
+#define PageGetColumnData(page,column) \
+ ((char*)page + PageGetColumnDataOffset(page, column))
+
+#define PageGetColumnDataNext(page,column) \
+ ((char*)page + PageGetColumnDataOffsetNext(page, column))
+
+#define PageGetColumnNulls(page,column) \
+ ((char*)page + PageGetColumnNullsOffset(page, column))
+
+#define PageGetColumnNullsSetBytes(page,column, len) \
+ (PageGetColumnInfo(page,column).null_bytes = len)
+
+#define PageGetTupleIds(page) \
+ ((char*)page + (((ColumnarPageHeader)page)->pd_tupleids))
+
+#define PageGetNextTupleId(page) \
+ ((ItemPointer)PageGetTupleIds(page) + PageGetNumOfItems(page))
+
+extern void ColumnarPageInit(ColumnarPage page, Size pageSize, Relation rel);
+
+#endif /* COLSTOREDUMMY_H */
--
2.1.4
["0017-Add-ColumnStoreMaterial-node.patch" (text/x-diff)]
From ff6bc004dc3fa48cb9c8185944b524bbe4ab6e3f Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 23 Jun 2015 19:32:02 +0200
Subject: [PATCH 17/24] Add ColumnStoreMaterial node
Defines the plan and executor nodes.
---
src/backend/nodes/copyfuncs.c | 19 +++++++++++++++++++
src/backend/nodes/outfuncs.c | 11 +++++++++++
src/include/nodes/execnodes.h | 25 +++++++++++++++++++++++++
src/include/nodes/nodes.h | 2 ++
src/include/nodes/plannodes.h | 31 +++++++++++++++++++++++++++++++
5 files changed, 88 insertions(+)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 3aaf221..ae5b0d3 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -786,6 +786,22 @@ _copyMaterial(const Material *from)
return newnode;
}
+/*
+ * _copyColumnStoreMaterial
+ */
+static ColumnStoreMaterial *
+_copyColumnStoreMaterial(const ColumnStoreMaterial *from)
+{
+ ColumnStoreMaterial *newnode = makeNode(ColumnStoreMaterial);
+
+ /*
+ * copy node superclass fields
+ */
+ CopyPlanFields((const Plan *) from, (Plan *) newnode);
+
+ return newnode;
+}
+
/*
* _copySort
@@ -4316,6 +4332,9 @@ copyObject(const void *from)
case T_Material:
retval = _copyMaterial(from);
break;
+ case T_ColumnStoreMaterial:
+ retval = _copyColumnStoreMaterial(from);
+ break;
case T_Sort:
retval = _copySort(from);
break;
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 235b4e3..92a14fe 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -751,6 +751,14 @@ _outMaterial(StringInfo str, const Material *node)
}
static void
+_outColumnStoreMaterial(StringInfo str, const ColumnStoreMaterial *node)
+{
+ WRITE_NODE_TYPE("COLUMNSTOREMATERIAL");
+
+ _outPlanInfo(str, (const Plan *) node);
+}
+
+static void
_outSort(StringInfo str, const Sort *node)
{
int i;
@@ -3093,6 +3101,9 @@ _outNode(StringInfo str, const void *obj)
case T_Material:
_outMaterial(str, obj);
break;
+ case T_ColumnStoreMaterial:
+ _outColumnStoreMaterial(str, obj);
+ break;
case T_Sort:
_outSort(str, obj);
break;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 5816a37..c0a9862 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1801,6 +1801,31 @@ typedef struct MaterialState
} MaterialState;
/* ----------------
+ * ColumnStoreMaterialState information
+ *
+ * materialize nodes are used to fill-in values from a column store
+ * into an ordinary tuple
+ *
+ * unlike the regular Materialize node it does not store the data
+ * into a tuple store or so, it just fills-in the values and
+ * streams the tuple out (it may perform some internal batching
+ * to benefit from the columnar storage, but it's not required to
+ * materialize the whole result set)
+ *
+ * ss.ss_ScanTupleSlot refers to output of underlying plan.
+ * ----------------
+ */
+typedef struct ColumnStoreMaterialState
+{
+ ScanState ss; /* its first field is NodeTag */
+ int eflags; /* capability flags */
+
+ /* TODO This likely requires additional fields (info about the
+ * colstore (ColumnScanDesc?) etc. */
+
+} ColumnStoreMaterialState;
+
+/* ----------------
* SortState information
* ----------------
*/
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 97efd24..f04dbcf 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -83,6 +83,7 @@ typedef enum NodeTag
T_NestLoopParam,
T_PlanRowMark,
T_PlanInvalItem,
+ T_ColumnStoreMaterial,
/*
* TAGS FOR PLAN STATE NODES (execnodes.h)
@@ -126,6 +127,7 @@ typedef enum NodeTag
T_SetOpState,
T_LockRowsState,
T_LimitState,
+ T_ColumnStoreMaterialState,
/*
* TAGS FOR PRIMITIVE NODES (primnodes.h)
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 0654d02..5167492 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -659,6 +659,37 @@ typedef struct Material
} Material;
/* ----------------
+ * column store materialization node
+ * ----------------
+ *
+ * Reads tuples from the plan, and materializes columns from the column store,
+ * i.e. fetches values from the column store (identified by cststoreid) and
+ * sets them in the tuples. This is a bit like a special kind of join, except
+ * that we're not joining the relations using regular columns but using logical
+ * or physical ROWIDs.
+ *
+ * We'll want some join planning features (e.g. if there are multiple
+ * colstores, then decide in what order they should be materialized),
+ * but a regular join would require heap-ification of the column store,
+ * and that's not something we want to happen.
+ *
+ * As we foresee the option to place a column into multiple colstores,
+ * we also need to specify what attributes to materialize at this point
+ * (and what to leave for the next materializations). That's what
+ * matColIdx is for.
+ *
+ * XXX We're using TID as a ROWID for now, but in the future we expect other
+ * kinds of IDs.
+ */
+typedef struct ColumnStoreMaterial
+{
+ Plan plan;
+ Index rti; /* range table index of column store */
+ Oid colstoreid; /* OID of column store to materialize */
+ AttrNumber *matColIdx; /* which attributes to materialize */
+} ColumnStoreMaterial;
+
+/* ----------------
* sort node
* ----------------
*/
--
2.1.4
["0018-initial-planning-of-ColumnStoreMaterialize-nodes.patch" (text/x-diff)]
From 35a1f3d8f5cc214b9b3a8de66ed0ef390b30efc9 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 14 Jul 2015 21:35:56 +0200
Subject: [PATCH 18/24] initial planning of ColumnStoreMaterialize nodes
Note: This isn't really intended to be the definitive form of this code.
We envision modifying earlier stages of planning so that column stores
are added as range table entries, and have Path structures created for
them. Therefore, don't look at this code too much.
- modifies create_seqscan to create seqscan + materialize node for
each column store (trivial and naive, needs a number of fixes)
- defines trivial costing (reuse costing from subpath)
- creates plan nodes from the path
- allows EXPLAIN on the plan
- does not currently implement executor node (points to Materialize)
For a table with 3 column stores, the EXPLAIN looks like this:
test=# explain select * from parent_table ;
QUERY PLAN
-----------------------------------------------------------------------------------
Column Store Materialize (cost=0.00..28.50 rows=1850 width=16)
-> Column Store Materialize (cost=0.00..28.50 rows=1850 width=16)
-> Column Store Materialize (cost=0.00..28.50 rows=1850 width=16)
-> Seq Scan on parent_table (cost=0.00..28.50 rows=1850 width=16)
(4 rows)
Implementation notes:
- This doesn't handle anything other than SeqScan (trying to access a
table via indexscans or other methods is going to fail).
---
src/backend/commands/explain.c | 3 +++
src/backend/executor/execProcnode.c | 5 ++++
src/backend/optimizer/path/costsize.c | 27 +++++++++++++++++++
src/backend/optimizer/plan/createplan.c | 48 +++++++++++++++++++++++++++++++++
src/backend/optimizer/plan/setrefs.c | 1 +
src/backend/optimizer/util/pathnode.c | 30 +++++++++++++++++++++
src/include/nodes/nodes.h | 1 +
src/include/nodes/relation.h | 11 ++++++++
src/include/optimizer/cost.h | 2 ++
9 files changed, 128 insertions(+)
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 5d06fa4..6192091 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -1045,6 +1045,9 @@ ExplainNode(PlanState *planstate, List *ancestors,
case T_Hash:
pname = sname = "Hash";
break;
+ case T_ColumnStoreMaterial:
+ pname = sname = "Column Store Materialize";
+ break;
default:
pname = sname = "???";
break;
diff --git a/src/backend/executor/execProcnode.c b/src/backend/executor/execProcnode.c
index 03c2feb..5d68b3e 100644
--- a/src/backend/executor/execProcnode.c
+++ b/src/backend/executor/execProcnode.c
@@ -282,6 +282,11 @@ ExecInitNode(Plan *node, EState *estate, int eflags)
estate, eflags);
break;
+ case T_ColumnStoreMaterial: /* FIXME this is wrong (no exec implementation yet!!!) */
+ result = (PlanState *) ExecInitMaterial((Material *) node,
+ estate, eflags);
+ break;
+
case T_Sort:
result = (PlanState *) ExecInitSort((Sort *) node,
estate, eflags);
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 7069f60..6624811 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -290,6 +290,33 @@ cost_samplescan(Path *path, PlannerInfo *root,
}
/*
+ * cost_colmaterial
+ * Determines and returns the cost of materializing a column store.
+ *
+ * 'baserel' is the relation to be scanned
+ * 'param_info' is the ParamPathInfo if this is a parameterized path, else NULL
+ * 'info' is the column store being materialized
+ */
+void
+cost_colstore_material(ColumnStoreMaterialPath *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info,
+ ColumnStoreOptInfo *info)
+{
+ /* Should only be applied to base relations */
+ Assert(baserel->relid > 0);
+ Assert(baserel->rtekind == RTE_RELATION);
+
+ /*
+ * FIXME Very naive costing - just reuse cost from the subpath. Ultimately
+ * we want something that (at least) considers sizes of the column
+ * stores, and ideally some costing for the 'join'.
+ */
+ path->path.startup_cost = path->subpath->startup_cost;
+ path->path.total_cost = path->subpath->total_cost;
+ path->path.rows = path->subpath->rows;
+}
+
+/*
* cost_index
* Determines and returns the cost of scanning a relation using an index.
*
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 404c6f5..28b96ca 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -55,6 +55,8 @@ static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path);
static Plan *create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path);
static Result *create_result_plan(PlannerInfo *root, ResultPath *best_path);
static Material *create_material_plan(PlannerInfo *root, MaterialPath *best_path);
+static ColumnStoreMaterial *create_colstore_material_plan(PlannerInfo *root,
+ ColumnStoreMaterialPath *best_path);
static Plan *create_unique_plan(PlannerInfo *root, UniquePath *best_path);
static SeqScan *create_seqscan_plan(PlannerInfo *root, Path *best_path,
List *tlist, List *scan_clauses);
@@ -176,6 +178,7 @@ static EquivalenceMember *find_ec_member_for_tle(EquivalenceClass *ec,
TargetEntry *tle,
Relids relids);
static Material *make_material(Plan *lefttree);
+static ColumnStoreMaterial *make_colstore_material(Plan *lefttree);
/*
@@ -269,6 +272,10 @@ create_plan_recurse(PlannerInfo *root, Path *best_path)
plan = (Plan *) create_material_plan(root,
(MaterialPath *) best_path);
break;
+ case T_ColumnStoreMaterial:
+ plan = (Plan *) create_colstore_material_plan(root,
+ (ColumnStoreMaterialPath *) best_path);
+ break;
case T_Unique:
plan = create_unique_plan(root,
(UniquePath *) best_path);
@@ -899,6 +906,32 @@ create_material_plan(PlannerInfo *root, MaterialPath *best_path)
}
/*
+ * create_colstore_material_plan
+ * Create a ColumnStoreMaterial plan for 'best_path' and (recursively) plans
+ * for its subpaths.
+ *
+ * Returns a Plan node.
+ */
+static ColumnStoreMaterial *
+create_colstore_material_plan(PlannerInfo *root, ColumnStoreMaterialPath *best_path)
+{
+ ColumnStoreMaterial *plan;
+ Plan *subplan;
+
+ subplan = create_plan_recurse(root, best_path->subpath);
+
+ /* We don't want any excess columns in the materialized tuples */
+ disuse_physical_tlist(root, subplan, best_path->subpath);
+
+ plan = make_colstore_material(subplan);
+
+ copy_path_costsize(&plan->plan, (Path *) best_path);
+
+ return plan;
+}
+
+
+/*
* create_unique_plan
* Create a Unique plan for 'best_path' and (recursively) plans
* for its subpaths.
@@ -4469,6 +4502,21 @@ make_material(Plan *lefttree)
return node;
}
+static ColumnStoreMaterial *
+make_colstore_material(Plan *lefttree)
+{
+ ColumnStoreMaterial *node = makeNode(ColumnStoreMaterial);
+ Plan *plan = &node->plan;
+
+ /* cost should be inserted by caller */
+ plan->targetlist = lefttree->targetlist;
+ plan->qual = NIL;
+ plan->lefttree = lefttree;
+ plan->righttree = NULL;
+
+ return node;
+}
+
/*
* materialize_finished_plan: stick a Material node atop a completed plan
*
diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c
index ee8710d..e9e679e 100644
--- a/src/backend/optimizer/plan/setrefs.c
+++ b/src/backend/optimizer/plan/setrefs.c
@@ -599,6 +599,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset)
case T_Hash:
case T_Material:
+ case T_ColumnStoreMaterial:
case T_Sort:
case T_Unique:
case T_SetOp:
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index 935bc2b..14602e8 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -698,6 +698,7 @@ add_path_precheck(RelOptInfo *parent_rel,
Path *
create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer)
{
+ ListCell *cell;
Path *pathnode = makeNode(Path);
pathnode->pathtype = T_SeqScan;
@@ -708,6 +709,35 @@ create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer)
cost_seqscan(pathnode, root, rel, pathnode->param_info);
+ /*
+ * Add materialization node for each column store.
+ *
+ * FIXME At the moment we add materialization for each column store - this
+ * needs to only choose the required column stores.
+ *
+ * FIXME This probably needs to fix parametrization somehow, because the
+ * lower nodes may not contain the required columns.
+ *
+ * FIXME This also has to fix the tuple width, because right now we assume
+ * "full" width in all steps (which is nonsense). That's not a problem
+ * now, but once we start moving the materialization steps around
+ * (to get late materialization), this might be an issue.
+ */
+ foreach (cell, rel->cstlist)
+ {
+ ColumnStoreOptInfo *info = (ColumnStoreOptInfo *)lfirst(cell);
+ ColumnStoreMaterialPath *cstnode = makeNode(ColumnStoreMaterialPath);
+
+ cstnode->path.pathtype = T_ColumnStoreMaterial;
+ cstnode->path.parent = rel;
+ cstnode->colstore = info;
+ cstnode->subpath = pathnode;
+
+ cost_colstore_material(cstnode, root, rel, pathnode->param_info, info);
+
+ pathnode = (Path*)cstnode;
+ }
+
return pathnode;
}
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index f04dbcf..bb664d4 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -253,6 +253,7 @@ typedef enum NodeTag
T_MinMaxAggInfo,
T_PlannerParamItem,
T_ColumnStoreOptInfo,
+ T_ColumnStoreMaterialPath,
/*
* TAGS FOR MEMORY NODES (memnodes.h)
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 18a8891..a8b2a28 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -1042,6 +1042,17 @@ typedef struct MaterialPath
} MaterialPath;
/*
+ * ColumnStoreMaterialPath
+ * ... FIXME comments
+ */
+typedef struct ColumnStoreMaterialPath
+{
+ Path path; /* this path node */
+ Path *subpath; /* child path node (seqscan ...) */
+ ColumnStoreOptInfo *colstore; /* column store info */
+} ColumnStoreMaterialPath;
+
+/*
* UniquePath represents elimination of distinct rows from the output of
* its subpath.
*
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index dd43e45..01e3cef 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -70,6 +70,8 @@ extern void cost_seqscan(Path *path, PlannerInfo *root, RelOptInfo *baserel,
ParamPathInfo *param_info);
extern void cost_samplescan(Path *path, PlannerInfo *root, RelOptInfo *baserel,
ParamPathInfo *param_info);
+extern void cost_colstore_material(ColumnStoreMaterialPath *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info, ColumnStoreOptInfo *info);
extern void cost_index(IndexPath *path, PlannerInfo *root,
double loop_count);
extern void cost_bitmap_heap_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel,
--
2.1.4
["0019-Some-stub-executor-code.patch" (text/x-diff)]
From b32c890caf6625983ddcd17cafd8b6b445a1d6c5 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Tue, 14 Jul 2015 23:33:29 +0200
Subject: [PATCH 19/24] Some stub executor code
This provides a new file, nodeColumStoreMaterial, implementing the
ColumnStoreMaterialize node.
- dummy implementation of the executor methods (doing nothing)
- we might probably avoid using the term 'Materialize' as that means
something different in PostgreSQL (stuff the tuples into tuple store)
---
src/backend/executor/Makefile | 6 +-
src/backend/executor/execAmi.c | 15 +++
src/backend/executor/execProcnode.c | 14 +-
src/backend/executor/nodeColumnStoreMaterial.c | 179 +++++++++++++++++++++++++
src/backend/nodes/outfuncs.c | 13 ++
src/backend/optimizer/plan/subselect.c | 1 +
src/include/executor/nodeColumnStoreMaterial.h | 26 ++++
7 files changed, 249 insertions(+), 5 deletions(-)
create mode 100644 src/backend/executor/nodeColumnStoreMaterial.c
create mode 100644 src/include/executor/nodeColumnStoreMaterial.h
diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile
index 08cba6f..703a9c8 100644
--- a/src/backend/executor/Makefile
+++ b/src/backend/executor/Makefile
@@ -16,9 +16,9 @@ OBJS = execAmi.o execCurrent.o execGrouping.o execIndexing.o \
execJunk.o \ execMain.o execProcnode.o execQual.o execScan.o execTuples.o \
execUtils.o functions.o instrument.o nodeAppend.o nodeAgg.o \
nodeBitmapAnd.o nodeBitmapOr.o \
- nodeBitmapHeapscan.o nodeBitmapIndexscan.o nodeCustom.o nodeHash.o \
- nodeHashjoin.o nodeIndexscan.o nodeIndexonlyscan.o \
- nodeLimit.o nodeLockRows.o \
+ nodeBitmapHeapscan.o nodeBitmapIndexscan.o nodeColumnStoreMaterial.o \
+ nodeCustom.o nodeHash.o nodeHashjoin.o \
+ nodeIndexscan.o nodeIndexonlyscan.o nodeLimit.o nodeLockRows.o \
nodeMaterial.o nodeMergeAppend.o nodeMergejoin.o nodeModifyTable.o \
nodeNestloop.o nodeFunctionscan.o nodeRecursiveunion.o nodeResult.o \
nodeSamplescan.o nodeSeqscan.o nodeSetOp.o nodeSort.o nodeUnique.o \
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index 93e1e9a..1ca905b 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -20,6 +20,7 @@
#include "executor/nodeBitmapHeapscan.h"
#include "executor/nodeBitmapIndexscan.h"
#include "executor/nodeBitmapOr.h"
+#include "executor/nodeColumnStoreMaterial.h"
#include "executor/nodeCtescan.h"
#include "executor/nodeCustom.h"
#include "executor/nodeForeignscan.h"
@@ -224,6 +225,10 @@ ExecReScan(PlanState *node)
ExecReScanMaterial((MaterialState *) node);
break;
+ case T_ColumnStoreMaterialState:
+ ExecReScanColumnStoreMaterial((ColumnStoreMaterialState *) node);
+ break;
+
case T_SortState:
ExecReScanSort((SortState *) node);
break;
@@ -306,6 +311,10 @@ ExecMarkPos(PlanState *node)
ExecMaterialMarkPos((MaterialState *) node);
break;
+ case T_ColumnStoreMaterialState:
+ ExecColumnStoreMaterialMarkPos((ColumnStoreMaterialState *) node);
+ break;
+
case T_SortState:
ExecSortMarkPos((SortState *) node);
break;
@@ -355,6 +364,10 @@ ExecRestrPos(PlanState *node)
ExecMaterialRestrPos((MaterialState *) node);
break;
+ case T_ColumnStoreMaterialState:
+ ExecColumnStoreMaterialRestrPos((ColumnStoreMaterialState *) node);
+ break;
+
case T_SortState:
ExecSortRestrPos((SortState *) node);
break;
@@ -389,6 +402,7 @@ ExecSupportsMarkRestore(Path *pathnode)
case T_IndexScan:
case T_IndexOnlyScan:
case T_Material:
+ case T_ColumnStoreMaterial:
case T_Sort:
return true;
@@ -490,6 +504,7 @@ ExecSupportsBackwardScan(Plan *node)
return false;
case T_Material:
+ case T_ColumnStoreMaterial:
case T_Sort:
/* these don't evaluate tlist */
return true;
diff --git a/src/backend/executor/execProcnode.c \
b/src/backend/executor/execProcnode.c index 5d68b3e..5931049 100644
--- a/src/backend/executor/execProcnode.c
+++ b/src/backend/executor/execProcnode.c
@@ -84,6 +84,7 @@
#include "executor/nodeBitmapHeapscan.h"
#include "executor/nodeBitmapIndexscan.h"
#include "executor/nodeBitmapOr.h"
+#include "executor/nodeColumnStoreMaterial.h"
#include "executor/nodeCtescan.h"
#include "executor/nodeCustom.h"
#include "executor/nodeForeignscan.h"
@@ -282,8 +283,9 @@ ExecInitNode(Plan *node, EState *estate, int eflags)
estate, eflags);
break;
- case T_ColumnStoreMaterial: /* FIXME this is wrong (no exec implementation yet!!!) \
*/
- result = (PlanState *) ExecInitMaterial((Material *) node,
+ case T_ColumnStoreMaterial:
+ result = (PlanState *) ExecInitColumnStoreMaterial(
+ (ColumnStoreMaterial *) node,
estate, eflags);
break;
@@ -489,6 +491,10 @@ ExecProcNode(PlanState *node)
result = ExecMaterial((MaterialState *) node);
break;
+ case T_ColumnStoreMaterialState:
+ result = ExecColumnStoreMaterial((ColumnStoreMaterialState *) node);
+ break;
+
case T_SortState:
result = ExecSort((SortState *) node);
break;
@@ -733,6 +739,10 @@ ExecEndNode(PlanState *node)
ExecEndMaterial((MaterialState *) node);
break;
+ case T_ColumnStoreMaterialState:
+ ExecEndColumnStoreMaterial((ColumnStoreMaterialState *) node);
+ break;
+
case T_SortState:
ExecEndSort((SortState *) node);
break;
diff --git a/src/backend/executor/nodeColumnStoreMaterial.c \
b/src/backend/executor/nodeColumnStoreMaterial.c new file mode 100644
index 0000000..0ee1444
--- /dev/null
+++ b/src/backend/executor/nodeColumnStoreMaterial.c
@@ -0,0 +1,179 @@
+/*-------------------------------------------------------------------------
+ *
+ * nodeColumnStoreMaterial.c
+ * Routines to handle column store materialization nodes.
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/executor/nodeColumnStoreMaterial.c
+ *
+ *-------------------------------------------------------------------------
+ */
+/*
+ * INTERFACE ROUTINES
+ * ExecColumnStoreMaterial - materialize the result of a subplan
+ * ExecInitColumnStoreMaterial - initialize node and subnodes
+ * ExecEndColumnStoreMaterial - shutdown node and subnodes
+ *
+ */
+#include "postgres.h"
+
+#include "executor/executor.h"
+#include "executor/nodeColumnStoreMaterial.h"
+#include "miscadmin.h"
+
+/* ----------------------------------------------------------------
+ * ExecColumnStoreMaterial
+ *
+ * As long as we are at the end of the data collected in the tuplestore,
+ * we collect one new row from the subplan on each call, and stash it
+ * aside in the tuplestore before returning it. The tuplestore is
+ * only read if we are asked to scan backwards, rescan, or mark/restore.
+ *
+ * ----------------------------------------------------------------
+ */
+TupleTableSlot * /* result tuple from subplan */
+ExecColumnStoreMaterial(ColumnStoreMaterialState *node)
+{
+// EState *estate;
+// ScanDirection dir;
+// bool forward;
+
+ PlanState *outerNode;
+ TupleTableSlot *outerslot;
+
+ /*
+ * get state info from node
+ */
+// estate = node->ss.ps.state;
+// dir = estate->es_direction;
+// forward = ScanDirectionIsForward(dir);
+
+ /* simply read tuple from the outer node (left subtree) */
+ outerNode = outerPlanState(node);
+ outerslot = ExecProcNode(outerNode);
+
+ if (TupIsNull(outerslot))
+ return NULL;
+
+ return outerslot;
+}
+
+/* ----------------------------------------------------------------
+ * ExecInitMaterial
+ * ----------------------------------------------------------------
+ */
+ColumnStoreMaterialState *
+ExecInitColumnStoreMaterial(ColumnStoreMaterial *node, EState *estate, int eflags)
+{
+ ColumnStoreMaterialState *colmatstate;
+ Plan *outerPlan;
+
+ /*
+ * create state structure
+ */
+ colmatstate = makeNode(ColumnStoreMaterialState);
+ colmatstate->ss.ps.plan = (Plan *) node;
+ colmatstate->ss.ps.state = estate;
+
+ colmatstate->eflags = (eflags & (EXEC_FLAG_REWIND |
+ EXEC_FLAG_BACKWARD |
+ EXEC_FLAG_MARK));
+
+ /*
+ * Miscellaneous initialization
+ *
+ * Materialization nodes don't need ExprContexts because they never call
+ * ExecQual or ExecProject.
+ */
+
+ /*
+ * tuple table initialization
+ *
+ * material nodes only return tuples from their materialized relation.
+ */
+ ExecInitResultTupleSlot(estate, &colmatstate->ss.ps);
+ ExecInitScanTupleSlot(estate, &colmatstate->ss);
+
+ /*
+ * initialize child nodes
+ *
+ * We shield the child node from the need to support REWIND, BACKWARD, or
+ * MARK/RESTORE.
+ */
+ eflags &= ~(EXEC_FLAG_REWIND | EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK);
+
+ outerPlan = outerPlan(node);
+ outerPlanState(colmatstate) = ExecInitNode(outerPlan, estate, eflags);
+
+ /*
+ * initialize tuple type. no need to initialize projection info because
+ * this node doesn't do projections.
+ */
+ ExecAssignResultTypeFromTL(&colmatstate->ss.ps);
+ ExecAssignScanTypeFromOuterPlan(&colmatstate->ss);
+ colmatstate->ss.ps.ps_ProjInfo = NULL;
+
+ return colmatstate;
+}
+
+/* ----------------------------------------------------------------
+ * ExecColumnStoreEndMaterial
+ * ----------------------------------------------------------------
+ */
+void
+ExecEndColumnStoreMaterial(ColumnStoreMaterialState *node)
+{
+ /*
+ * clean out the tuple table
+ */
+ ExecClearTuple(node->ss.ss_ScanTupleSlot);
+
+ /*
+ * shut down the subplan
+ */
+ ExecEndNode(outerPlanState(node));
+}
+
+/* ----------------------------------------------------------------
+ * ExecColumnStoreMaterialMarkPos
+ * ----------------------------------------------------------------
+ */
+void
+ExecColumnStoreMaterialMarkPos(ColumnStoreMaterialState *node)
+{
+ Assert(node->eflags & EXEC_FLAG_MARK);
+
+ // FIXME
+}
+
+/* ----------------------------------------------------------------
+ * ExecColumnStoreMaterialRestrPos
+ * ----------------------------------------------------------------
+ */
+void
+ExecColumnStoreMaterialRestrPos(ColumnStoreMaterialState *node)
+{
+ Assert(node->eflags & EXEC_FLAG_MARK);
+
+ // FIXME
+}
+
+/* ----------------------------------------------------------------
+ * ExecColumnStoreReScanMaterial
+ *
+ * Rescans the materialized relation.
+ * ----------------------------------------------------------------
+ */
+void
+ExecReScanColumnStoreMaterial(ColumnStoreMaterialState *node)
+{
+ ExecClearTuple(node->ss.ps.ps_ResultTupleSlot);
+
+ if (node->ss.ps.lefttree->chgParam == NULL)
+ ExecReScan(node->ss.ps.lefttree);
+
+}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 92a14fe..2e550d2 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1733,6 +1733,16 @@ _outMaterialPath(StringInfo str, const MaterialPath *node)
}
static void
+_outColumnStoreMaterialPath(StringInfo str, const ColumnStoreMaterialPath *node)
+{
+ WRITE_NODE_TYPE("COLUMNSTOREMATERIALPATH");
+
+ _outPathInfo(str, (const Path *) node);
+
+ WRITE_NODE_FIELD(subpath);
+}
+
+static void
_outUniquePath(StringInfo str, const UniquePath *node)
{
WRITE_NODE_TYPE("UNIQUEPATH");
@@ -3311,6 +3321,9 @@ _outNode(StringInfo str, const void *obj)
case T_MaterialPath:
_outMaterialPath(str, obj);
break;
+ case T_ColumnStoreMaterialPath:
+ _outColumnStoreMaterialPath(str, obj);
+ break;
case T_UniquePath:
_outUniquePath(str, obj);
break;
diff --git a/src/backend/optimizer/plan/subselect.c \
b/src/backend/optimizer/plan/subselect.c index d0bc412..3105df6 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -2582,6 +2582,7 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset \
*valid_params, case T_Hash:
case T_Agg:
case T_Material:
+ case T_ColumnStoreMaterial:
case T_Sort:
case T_Unique:
case T_SetOp:
diff --git a/src/include/executor/nodeColumnStoreMaterial.h \
b/src/include/executor/nodeColumnStoreMaterial.h new file mode 100644
index 0000000..cd588d2
--- /dev/null
+++ b/src/include/executor/nodeColumnStoreMaterial.h
@@ -0,0 +1,26 @@
+/*-------------------------------------------------------------------------
+ *
+ * nodeColumnStoreMaterial.h
+ *
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/executor/nodeColumnStoreMaterial.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef NODECOLSTOREMATERIAL_H
+#define NODECOLSTOREMATERIAL_H
+
+#include "nodes/execnodes.h"
+
+extern ColumnStoreMaterialState *ExecInitColumnStoreMaterial(ColumnStoreMaterial \
*node, EState *estate, int eflags); +extern TupleTableSlot \
*ExecColumnStoreMaterial(ColumnStoreMaterialState *node); +extern void \
ExecEndColumnStoreMaterial(ColumnStoreMaterialState *node); +extern void \
ExecColumnStoreMaterialMarkPos(ColumnStoreMaterialState *node); +extern void \
ExecColumnStoreMaterialRestrPos(ColumnStoreMaterialState *node); +extern void \
ExecReScanColumnStoreMaterial(ColumnStoreMaterialState *node); +
+#endif /* NODECOLSTOREMATERIAL_H */
--
2.1.4
["0020-Add-FormColumnStoreDatum-and-FilterHeapTuple.patch" (text/x-diff)]
From 14bb38e179ffd497a3303e730bbe2674bd5d4d45 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Sun, 30 Aug 2015 01:28:47 -0300
Subject: [PATCH 20/24] Add FormColumnStoreDatum and FilterHeapTuple
These are temporary hacks only -- don't look at them too much.
FilterHeapTuple: this code is intended to take a full HeapTuple, and
create a new version of it stripped of columns that are not stored in
the heap. The resulting tuple can be inserted into the heap. This also
returns a TupleDescriptor useful for the raw heap.
FormColumnStoreDatum: this creates values/isnulls arrays that can be
passed to a column store for insert.
The definitive version of these routines probably ought to use
TupleTableSlots.
Implementation notes:
This commit includes necessary fields in struct ResultRelInfo, just so
that the functions compile. These fields are not filled yet with this
commit.
---
src/backend/catalog/colstore.c | 92 ++++++++++++++++++++++++++++++++++++++++++
src/include/catalog/colstore.h | 8 ++++
src/include/nodes/execnodes.h | 6 +++
3 files changed, 106 insertions(+)
diff --git a/src/backend/catalog/colstore.c b/src/backend/catalog/colstore.c
index 3954e1d..15cb8d4 100644
--- a/src/backend/catalog/colstore.c
+++ b/src/backend/catalog/colstore.c
@@ -579,6 +579,98 @@ BuildColumnStoreInfo(Relation cstore)
return csi;
}
+/* ----------------
+ * FormColumnStoreDatum
+ * Construct values[] and isnull[] arrays for a new column store tuple.
+ *
+ * columnStoreInfo Info about the column store
+ * slot Heap tuple for which we must prepare a column store entry
+ * values Array of column store Datums (output area)
+ * isnull Array of is-null indicators (output area)
+ * ----------------
+ */
+void
+FormColumnStoreDatum(ColumnStoreInfo *columnStoreInfo,
+ HeapTuple tuple,
+ TupleDesc tupdesc,
+ Datum *values,
+ bool *isnull)
+{
+ int i;
+
+ for (i = 0; i < columnStoreInfo->csi_NumColumnStoreAttrs; i++)
+ values[i] = heap_getattr(tuple,
+ columnStoreInfo->csi_KeyAttrNumbers[i],
+ tupdesc,
+ &isnull[i]);
+}
+
+/*
+ * Builds a descriptor for the heap part of the relation, and a tuple with only
+ * the relevant attributes.
+ */
+HeapTuple
+FilterHeapTuple(ResultRelInfo *resultRelInfo, HeapTuple tuple, TupleDesc *heapdesc)
+{
+ int i, j, attnum;
+ Bitmapset *cstoreatts = NULL; /* attributes mentioned in colstore */
+
+ TupleDesc origdesc = resultRelInfo->ri_RelationDesc->rd_att;
+ HeapTuple newtup;
+
+ /* used to build the new descriptor / tuple */
+ int natts;
+ Datum *values;
+ bool *nulls;
+
+ /* should not be called with no column stores */
+ Assert(resultRelInfo->ri_NumColumnStores > 0);
+
+ for (i = 0; i < resultRelInfo->ri_NumColumnStores; i++)
+ {
+ ColumnStoreInfo *cstinfo = resultRelInfo->ri_ColumnStoreRelationInfo[i];
+
+ for (j = 0; j < cstinfo->csi_NumColumnStoreAttrs; j++)
+ cstoreatts = bms_add_member(cstoreatts,
+ cstinfo->csi_KeyAttrNumbers[j]);
+ }
+
+ /* we should get some columns from column stores */
+ Assert(bms_num_members(cstoreatts) > 0);
+
+ /* the new descriptor contains only the remaining attributes */
+ natts = origdesc->natts - bms_num_members(cstoreatts);
+
+ *heapdesc = CreateTemplateTupleDesc(natts, false);
+
+ values = (Datum *) palloc0(sizeof(Datum) * natts);
+ nulls = (bool *) palloc0(sizeof(bool) * natts);
+
+ attnum = 1;
+ for (i = 0; i < origdesc->natts; i++)
+ {
+ /* if part of a column store, skip the attribute */
+ if (bms_is_member(origdesc->attrs[i]->attnum, cstoreatts))
+ continue;
+
+ values[attnum - 1] = heap_getattr(tuple, i + 1, origdesc,
+ &nulls[attnum - 1]);
+
+ TupleDescCopyEntry(*heapdesc, attnum++, origdesc, i + 1);
+ }
+
+ newtup = heap_form_tuple(*heapdesc, values, nulls);
+
+ /* copy important header fields */
+ newtup->t_self = tuple->t_self;
+ newtup->t_data->t_ctid = tuple->t_data->t_ctid;
+
+ pfree(values);
+ pfree(nulls);
+
+ return newtup;
+}
+
/*
* GetColumnStoreRoutine - call the specified column store handler routine
* to get its ColumnStoreRoutine struct.
diff --git a/src/include/catalog/colstore.h b/src/include/catalog/colstore.h
index 638c0e5..20be191 100644
--- a/src/include/catalog/colstore.h
+++ b/src/include/catalog/colstore.h
@@ -41,4 +41,12 @@ extern Oid GetColumnStoreAMByName(char *cstamname, bool missing_ok);
extern ColumnStoreInfo *BuildColumnStoreInfo(Relation cstore);
+/* XXX these are temporary hacks */
+extern void FormColumnStoreDatum(ColumnStoreInfo *columnStoreInfo,
+ HeapTuple tuple, TupleDesc tupdesc,
+ Datum *values, bool *isnull);
+extern HeapTuple FilterHeapTuple(ResultRelInfo *resultRelInfo, HeapTuple tuple,
+ TupleDesc *heapdesc);
+
+
#endif /* COLSTORE_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index c0a9862..574b904 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -324,6 +324,9 @@ typedef struct JunkFilter
* NumIndices # of indices existing on result relation
* IndexRelationDescs array of relation descriptors for indices
* IndexRelationInfo array of key/attr info for indices
+ * NumColumnStores # of column stores existing on result relation
+ * ColumnStoreRelationDescs array of rel. descriptors for column stores
+ * ColumnStoreRelationInfo array of key/attr info for column stores
* TrigDesc triggers to be fired, if any
* TrigFunctions cached lookup info for trigger functions
* TrigWhenExprs array of trigger WHEN expr states
@@ -347,6 +350,9 @@ typedef struct ResultRelInfo
int ri_NumIndices;
RelationPtr ri_IndexRelationDescs;
IndexInfo **ri_IndexRelationInfo;
+ int ri_NumColumnStores;
+ RelationPtr ri_ColumnStoreRelationDescs;
+ ColumnStoreInfo **ri_ColumnStoreRelationInfo;
TriggerDesc *ri_TrigDesc;
FmgrInfo *ri_TrigFunctions;
List **ri_TrigWhenExprs;
--
2.1.4
["0021-initial-implementation-of-nodeModifyTable.patch" (text/x-diff)]
From 0304a86ce5a7e4113cb2203b4b19f755c2107628 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Wed, 15 Jul 2015 21:43:54 +0200
Subject: [PATCH 21/24] initial implementation of nodeModifyTable
- defines execColumnStore mimicking execIndexing, with several methods
a) ExecOpenColumnStores (~ ExecOpenIndices)
b) ExecCloseColumnStores (~ ExecCloseIndices)
c) ExecInsertColStoreTuples (~ ExecInsertIndexTuples)
Notes:
1) This disables HOT when there are column stores on the relation.
We may eventually fix this and relax the check so that we only disable
HOT when we update columns referenced in column stores (so we can allow
HOT on the heap part).
2) The inserts seem to be working just fine, but for updates this
complaints about missing ctid junk attribute.
3) Some of the signatures were simplified to use HeapTuple instead of
TupleTableSlot, but this seems a bad idea because everything else works
with TupleTableSlot (which also combine TupleDesc + HeapTuple).
---
src/backend/access/heap/heapam.c | 9 +
src/backend/catalog/colstore.c | 1 +
src/backend/commands/copy.c | 4 +
src/backend/commands/trigger.c | 1 +
src/backend/executor/Makefile | 4 +-
src/backend/executor/execColumnStore.c | 323 +++++++++++++++++++++++++++++++++
src/backend/executor/execMain.c | 3 +
src/backend/executor/nodeModifyTable.c | 137 ++++++++++++--
src/include/catalog/colstore.h | 1 +
src/include/executor/executor.h | 8 +
10 files changed, 474 insertions(+), 17 deletions(-)
create mode 100644 src/backend/executor/execColumnStore.c
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 3701d8e..fdf3ffd 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -3328,6 +3328,15 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup,
}
/*
+ * FIXME If there are any column stores on the table, we'll just disable
+ * HOT for now. This is effectively wrong and rather awful - we can
+ * do the HOT if no columns from column stores are updated, so we
+ * should fix this eventually.
+ */
+ if (relation->rd_rel->relhascstore > 0)
+ satisfies_hot = false;
+
+ /*
* Note: beyond this point, use oldtup not otid to refer to old tuple.
* otid may very well point at newtup->t_self, which we will overwrite
* with the new tuple's location, so there's great risk of confusion if we
diff --git a/src/backend/catalog/colstore.c b/src/backend/catalog/colstore.c
index 15cb8d4..25d490d 100644
--- a/src/backend/catalog/colstore.c
+++ b/src/backend/catalog/colstore.c
@@ -29,6 +29,7 @@
#include "miscadmin.h"
#include "nodes/bitmapset.h"
#include "nodes/parsenodes.h"
+#include "nodes/execnodes.h"
#include "utils/builtins.h"
#include "utils/fmgroids.h"
#include "utils/lsyscache.h"
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 8db1b35..2d397d1 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -2302,6 +2302,8 @@ CopyFrom(CopyState cstate)
ExecOpenIndices(resultRelInfo, false);
+ ExecOpenColumnStores(resultRelInfo);
+
estate->es_result_relations = resultRelInfo;
estate->es_num_result_relations = 1;
estate->es_result_relation_info = resultRelInfo;
@@ -2508,6 +2510,8 @@ CopyFrom(CopyState cstate)
ExecCloseIndices(resultRelInfo);
+ ExecCloseColumnStores(resultRelInfo);
+
FreeExecutorState(estate);
/*
diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c
index 43421d6..d9cb227 100644
--- a/src/backend/commands/trigger.c
+++ b/src/backend/commands/trigger.c
@@ -3884,6 +3884,7 @@ afterTriggerInvokeEvents(AfterTriggerEventList *events,
/* Close indices and then the relation itself */
ExecCloseIndices(resultRelInfo);
+ ExecCloseColumnStores(resultRelInfo);
heap_close(resultRelInfo->ri_RelationDesc, NoLock);
}
FreeExecutorState(estate);
diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefile
index 703a9c8..348d21b 100644
--- a/src/backend/executor/Makefile
+++ b/src/backend/executor/Makefile
@@ -14,8 +14,8 @@ include $(top_builddir)/src/Makefile.global
OBJS = execAmi.o execCurrent.o execGrouping.o execIndexing.o execJunk.o \
execMain.o execProcnode.o execQual.o execScan.o execTuples.o \
- execUtils.o functions.o instrument.o nodeAppend.o nodeAgg.o \
- nodeBitmapAnd.o nodeBitmapOr.o \
+ execUtils.o execColumnStore.o functions.o instrument.o \
+ nodeAppend.o nodeAgg.o nodeBitmapAnd.o nodeBitmapOr.o \
nodeBitmapHeapscan.o nodeBitmapIndexscan.o nodeColumnStoreMaterial.o \
nodeCustom.o nodeHash.o nodeHashjoin.o \
nodeIndexscan.o nodeIndexonlyscan.o nodeLimit.o nodeLockRows.o \
diff --git a/src/backend/executor/execColumnStore.c b/src/backend/executor/execColumnStore.c
new file mode 100644
index 0000000..49c123c
--- /dev/null
+++ b/src/backend/executor/execColumnStore.c
@@ -0,0 +1,323 @@
+/*-------------------------------------------------------------------------
+ *
+ * execColumnStore.c
+ * routines for inserting tuples into column stores.
+ *
+ * ExecInsertColStoreTuples() is the main entry point. It's called after
+ * inserting a tuple to the heap, and it inserts corresponding values
+ * into all column stores.
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/executor/execColumnStore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/relscan.h"
+#include "catalog/colstore.h"
+#include "colstore/colstoreapi.h"
+#include "executor/executor.h"
+#include "nodes/nodeFuncs.h"
+#include "storage/lmgr.h"
+#include "utils/tqual.h"
+
+/* ----------------------------------------------------------------
+ * ExecOpenColumnStores
+ *
+ * Find the column stores associated with a result relation, open them,
+ * and save information about them in the result ResultRelInfo.
+ *
+ * At entry, caller has already opened and locked
+ * resultRelInfo->ri_RelationDesc.
+ * ----------------------------------------------------------------
+ */
+void
+ExecOpenColumnStores(ResultRelInfo *resultRelInfo)
+{
+ Relation resultRelation = resultRelInfo->ri_RelationDesc;
+ List *colstoreoidlist;
+ ListCell *l;
+ int len,
+ i;
+ RelationPtr relationDescs;
+ ColumnStoreInfo **columnStoreInfoArray;
+
+ resultRelInfo->ri_NumColumnStores = 0;
+
+ /* fast path if no column stores */
+ if (!RelationGetForm(resultRelation)->relhascstore)
+ return;
+
+ /*
+ * Get cached list of colstore OIDs
+ */
+ colstoreoidlist = RelationGetColStoreList(resultRelation);
+ len = list_length(colstoreoidlist);
+ if (len == 0)
+ return;
+
+ /*
+ * allocate space for result arrays
+ */
+ relationDescs = (RelationPtr) palloc(len * sizeof(Relation));
+ columnStoreInfoArray
+ = (ColumnStoreInfo **) palloc(len * sizeof(ColumnStoreInfo *));
+
+ resultRelInfo->ri_NumColumnStores = len;
+ resultRelInfo->ri_ColumnStoreRelationDescs = relationDescs;
+ resultRelInfo->ri_ColumnStoreRelationInfo = columnStoreInfoArray;
+
+ /*
+ * For each column store, open the column store relation and save pg_cstore
+ * info. We acquire RowExclusiveLock, signifying we will update the column
+ * store.
+ */
+ i = 0;
+ foreach(l, colstoreoidlist)
+ {
+ Oid cstoreOid = lfirst_oid(l);
+ Relation cstoreDesc;
+ ColumnStoreInfo *csi;
+
+ cstoreDesc = relation_open(cstoreOid, RowExclusiveLock);
+
+ /* extract column store information from the pg_cstore info */
+ csi = BuildColumnStoreInfo(cstoreDesc);
+
+ relationDescs[i] = cstoreDesc;
+ columnStoreInfoArray[i] = csi;
+ i++;
+ }
+
+ list_free(colstoreoidlist);
+}
+
+/* ----------------------------------------------------------------
+ * ExecCloseColumnStores
+ *
+ * Close the column store relations stored in resultRelInfo
+ * ----------------------------------------------------------------
+ */
+void
+ExecCloseColumnStores(ResultRelInfo *resultRelInfo)
+{
+ int i;
+ int numColumnStores;
+ RelationPtr cstoreDescs;
+
+ numColumnStores = resultRelInfo->ri_NumColumnStores;
+ cstoreDescs = resultRelInfo->ri_ColumnStoreRelationDescs;
+
+ for (i = 0; i < numColumnStores; i++)
+ {
+ if (cstoreDescs[i] == NULL)
+ continue; /* shouldn't happen? */
+
+ /* Drop lock acquired by ExecOpenColumnStores */
+ relation_close(cstoreDescs[i], RowExclusiveLock);
+ }
+
+ /*
+ * XXX should free ColumnStoreInfo array here too? Currently we assume that
+ * such stuff will be cleaned up automatically in FreeExecutorState.
+ */
+}
+
+/* ----------------------------------------------------------------
+ * ExecInsertColStoreTuples
+ *
+ * This routine takes care of inserting column store tuples
+ * into all the relations vertically partitioning the result relation
+ * when a heap tuple is inserted into the result relation.
+ *
+ * CAUTION: this must not be called for a HOT update.
+ * We can't defend against that here for lack of info.
+ * Should we change the API to make it safer?
+ * ----------------------------------------------------------------
+ */
+void
+ExecInsertColStoreTuples(HeapTuple tuple, EState *estate)
+{
+ ResultRelInfo *resultRelInfo;
+ int i;
+ int numColumnStores;
+ RelationPtr relationDescs;
+ Relation heapRelation;
+ ColumnStoreInfo **columnStoreInfoArray;
+ Datum values[INDEX_MAX_KEYS]; /* FIXME INDEX_MAX_KEYS=32 seems a bit low */
+ bool isnull[INDEX_MAX_KEYS];
+ ItemPointer tupleid = &(tuple->t_self);
+ TupleDesc tupdesc;
+
+ /*
+ * Get information from the result relation info structure.
+ */
+ resultRelInfo = estate->es_result_relation_info;
+ numColumnStores = resultRelInfo->ri_NumColumnStores;
+ relationDescs = resultRelInfo->ri_ColumnStoreRelationDescs;
+ columnStoreInfoArray = resultRelInfo->ri_ColumnStoreRelationInfo;
+ heapRelation = resultRelInfo->ri_RelationDesc;
+ tupdesc = RelationGetDescr(resultRelInfo->ri_RelationDesc);
+
+ /*
+ * for each column store, form and insert the tuple
+ */
+ for (i = 0; i < numColumnStores; i++)
+ {
+ Relation cstoreRelation = relationDescs[i];
+ ColumnStoreInfo *cstoreInfo;
+
+ if (cstoreRelation == NULL) /* XXX seems a bit strange ... */
+ continue;
+
+ cstoreInfo = columnStoreInfoArray[i];
+
+ if (cstoreInfo->csi_ColumnStoreRoutine == NULL)
+ elog(ERROR, "column store routine not available");
+
+ if (cstoreInfo->csi_ColumnStoreRoutine->ExecColumnStoreInsert == NULL)
+ elog(ERROR, "ExecColumnStoreInsert routine not available");
+
+ /*
+ * FormColumnStoreDatum fills in its values and isnull parameters with
+ * the appropriate values for the column(s) of the column store.
+ */
+ FormColumnStoreDatum(cstoreInfo, tuple, tupdesc, values, isnull);
+
+ cstoreInfo->csi_ColumnStoreRoutine->ExecColumnStoreInsert(
+ heapRelation, /* heap relation */
+ cstoreRelation, /* column store relation */
+ cstoreInfo, /* column store info */
+ cstoreInfo->csi_NumColumnStoreAttrs,
+ values, /* array of column store Datums */
+ isnull, /* null flags */
+ tupleid); /* tid of heap tuple */
+
+ }
+}
+
+void
+ExecBatchInsertColStoreTuples(int ntuples, HeapTuple *tuples, EState *estate)
+{
+ ResultRelInfo *resultRelInfo;
+ int i, j, natts;
+ int numColumnStores;
+ RelationPtr relationDescs;
+ Relation heapRelation;
+ ColumnStoreInfo **columnStoreInfoArray;
+ Datum **values;
+ bool **isnull;
+ ItemPointer *tupleids;
+ TupleDesc tupdesc;
+
+ /*
+ * Get information from the result relation info structure.
+ */
+ resultRelInfo = estate->es_result_relation_info;
+ numColumnStores = resultRelInfo->ri_NumColumnStores;
+ relationDescs = resultRelInfo->ri_ColumnStoreRelationDescs;
+ columnStoreInfoArray = resultRelInfo->ri_ColumnStoreRelationInfo;
+ heapRelation = resultRelInfo->ri_RelationDesc;
+
+ tupdesc = RelationGetDescr(resultRelInfo->ri_RelationDesc);
+ natts = tupdesc->natts;
+
+ /* we'll build the Datum / isnull arrays once for all the column stores */
+ values = (Datum**)palloc0(sizeof(Datum*) * natts);
+ isnull = ( bool**)palloc0(sizeof(bool*) * natts);
+
+ /* allocate arrays only for attributes that are referenced in column stores */
+ for (i = 0; i < numColumnStores; i++)
+ {
+ ColumnStoreInfo *cstinfo = resultRelInfo->ri_ColumnStoreRelationInfo[i];
+
+ for (j = 0; j < cstinfo->csi_NumColumnStoreAttrs; j++)
+ {
+ AttrNumber attnum = cstinfo->csi_KeyAttrNumbers[j];
+ if (values[attnum-1] == NULL)
+ {
+ values[attnum-1] = (Datum*)palloc0(sizeof(Datum) * ntuples);
+ isnull[attnum-1] = ( bool*)palloc0(sizeof(bool) * ntuples);
+ }
+ }
+ }
+
+ /* we also need an array for tuple IDs */
+ tupleids = (ItemPointer*)palloc0(sizeof(ItemPointer) * ntuples);
+
+ /* populate the arrays */
+ for (i = 0; i < ntuples; i++)
+ {
+ tupleids[i] = &(tuples[i]->t_self);
+
+ for (j = 0; j < natts; j++)
+ {
+ if (values[j] != NULL)
+ values[j][i] = heap_getattr(tuples[i], j, tupdesc, &isnull[j][i]);
+ }
+ }
+
+ /*
+ * for each column store, form and insert the tuple
+ */
+ for (i = 0; i < numColumnStores; i++)
+ {
+ Relation cstoreRelation = relationDescs[i];
+ ColumnStoreInfo *cstoreInfo;
+
+ /* local subset of values */
+ Datum **lvalues;
+ bool **lisnull;
+
+ if (cstoreRelation == NULL) /* XXX seems a bit strange ... */
+ continue;
+
+ cstoreInfo = columnStoreInfoArray[i];
+
+ if (cstoreInfo->csi_ColumnStoreRoutine == NULL)
+ elog(ERROR, "column store routine not available");
+
+ if (cstoreInfo->csi_ColumnStoreRoutine->ExecColumnStoreBatchInsert == NULL)
+ elog(ERROR, "ExecColumnStoreBatchInsert routine not available");
+
+ lvalues = (Datum**)palloc0(sizeof(Datum*) * cstoreInfo->csi_NumColumnStoreAttrs);
+ lisnull = ( bool**)palloc0(sizeof(bool*) * cstoreInfo->csi_NumColumnStoreAttrs);
+
+ for (j = 0; j < cstoreInfo->csi_NumColumnStoreAttrs; j++)
+ {
+ AttrNumber attnum = cstoreInfo->csi_KeyAttrNumbers[j];
+ lvalues[j] = values[attnum-1];
+ lisnull[j] = isnull[attnum-1];
+ }
+
+ cstoreInfo->csi_ColumnStoreRoutine->ExecColumnStoreBatchInsert(
+ heapRelation, /* heap relation */
+ cstoreRelation, /* column store relation */
+ cstoreInfo, /* column store info */
+ ntuples, /* number of rows in the batch */
+ cstoreInfo->csi_NumColumnStoreAttrs,
+ lvalues, /* arrays of column store Datums */
+ lisnull, /* arrays of null flags */
+ tupleids); /* array of tid of heap tuples */
+
+ pfree(lvalues);
+ pfree(lisnull);
+
+ }
+
+ for (j = 0; j < natts; j++)
+ {
+ if (values[j] != NULL)
+ {
+ pfree(values[j]);
+ pfree(isnull[j]);
+ }
+ }
+
+}
diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c
index 2c65a90..f379ff9 100644
--- a/src/backend/executor/execMain.c
+++ b/src/backend/executor/execMain.c
@@ -1473,6 +1473,7 @@ ExecEndPlan(PlanState *planstate, EState *estate)
{
/* Close indices and then the relation itself */
ExecCloseIndices(resultRelInfo);
+ ExecCloseColumnStores(resultRelInfo);
heap_close(resultRelInfo->ri_RelationDesc, NoLock);
resultRelInfo++;
}
@@ -1485,6 +1486,7 @@ ExecEndPlan(PlanState *planstate, EState *estate)
resultRelInfo = (ResultRelInfo *) lfirst(l);
/* Close indices and then the relation itself */
ExecCloseIndices(resultRelInfo);
+ ExecCloseColumnStores(resultRelInfo);
heap_close(resultRelInfo->ri_RelationDesc, NoLock);
}
@@ -2868,6 +2870,7 @@ EvalPlanQualEnd(EPQState *epqstate)
/* Close indices and then the relation itself */
ExecCloseIndices(resultRelInfo);
+ ExecCloseColumnStores(resultRelInfo);
heap_close(resultRelInfo->ri_RelationDesc, NoLock);
}
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 1ef76d0..c43ee9f 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -39,6 +39,7 @@
#include "access/htup_details.h"
#include "access/xact.h"
+#include "catalog/colstore.h"
#include "commands/trigger.h"
#include "executor/executor.h"
#include "executor/nodeModifyTable.h"
@@ -402,11 +403,48 @@ ExecInsert(ModifyTableState *mtstate,
specToken = SpeculativeInsertionLockAcquire(GetCurrentTransactionId());
HeapTupleHeaderSetSpeculativeToken(tuple->t_data, specToken);
- /* insert the tuple, with the speculative token */
- newId = heap_insert(resultRelationDesc, tuple,
- estate->es_output_cid,
- HEAP_INSERT_SPECULATIVE,
- NULL);
+ /*
+ * insert the tuple, with the speculative token
+ *
+ * Note: heap_insert returns the tid (location) of the new tuple in
+ * the t_self field.
+ *
+ * We need to remove the columns that are stored in the column store
+ * from the descriptor and heap tuple, so that we only store the heap
+ * part using heap_insert. We'll create a new tuple descriptor with
+ * only the heap attributes, and create a small 'heap tuple' matching
+ * the descriptor.
+ *
+ * FIXME This is just temporary solution, a bit dirty. Needs to be
+ * done properly (moved to methods, possibly applied to other
+ * places, etc.).
+ */
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ {
+ HeapTuple heaptuple;
+ TupleDesc heapdesc;
+ TupleDesc fulldesc;
+
+ heaptuple = FilterHeapTuple(resultRelInfo, tuple, &heapdesc);
+
+ fulldesc = resultRelationDesc->rd_att;
+ resultRelationDesc->rd_att = heapdesc;
+
+ newId = heap_insert(resultRelationDesc, heaptuple,
+ estate->es_output_cid,
+ HEAP_INSERT_SPECULATIVE,
+ NULL);
+
+ resultRelationDesc->rd_att = fulldesc;
+
+ heap_freetuple(heaptuple);
+ FreeTupleDesc(heapdesc);
+ }
+ else
+ newId = heap_insert(resultRelationDesc, tuple,
+ estate->es_output_cid,
+ HEAP_INSERT_SPECULATIVE,
+ NULL);
/* insert index entries for tuple */
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
@@ -449,9 +487,30 @@ ExecInsert(ModifyTableState *mtstate,
* Note: heap_insert returns the tid (location) of the new tuple
* in the t_self field.
*/
- newId = heap_insert(resultRelationDesc, tuple,
- estate->es_output_cid,
- 0, NULL);
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ {
+ HeapTuple heaptuple;
+ TupleDesc heapdesc;
+ TupleDesc fulldesc;
+
+ heaptuple = FilterHeapTuple(resultRelInfo, tuple, &heapdesc);
+
+ fulldesc = resultRelationDesc->rd_att;
+ resultRelationDesc->rd_att = heapdesc;
+
+ newId = heap_insert(resultRelationDesc, heaptuple,
+ estate->es_output_cid,
+ 0, NULL);
+
+ resultRelationDesc->rd_att = fulldesc;
+
+ heap_freetuple(heaptuple);
+ FreeTupleDesc(heapdesc);
+ }
+ else
+ newId = heap_insert(resultRelationDesc, tuple,
+ estate->es_output_cid,
+ 0, NULL);
/* insert index entries for tuple */
if (resultRelInfo->ri_NumIndices > 0)
@@ -459,6 +518,12 @@ ExecInsert(ModifyTableState *mtstate,
estate, false, NULL,
arbiterIndexes);
}
+
+ /*
+ * insert column store entries for tuple
+ */
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ ExecInsertColStoreTuples(tuple, estate);
}
if (canSetTag)
@@ -880,12 +945,47 @@ lreplace:;
* can't-serialize error if not. This is a special-case behavior
* needed for referential integrity updates in transaction-snapshot
* mode transactions.
+ *
+ * We need to remove the columns that are stored in the column store
+ * from the descriptor and heap tuple, so that we only store the heap
+ * part using heap_insert. We'll create a new tuple descriptor with
+ * only the heap attributes, and create a small 'heap tuple' matching
+ * the descriptor.
+ *
+ * FIXME This is just temporary solution, a bit dirty. Needs to be
+ * done properly (moved to methods, possibly applied to other
+ * places, etc.).
*/
- result = heap_update(resultRelationDesc, tupleid, tuple,
- estate->es_output_cid,
- estate->es_crosscheck_snapshot,
- true /* wait for commit */ ,
- &hufd, &lockmode);
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ {
+ HeapTuple heaptuple;
+ TupleDesc heapdesc;
+ TupleDesc fulldesc;
+
+ heaptuple = FilterHeapTuple(resultRelInfo, tuple, &heapdesc);
+
+ fulldesc = resultRelationDesc->rd_att;
+ resultRelationDesc->rd_att = heapdesc;
+
+ result = heap_update(resultRelationDesc, tupleid, heaptuple,
+ estate->es_output_cid,
+ estate->es_crosscheck_snapshot,
+ true /* wait for commit */ ,
+ &hufd, &lockmode);
+
+ resultRelationDesc->rd_att = fulldesc;
+
+ heap_freetuple(heaptuple);
+ FreeTupleDesc(heapdesc);
+ }
+ else
+ result = heap_update(resultRelationDesc, tupleid, tuple,
+ estate->es_output_cid,
+ estate->es_crosscheck_snapshot,
+ true /* wait for commit */ ,
+ &hufd, &lockmode);
+
+
switch (result)
{
case HeapTupleSelfUpdated:
@@ -966,16 +1066,20 @@ lreplace:;
*/
/*
- * insert index entries for tuple
+ * insert index and column store entries for tuple
*
* Note: heap_update returns the tid (location) of the new tuple in
* the t_self field.
*
- * If it's a HOT update, we mustn't insert new index entries.
+ * If it's a HOT update, we mustn't insert new index and column store
+ * entries.
*/
if (resultRelInfo->ri_NumIndices > 0 && !HeapTupleIsHeapOnly(tuple))
recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self),
estate, false, NULL, NIL);
+
+ if (resultRelInfo->ri_NumColumnStores > 0 && !HeapTupleIsHeapOnly(tuple))
+ ExecInsertColStoreTuples(tuple, estate);
}
if (canSetTag)
@@ -1546,6 +1650,9 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags)
resultRelInfo->ri_IndexRelationDescs == NULL)
ExecOpenIndices(resultRelInfo, mtstate->mt_onconflict != ONCONFLICT_NONE);
+ /* TODO should use relhascolstore just like indexes*/
+ ExecOpenColumnStores(resultRelInfo);
+
/* Now init the plan for this result rel */
estate->es_result_relation_info = resultRelInfo;
mtstate->mt_plans[i] = ExecInitNode(subplan, estate, eflags);
diff --git a/src/include/catalog/colstore.h b/src/include/catalog/colstore.h
index 20be191..93806d2 100644
--- a/src/include/catalog/colstore.h
+++ b/src/include/catalog/colstore.h
@@ -4,6 +4,7 @@
#include "nodes/execnodes.h"
#include "nodes/nodes.h"
#include "nodes/parsenodes.h"
+#include "nodes/execnodes.h"
#include "nodes/pg_list.h"
#include "utils/relcache.h"
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h
index 226f905..16dd529 100644
--- a/src/include/executor/executor.h
+++ b/src/include/executor/executor.h
@@ -375,5 +375,13 @@ extern void check_exclusion_constraint(Relation heap, Relation index,
Datum *values, bool *isnull,
EState *estate, bool newIndex);
+/*
+ * prototypes from functions in execColumnStore.c
+ */
+extern void ExecOpenColumnStores(ResultRelInfo *resultRelInfo);
+extern void ExecCloseColumnStores(ResultRelInfo *resultRelInfo);
+extern void ExecInsertColStoreTuples(HeapTuple tuple, EState *estate);
+extern void ExecBatchInsertColStoreTuples(int ntuples, HeapTuple *tuples,
+ EState *estate);
#endif /* EXECUTOR_H */
--
2.1.4
["0022-COPY-use-colstore-batch-stuff.patch" (text/x-diff)]
From ce19e0ec6e278b31b19cdd623686e1081fd1a04b Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Sun, 30 Aug 2015 01:25:38 -0300
Subject: [PATCH 22/24] COPY: use colstore batch stuff
---
src/backend/commands/copy.c | 66 ++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 60 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 2d397d1..e98f0fe 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -25,6 +25,7 @@
#include "access/sysattr.h"
#include "access/xact.h"
#include "access/xlog.h"
+#include "catalog/colstore.h"
#include "catalog/namespace.h"
#include "catalog/pg_type.h"
#include "commands/copy.h"
@@ -2452,6 +2453,12 @@ CopyFrom(CopyState cstate)
{
List *recheckIndexes = NIL;
+ /*
+ * FIXME This needs to handle the column stores (it's not
+ * handled by the batching code because of the before
+ * row insert triggers or something).
+ */
+
/* OK, store the tuple and create index entries for it */
heap_insert(cstate->rel, tuple, mycid, hi_options, bistate);
@@ -2550,17 +2557,64 @@ CopyFromInsertBatch(CopyState cstate, EState *estate, CommandId mycid,
/*
* heap_multi_insert leaks memory, so switch to short-lived memory context
* before calling it.
+ *
+ * FIXME If there are column stores, we need to filter only the heap part of
+ * the tuples, similarly to ExecInsert/ExecUpdate.
*/
oldcontext = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));
- heap_multi_insert(cstate->rel,
- bufferedTuples,
- nBufferedTuples,
- mycid,
- hi_options,
- bistate);
+
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ {
+ TupleDesc heapdesc, fulldesc;
+ HeapTuple *filtered = (HeapTuple*)palloc0(sizeof(HeapTuple) * nBufferedTuples);
+
+ ResultRelInfo *resultRelInfo = estate->es_result_relation_info;
+ Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
+
+ /* build the filtered tuples with only the heap part */
+ for (i = 0; i < nBufferedTuples; i++)
+ filtered[i] = FilterHeapTuple(resultRelInfo,
+ bufferedTuples[i], &heapdesc);
+
+ fulldesc = resultRelationDesc->rd_att;
+ resultRelationDesc->rd_att = heapdesc;
+
+ heap_multi_insert(cstate->rel,
+ filtered,
+ nBufferedTuples,
+ mycid,
+ hi_options,
+ bistate);
+
+ resultRelationDesc->rd_att = fulldesc;
+
+ /* now we need to transfer the item pointers to the original tuples */
+ for (i = 0; i < nBufferedTuples; i++)
+ ItemPointerCopy(&(filtered[i]->t_self), &(bufferedTuples[i]->t_self));
+
+ }
+ else /* no column stores, so do the insert with original tuples */
+ heap_multi_insert(cstate->rel,
+ bufferedTuples,
+ nBufferedTuples,
+ mycid,
+ hi_options,
+ bistate);
+
MemoryContextSwitchTo(oldcontext);
/*
+ * If there are any column stores on the table, insert the data into them
+ * now. This needs to happen before indexes, because we're running AFTER
+ * ROW INSERT triggers in there, and those may do lookups from this table.
+ *
+ * If some of the indexes rechecks fail, that's not a big problem. We'll
+ * eventually remove the values from the columns stores during VACUUM.
+ */
+ if (resultRelInfo->ri_NumColumnStores > 0)
+ ExecBatchInsertColStoreTuples(nBufferedTuples, bufferedTuples, estate);
+
+ /*
* If there are any indexes, update them for all the inserted tuples, and
* run AFTER ROW INSERT triggers.
*/
--
2.1.4
["0023-regression-tests-for-cstore.patch" (text/x-diff)]
From 8bea1dc29e059513f7bcb7507012d4d13dc06400 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <tomas@pgaddict.com>
Date: Sun, 12 Jul 2015 20:30:58 +0200
Subject: [PATCH 23/24] regression tests for cstore
---
src/test/regress/expected/cstore.out | 367 +++++++++++++++++++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/serial_schedule | 1 +
src/test/regress/sql/cstore.sql | 263 +++++++++++++++++++++++++
4 files changed, 632 insertions(+), 1 deletion(-)
create mode 100644 src/test/regress/expected/cstore.out
create mode 100644 src/test/regress/sql/cstore.sql
diff --git a/src/test/regress/expected/cstore.out \
b/src/test/regress/expected/cstore.out new file mode 100644
index 0000000..dfe1689
--- /dev/null
+++ b/src/test/regress/expected/cstore.out
@@ -0,0 +1,367 @@
+-- create two fake cstore AM instances (no implementation)
+CREATE COLUMN STORE ACCESS METHOD store_am_one HANDLER cstore_dummy_handler;
+CREATE COLUMN STORE ACCESS METHOD store_am_two HANDLER cstore_dummy_handler;
+-- column-level column store definition
+-- missing USING clause
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE foo,
+ c INT
+);
+ERROR: syntax error at or near ","
+LINE 3: b INT COLUMN STORE foo,
+ ^
+-- missing column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE USING store_am_one,
+ c INT
+);
+ERROR: syntax error at or near "USING"
+LINE 3: b INT COLUMN STORE USING store_am_one,
+ ^
+-- missing USING and column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE,
+ c INT
+);
+ERROR: syntax error at or near ","
+LINE 3: b INT COLUMN STORE,
+ ^
+-- missing column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE USING store_am_one,
+ c INT
+);
+ERROR: syntax error at or near "USING"
+LINE 3: b INT COLUMN STORE USING store_am_one,
+ ^
+-- unknown name of a column store AM
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE foo USING no_store_am,
+ c INT
+);
+ERROR: column store access method "no_store_am" does not exist
+-- conflicting column store name
+CREATE TABLE test_columnar_single_conflict (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT COLUMN STORE foo USING store_am_one,
+ d INT
+);
+ERROR: duplicate column store name "foo"
+-- correct definition (single store)
+CREATE TABLE test_columnar_single_ok (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT
+);
+\d test_columnar_single_ok
+Table "public.test_columnar_single_ok"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+Column stores: foo1 USING store_am_one ({b})
+
+-- correct definition (two stores)
+CREATE TABLE test_columnar_single_ok2 (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT COLUMN STORE foo2 USING store_am_two,
+ d INT
+);
+\d test_columnar_single_ok2
+Table "public.test_columnar_single_ok2"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+Column stores: foo1 USING store_am_one ({b}),
+ foo2 USING store_am_two ({c})
+
+-- table-level column store definition
+-- no column list
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one
+);
+ERROR: syntax error at or near ")"
+LINE 7: );
+ ^
+-- empty column list
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one ()
+);
+ERROR: syntax error at or near ")"
+LINE 6: COLUMN STORE foo USING store_am_one ()
+ ^
+-- invalid column in store
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (z)
+);
+ERROR: no column "z" in the table
+-- unknown name of a column store AM
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING no_store_am (b,c)
+);
+ERROR: column store access method "no_store_am" does not exist
+-- conflicting column store name
+CREATE TABLE test_columnar_multi_conflict (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (a,b),
+ COLUMN STORE foo USING store_am_one (c,d)
+);
+ERROR: duplicate column store name "foo"
+-- overlapping list of columns
+CREATE TABLE test_columnar_multi_conflict2 (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo1 USING store_am_one (a,b),
+ COLUMN STORE foo2 USING store_am_two (b,c,d)
+);
+ERROR: column already in a store
+-- correct definition (single store)
+CREATE TABLE test_columnar_multi_ok (
+ a INT,
+ b INT,
+ c INT,
+ COLUMN STORE foo USING store_am_one (a,b)
+);
+\d test_columnar_multi_ok
+Table "public.test_columnar_multi_ok"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+Column stores: foo USING store_am_one ({a,b})
+
+-- correct definition (two stores)
+CREATE TABLE test_columnar_multi_ok2 (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo1 USING store_am_one (a,b),
+ COLUMN STORE foo2 USING store_am_one (c,d)
+);
+\d test_columnar_multi_ok2
+Table "public.test_columnar_multi_ok2"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+Column stores: foo1 USING store_am_one ({a,b}),
+ foo2 USING store_am_one ({c,d})
+
+-- combination of column-level and table-level column stores
+-- conflicting column store name
+CREATE TABLE test_columnar_multi_conflict (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (c,d)
+);
+ERROR: duplicate column store name "foo"
+-- overlapping list of columns
+CREATE TABLE test_columnar_combi_conflict2 (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo2 USING store_am_two (b,c,d)
+);
+ERROR: column already in a store
+-- correct definition (two stores)
+CREATE TABLE test_columnar_combi_ok (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo2 USING store_am_one (c,d)
+);
+\d test_columnar_combi_ok
+Table "public.test_columnar_combi_ok"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+Column stores: foo USING store_am_one ({b}),
+ foo2 USING store_am_one ({c,d})
+
+-- test cleanup
+CREATE TABLE cstore_oids AS
+SELECT cststoreid
+ FROM pg_cstore JOIN pg_class ON (pg_cstore.cstrelid = pg_class.oid)
+ WHERE relname IN ('test_columnar_single_ok',
+ 'test_columnar_single_ok2',
+ 'test_columnar_multi_ok',
+ 'test_columnar_multi_ok2',
+ 'test_columnar_combi_ok');
+CREATE TABLE cstore_oids_2 AS
+SELECT pg_class.oid
+ FROM pg_class JOIN cstore_oids ON (pg_class.oid = cstore_oids.cststoreid);
+DROP TABLE test_columnar_single_ok;
+DROP TABLE test_columnar_single_ok2;
+DROP TABLE test_columnar_multi_ok;
+DROP TABLE test_columnar_multi_ok2;
+DROP TABLE test_columnar_combi_ok;
+-- should return 0
+SELECT COUNT(*) FROM pg_class WHERE oid IN (SELECT cststoreid FROM cstore_oids);
+ count
+-------
+ 0
+(1 row)
+
+SELECT COUNT(*) FROM pg_class WHERE oid IN (SELECT oid FROM cstore_oids_2);
+ count
+-------
+ 0
+(1 row)
+
+SELECT COUNT(*) FROM pg_cstore WHERE cststoreid IN (SELECT oid FROM cstore_oids);
+ count
+-------
+ 0
+(1 row)
+
+SELECT COUNT(*) FROM pg_attribute WHERE attrelid IN (SELECT cststoreid FROM \
cstore_oids); + count
+-------
+ 0
+(1 row)
+
+SELECT COUNT(*) FROM pg_attribute WHERE attrelid IN (SELECT oid FROM cstore_oids_2);
+ count
+-------
+ 0
+(1 row)
+
+DROP TABLE cstore_oids;
+DROP TABLE cstore_oids_2;
+-- INHERITANCE
+-- parent table with two column stores
+CREATE TABLE parent_table (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT,
+ d INT,
+ e INT,
+ COLUMN STORE foo2 USING store_am_two (d,e)
+);
+-- child table with two separate column stores
+CREATE TABLE child_table_1 (
+ f INT,
+ g INT COLUMN STORE foo1c USING store_am_one,
+ h INT,
+ i INT,
+ COLUMN STORE foo2c USING store_am_two(h,i)
+) INHERITS (parent_table);
+-- FIXME BUG: should have cstores foo1 and foo2
+\d child_table_1
+ Table "public.child_table_1"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+ e | integer |
+ f | integer |
+ g | integer |
+ h | integer |
+ i | integer |
+Inherits: parent_table
+Column stores: foo1c USING store_am_one ({g}),
+ foo2c USING store_am_two ({h,i})
+
+-- child table with two column stores - one modifying, one redefining the parent
+CREATE TABLE child_table_2 (
+ f INT,
+ g INT COLUMN STORE foo1c USING store_am_one, -- new column store
+ h INT,
+ i INT,
+ COLUMN STORE foo2c USING store_am_two(b,h,i) -- redefines the parent colstore
+) INHERITS (parent_table);
+\d child_table_2
+ Table "public.child_table_2"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+ e | integer |
+ f | integer |
+ g | integer |
+ h | integer |
+ i | integer |
+Inherits: parent_table
+Column stores: foo2c USING store_am_two ({b,h,i}),
+ foo1c USING store_am_one ({g})
+
+-- child table with a single column store of the whole table
+CREATE TABLE child_table_3 (
+ f INT,
+ g INT,
+ h INT,
+ i INT,
+ COLUMN STORE foo1 USING store_am_one(a,b,c,d,e,f,g,h,i)
+) INHERITS (parent_table);
+\d child_table_3
+ Table "public.child_table_3"
+ Column | Type | Modifiers
+--------+---------+-----------
+ a | integer |
+ b | integer |
+ c | integer |
+ d | integer |
+ e | integer |
+ f | integer |
+ g | integer |
+ h | integer |
+ i | integer |
+Inherits: parent_table
+Column stores: foo1 USING store_am_one ({a,b,c,d,e,f,g,h,i})
+
+--- FIXME -- add tests with multiple inheritance
+DROP TABLE parent_table CASCADE;
+NOTICE: drop cascades to 3 other objects
+DETAIL: drop cascades to table child_table_1
+drop cascades to table child_table_2
+drop cascades to table child_table_3
+--- delete the fake cstore AM records
+-- FIXME -- this should be a DROP command
+DELETE FROM pg_cstore_am WHERE cstamname IN ('store_am_one', 'store_am_two');
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index 6fc5d1e..352c92e 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -60,7 +60,7 @@ test: create_index create_view
# ----------
# Another group of parallel tests
# ----------
-test: create_aggregate create_function_3 create_cast constraints triggers inherit \
create_table_like typed_table vacuum drop_if_exists updatable_views rolenames \
roleattributes +test: create_aggregate create_function_3 create_cast constraints \
triggers inherit create_table_like typed_table vacuum drop_if_exists updatable_views \
rolenames roleattributes cstore
# ----------
# sanity_check does a vacuum, affecting the sort order of SELECT *
diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule
index 2ae51cf..164d836 100644
--- a/src/test/regress/serial_schedule
+++ b/src/test/regress/serial_schedule
@@ -74,6 +74,7 @@ test: drop_if_exists
test: updatable_views
test: rolenames
test: roleattributes
+test: cstore
test: sanity_check
test: errors
test: select
diff --git a/src/test/regress/sql/cstore.sql b/src/test/regress/sql/cstore.sql
new file mode 100644
index 0000000..aa589b4
--- /dev/null
+++ b/src/test/regress/sql/cstore.sql
@@ -0,0 +1,263 @@
+-- create two fake cstore AM instances (no implementation)
+CREATE COLUMN STORE ACCESS METHOD store_am_one HANDLER cstore_dummy_handler;
+CREATE COLUMN STORE ACCESS METHOD store_am_two HANDLER cstore_dummy_handler;
+
+-- column-level column store definition
+
+-- missing USING clause
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE foo,
+ c INT
+);
+
+-- missing column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE USING store_am_one,
+ c INT
+);
+
+-- missing USING and column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE,
+ c INT
+);
+
+-- missing column store name
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE USING store_am_one,
+ c INT
+);
+
+-- unknown name of a column store AM
+CREATE TABLE test_columnar_single_missing (
+ a INT,
+ b INT COLUMN STORE foo USING no_store_am,
+ c INT
+);
+
+-- conflicting column store name
+CREATE TABLE test_columnar_single_conflict (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT COLUMN STORE foo USING store_am_one,
+ d INT
+);
+
+-- correct definition (single store)
+CREATE TABLE test_columnar_single_ok (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT
+);
+
+\d test_columnar_single_ok
+
+-- correct definition (two stores)
+CREATE TABLE test_columnar_single_ok2 (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT COLUMN STORE foo2 USING store_am_two,
+ d INT
+);
+
+\d test_columnar_single_ok2
+
+-- table-level column store definition
+
+-- no column list
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one
+);
+
+-- empty column list
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one ()
+);
+
+-- invalid column in store
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (z)
+);
+
+-- unknown name of a column store AM
+CREATE TABLE test_columnar_multi_missing (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING no_store_am (b,c)
+);
+
+-- conflicting column store name
+CREATE TABLE test_columnar_multi_conflict (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (a,b),
+ COLUMN STORE foo USING store_am_one (c,d)
+);
+
+-- overlapping list of columns
+CREATE TABLE test_columnar_multi_conflict2 (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo1 USING store_am_one (a,b),
+ COLUMN STORE foo2 USING store_am_two (b,c,d)
+);
+
+-- correct definition (single store)
+CREATE TABLE test_columnar_multi_ok (
+ a INT,
+ b INT,
+ c INT,
+ COLUMN STORE foo USING store_am_one (a,b)
+);
+
+\d test_columnar_multi_ok
+
+-- correct definition (two stores)
+CREATE TABLE test_columnar_multi_ok2 (
+ a INT,
+ b INT,
+ c INT,
+ d INT,
+ COLUMN STORE foo1 USING store_am_one (a,b),
+ COLUMN STORE foo2 USING store_am_one (c,d)
+);
+
+\d test_columnar_multi_ok2
+
+-- combination of column-level and table-level column stores
+
+-- conflicting column store name
+CREATE TABLE test_columnar_multi_conflict (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo USING store_am_one (c,d)
+);
+
+-- overlapping list of columns
+CREATE TABLE test_columnar_combi_conflict2 (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo2 USING store_am_two (b,c,d)
+);
+
+-- correct definition (two stores)
+CREATE TABLE test_columnar_combi_ok (
+ a INT,
+ b INT COLUMN STORE foo USING store_am_one,
+ c INT,
+ d INT,
+ COLUMN STORE foo2 USING store_am_one (c,d)
+);
+
+\d test_columnar_combi_ok
+
+-- test cleanup
+CREATE TABLE cstore_oids AS
+SELECT cststoreid
+ FROM pg_cstore JOIN pg_class ON (pg_cstore.cstrelid = pg_class.oid)
+ WHERE relname IN ('test_columnar_single_ok',
+ 'test_columnar_single_ok2',
+ 'test_columnar_multi_ok',
+ 'test_columnar_multi_ok2',
+ 'test_columnar_combi_ok');
+
+CREATE TABLE cstore_oids_2 AS
+SELECT pg_class.oid
+ FROM pg_class JOIN cstore_oids ON (pg_class.oid = cstore_oids.cststoreid);
+
+DROP TABLE test_columnar_single_ok;
+DROP TABLE test_columnar_single_ok2;
+DROP TABLE test_columnar_multi_ok;
+DROP TABLE test_columnar_multi_ok2;
+DROP TABLE test_columnar_combi_ok;
+
+-- should return 0
+SELECT COUNT(*) FROM pg_class WHERE oid IN (SELECT cststoreid FROM cstore_oids);
+SELECT COUNT(*) FROM pg_class WHERE oid IN (SELECT oid FROM cstore_oids_2);
+
+SELECT COUNT(*) FROM pg_cstore WHERE cststoreid IN (SELECT oid FROM cstore_oids);
+
+SELECT COUNT(*) FROM pg_attribute WHERE attrelid IN (SELECT cststoreid FROM \
cstore_oids); +SELECT COUNT(*) FROM pg_attribute WHERE attrelid IN (SELECT oid FROM \
cstore_oids_2); +
+DROP TABLE cstore_oids;
+DROP TABLE cstore_oids_2;
+
+-- INHERITANCE
+
+-- parent table with two column stores
+CREATE TABLE parent_table (
+ a INT,
+ b INT COLUMN STORE foo1 USING store_am_one,
+ c INT,
+ d INT,
+ e INT,
+ COLUMN STORE foo2 USING store_am_two (d,e)
+);
+
+-- child table with two separate column stores
+CREATE TABLE child_table_1 (
+ f INT,
+ g INT COLUMN STORE foo1c USING store_am_one,
+ h INT,
+ i INT,
+ COLUMN STORE foo2c USING store_am_two(h,i)
+) INHERITS (parent_table);
+-- FIXME BUG: should have cstores foo1 and foo2
+\d child_table_1
+
+-- child table with two column stores - one modifying, one redefining the parent
+CREATE TABLE child_table_2 (
+ f INT,
+ g INT COLUMN STORE foo1c USING store_am_one, -- new column store
+ h INT,
+ i INT,
+ COLUMN STORE foo2c USING store_am_two(b,h,i) -- redefines the parent colstore
+) INHERITS (parent_table);
+
+\d child_table_2
+
+-- child table with a single column store of the whole table
+CREATE TABLE child_table_3 (
+ f INT,
+ g INT,
+ h INT,
+ i INT,
+ COLUMN STORE foo1 USING store_am_one(a,b,c,d,e,f,g,h,i)
+) INHERITS (parent_table);
+
+\d child_table_3
+
+--- FIXME -- add tests with multiple inheritance
+
+DROP TABLE parent_table CASCADE;
+
+--- delete the fake cstore AM records
+-- FIXME -- this should be a DROP command
+DELETE FROM pg_cstore_am WHERE cstamname IN ('store_am_one', 'store_am_two');
--
2.1.4
["0024-Add-known-bugs-file.patch" (text/x-diff)]
From 253f487d17adac97e4b27f5755e6d88e84390ef2 Mon Sep 17 00:00:00 2001
From: Simon Riggs <simon@2ndQuadrant.com>
Date: Fri, 21 Aug 2015 19:22:42 +0100
Subject: [PATCH 24/24] Add known-bugs file
---
known-bugs | 10 ++++++++++
1 file changed, 10 insertions(+)
create mode 100644 known-bugs
diff --git a/known-bugs b/known-bugs
new file mode 100644
index 0000000..de7ce06
--- /dev/null
+++ b/known-bugs
@@ -0,0 +1,10 @@
+Known Bugs, Issues and Limitations
+
+* Heisenbug: shared memory error from >100 column stores
+ Simon Riggs can reproduce this, no one else can.
+
+* ERROR: cache lookup failed for column store 160595
+ This error shows up inexplicably when a table with over 100 column stores is
+ dropped. (Alvaro's guess is that it's related to pg_depend handling. Gotta
+ rework that and remove OIDs from pg_cstore).
+
--
2.1.4
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic