[prev in list] [next in list] [prev in thread] [next in thread]
List: monetdb-checkins
Subject: MonetDB: wlcrNov2019 - Fixing the WLCR code the system seems to ...
From: Martin Kersten <commits+mk=cwi.nl () monetdb ! org>
Date: 2019-09-30 11:38:24
Message-ID: hg.e0e23e007d7b.1569843504.6315528441665844383 () monetdb-vm0 ! spin-off ! cwi ! nl
[Download RAW message or body]
Changeset: e0e23e007d7b for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e0e23e007d7b
Added Files:
sql/scripts/61_wlcr.sql
sql/test/wlcr/Tests/wlr10.py
sql/test/wlcr/Tests/wlr10.stable.err
sql/test/wlcr/Tests/wlr10.stable.out
Modified Files:
monetdb5/modules/mal/wlc.c
monetdb5/modules/mal/wlc.h
monetdb5/modules/mal/wlc.mal
sql/backends/monet5/sql_scenario.c
sql/backends/monet5/wlr.c
sql/backends/monet5/wlr.h
sql/backends/monet5/wlr.mal
sql/scripts/60_wlcr.sql
sql/scripts/Makefile.ag
sql/test/wlcr/Tests/All
sql/test/wlcr/Tests/wlc01.py
sql/test/wlcr/Tests/wlc01.stable.out
sql/test/wlcr/Tests/wlc100.py
sql/test/wlcr/Tests/wlc20.py
sql/test/wlcr/Tests/wlc21.py
sql/test/wlcr/Tests/wlc30.py
sql/test/wlcr/Tests/wlc40.py
sql/test/wlcr/Tests/wlc50.py
sql/test/wlcr/Tests/wlc70.py
sql/test/wlcr/Tests/wlr01.py
sql/test/wlcr/Tests/wlr01.stable.err
sql/test/wlcr/Tests/wlr01.stable.out
sql/test/wlcr/Tests/wlr100.py
sql/test/wlcr/Tests/wlr20.py
sql/test/wlcr/Tests/wlr20.stable.err
sql/test/wlcr/Tests/wlr20.stable.out
sql/test/wlcr/Tests/wlr30.py
sql/test/wlcr/Tests/wlr40.py
sql/test/wlcr/Tests/wlr50.py
sql/test/wlcr/Tests/wlr70.py
Branch: wlcrNov2019
Log Message:
Fixing the WLCR code the system seems to generate a secondary
call to the SQL upgrade code when Mtesting is called.
The latest step was the addition of 61_wlcr.sql script
diffs (truncated from 2762 to 300 lines):
diff --git a/monetdb5/modules/mal/wlc.c b/monetdb5/modules/mal/wlc.c
--- a/monetdb5/modules/mal/wlc.c
+++ b/monetdb5/modules/mal/wlc.c
@@ -25,6 +25,7 @@
* Some systems also use the logical logs to REPLAY all (expensive) queries
* against the database. We skip this for the time being, as those queries
* can be captured already in the server.
+ * [A flag should be added to at least capture them]
*
* The goal of this module is to ease BACKUP and REPLICATION of a master database
* with a time-bounded delay. This means that both master and replica run at a \
certain beat @@ -44,7 +45,7 @@
* from most storage system related failures, e.g. using RAID disks or LSF systems.
*
* A database can be set into 'master' mode only once using the SQL command:
- * CALL master()
+ * CALL wrc_master.master() whose access permission is limited to the 'monetdb' \
user.[CHECK]
* An optional path to the log record directory can be given to reduce the IO \
latency,
* e.g. using a nearby SSD, or where there is ample of space to keep a long history,
* e.g. a HDD or cold storage location.
@@ -62,19 +63,22 @@
* A missing path to the snapshot denotes that we can start the clone with an empty \
database.
* The log files are stored as master/<dbname>_<batchnumber>. They belong to the \
snapshot.
*
- * Each wlc log file contains a serial log of committed compound transactions.
+ * Each wlc log file contains a serial log of a number of committed compound \
transactions.
* The log records are represented as ordinary MAL statement blocks, which
* are executed in serial mode. (parallelism can be considered for large updates \
later)
- * Each transaction job is identified by a unique id, its starting time, and the \
user responsible..
- * The log-record should end with a commit to be allowed for re-execution.
- * Log records with a rollback tag are merely for analysis by the DBA.
+ * Each transaction job is identified by a unique id, its starting time, and the \
original responsible user. + * Each log-record should end with a commit to be allowed \
for re-execution. + * Log records with a rollback tag are merely for analysis by the \
DBA, their statements are ignored.
*
* A transaction log file is created by the master using a heartbeat (in seconds).
* A new transaction log file is published when the system has been collecting \
transaction records for some time.
* The beat can be set using a SQL command, e.g.
- * CALL masterbeat(duration)
- * Setting it to zero leads to a log file per transaction and may cause a large log \
directory.
- * A default of 5 minutes should balance polling overhead in most practical \
situations. + * CALL wcr_master.beat(duration)
+ * Setting it to zero leads to a log file per transaction and may cause a large log \
directory + * with thousands of small files.
+ * The default of 5 minutes should balance polling overhead in most practical \
situations. + * Intermittent flush() during this period ensures the committed log \
records survive + * a crash.
*
* A minor problem here is that we should ensure that the log file is closed even if \
there
* are no transactions running. It is solved with a separate monitor thread, which \
ensures @@ -82,15 +86,21 @@
* After closing, the replicas can see from the master configuration file that a new \
log batch is available.
*
* The final step is to close stop transaction logging with the command
- * CALL stopmaster().
+ * CALL wcr_master.stop().
* It typically is the end-of-life-time for a snapshot. For example, when planning \
to do
* a large bulk load of the database, stopping logging avoids a double write into \
the
* database. The database can only be brought back into master mode using a fresh \
snapshot.
*
+ * [It is not advicable to temporarily stop logging and continue afterwards, because \
then there + * is no guarantee the user will see a consistent database.]
+ *
* One of the key challenges for a DBA is to keep the log directory manageable, \
because it grows
* with the speed up updates being applied to the database. This calls for regularly \
checking
* for their disk footprint and taking a new snapshot as a frame of reference.
*
+ * [TODO A trigger should be added to stop logging and call for a fresh snapshot \
first] + * [TODO the batch files might include the snapshot id for ease of rebuild]
+ *
* The DBA tool 'monetdb' provides options to create a master and its replicas.
* It will also maintain the list of replicas for inspection and managing their \
drift.
* For example,
@@ -101,16 +111,15 @@
* monetdb replicate <dbname> <mastername>
*
* Instead of using the monetdb command line we can use the SQL calls directly
- * master() and replicate(), provided we start with a fresh database.
+ * sys.master() and sys.replicate(), provided we start with a fresh database.
*
* CLONE
*
* Every clone should start off with a copy of the binary snapshot identified by \
'snapshot'.
* A fresh database can be turned into a clone using the call
- * CALL replicate('mastername')
+ * CALL wcr_replica.master('mastername')
* It will grab the latest snapshot of the master and applies all
- * available log files before releasing the system. Progress of
- * the replication can be monitored using the -fraw option in mclient.
+ * available log files before releasing the system.
* The master has no knowledge about the number of clones and their whereabouts.
*
* The clone process will iterate in the background through the log files,
@@ -120,20 +129,23 @@
* apply the logs until a given moment. This is particularly handy when an \
unexpected
* desastrous user action (drop persistent table) has to be recovered from.
*
- * CALL replicate('mastername');
- * CALL replicate('mastername',NOW()); -- stops after we are in sink
+ * CALL wcr_replica.master('mastername'); -- get logs from a specific master
+ * ...
+ * CALL wcr_replicate.replicate(tag); -- stops after we are in sink with tag
* ...
- * CALL replicate(NOW()); -- partial roll forward
+ * CALL wcr_replicate.replicate(NOW()); -- stop after we sinked all transactions
* ...
- * CALL replicate(); --continue nondisturbed synchronisation
+ * CALL wcr_replicate.replicate(); -- synchronize in background continuously
+ * ...
+ * CALL wcr_replicate.stop(); -- stop the synchroniation thread
*
- * SELECT replicaClock();
+ * SELECT wcr_replica.clock();
* returns the timestamp of the last replicated transaction.
- * SELECT replicaTick();
+ * SELECT wcr_replica.tick();
* returns the transaction id of the last replicated transaction.
- * SELECT masterClock();
+ * SELECT wcr_master.clock();
* return the timestamp of the last committed transaction in the master.
- * SELECT masterTick();
+ * SELECT wcr_master.tick();
* return the transaction id of the last committed transaction in the master.
*
* Any failure encountered during a log replay terminates the replication process,
@@ -142,7 +154,8 @@
* The wlc files purposely have a textual format derived from the MAL statements.
* This provides a stepping stone for remote execution later.
*
- * [TODO] consider the roll forward of SQL session variables, i.e. optimizer_pipe
+ * [TODO consider the roll logging of SQL session variables, i.e. optimizer_pipe
+ * as part of the log record]
* For updates we don't need special care for this.
*/
#include "monetdb_config.h"
@@ -150,6 +163,8 @@
#include "mal_builder.h"
#include "wlc.h"
+#undef _WLC_DEBUG_
+
MT_Lock wlc_lock = MT_LOCK_INITIALIZER("wlc_lock");
static char wlc_snapshot[FILENAME_MAX]; // The location of the snapshot against \
which the logs work @@ -158,8 +173,8 @@ static stream *wlc_fd = 0;
// These properties are needed by the replica to direct the roll-forward.
char wlc_dir[FILENAME_MAX]; // The location in the global file store for the logs
char wlc_name[IDLENGTH]; // The master database name
-lng wlc_id = 0; // next transaction id
-int wlc_state = 0; // The current status of the in the life cycle
+lng wlc_tag = 0; // next transaction id
+int wlc_state = 0; // The current status of the logger in the life cycle
char wlc_write[26]; // The timestamp of the last committed transaction
int wlc_batches = 0; // identifier of next batch
int wlc_beat = 10; // maximal period covered by a single log file in seconds
@@ -201,8 +216,8 @@ WLCreadConfig(FILE *fd)
goto bailout;
}
}
- if( strncmp("id=", path,3) == 0)
- wlc_id = atol(path+ 3);
+ if( strncmp("tag=", path,4) == 0)
+ wlc_tag = atol(path+ 4);
if( strncmp("write=", path,6) == 0) {
len = snprintf(wlc_write, 26, "%s", path + 6);
if (len == -1 || len >= 26) {
@@ -251,7 +266,7 @@ str WLCsetConfig(void){
if( wlc_snapshot[0] )
mnstr_printf(fd,"snapshot=%s\n", wlc_snapshot);
mnstr_printf(fd,"logs=%s\n", wlc_dir);
- mnstr_printf(fd,"id="LLFMT"\n", wlc_id );
+ mnstr_printf(fd,"tag="LLFMT"\n", wlc_tag );
mnstr_printf(fd,"write=%s\n", wlc_write );
mnstr_printf(fd,"state=%d\n", wlc_state );
mnstr_printf(fd,"batches=%d\n", wlc_batches );
@@ -303,6 +318,21 @@ WLCcloselogger(void)
return WLCsetConfig();
}
+/* force the current log file to its storage container, but dont create a new one \
yet */ +str
+WLCflush(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+{
+ (void) cntxt;
+ (void) mb;
+ (void) stk;
+ (void) pci;
+ if( wlc_fd == NULL)
+ return MAL_SUCCEED;
+ mnstr_flush(wlc_fd);
+ mnstr_fsync(wlc_fd);
+ return WLCsetConfig();
+}
+
void
WLCreset(void)
{
@@ -392,7 +422,7 @@ WLCinitCmd(Client cntxt, MalBlkPtr mb, M
}
str
-WLCgetmasterclock(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+WLCgetclock(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{ str *ret = getArgReference_str(stk,pci,0);
(void) cntxt;
(void) mb;
@@ -401,16 +431,16 @@ WLCgetmasterclock(Client cntxt, MalBlkPt
else
*ret = GDKstrdup(str_nil);
if(*ret == NULL)
- throw(MAL,"wlc.getmasterclock", MAL_MALLOC_FAIL);
+ throw(MAL,"wlc.getclock", MAL_MALLOC_FAIL);
return MAL_SUCCEED;
}
str
-WLCgetmastertick(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+WLCgettick(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{ lng *ret = getArgReference_lng(stk,pci,0);
(void) cntxt;
(void) mb;
- *ret = wlc_id;
+ *ret = wlc_tag;
return MAL_SUCCEED;
}
@@ -418,16 +448,19 @@ WLCgetmastertick(Client cntxt, MalBlkPtr
* It forces a new log file
*/
str
-WLCsetmasterbeat(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
-{
+WLCsetbeat(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+{ int beat;
(void) mb;
(void) cntxt;
- wlc_beat = * getArgReference_int(stk,pci,1);
+ beat = * getArgReference_int(stk,pci,1);
+ if ( beat < 0)
+ throw(MAL, "wlc.setbeat", "beat should be a positive number");
+ wlc_beat = beat;
return WLCcloselogger();
}
str
-WLCgetmasterbeat(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+WLCgetbeat(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{ int *ret = getArgReference_int(stk,pci,0);
(void) mb;
(void) cntxt;
@@ -474,20 +507,20 @@ WLCmaster(Client cntxt, MalBlkPtr mb, Ma
}
str
-WLCstopmaster(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
+WLCstop(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr pci)
{
(void) cntxt;
(void) mb;
(void) stk;
(void) pci;
if( wlc_state != WLC_RUN )
- throw(MAL,"master","WARNING: master role not active");
+ throw(MAL,"wlc.stop","WARNING: master role not active");
wlc_state = WLC_STOP;
return WLCsetConfig();
}
static str
-WLCsettime(Client cntxt, InstrPtr pci, InstrPtr p, str call)
+WLCsettime(Client cntxt, InstrPtr pci, InstrPtr p, str fcn)
{
struct timeval clock;
time_t clk ;
@@ -496,7 +529,7 @@ WLCsettime(Client cntxt, InstrPtr pci, I
(void) pci;
if(gettimeofday(&clock,NULL) == -1)
- throw(MAL,call,"Unable to retrieve current time");
+ throw(MAL, fcn, "Unable to retrieve current time");
clk = clock.tv_sec;
#ifdef HAVE_LOCALTIME_R
(void) localtime_r(&clk, &ctm);
@@ -505,24 +538,106 @@ WLCsettime(Client cntxt, InstrPtr pci, I
#endif
strftime(wlc_time, sizeof(wlc_time), "%Y-%m-%dT%H:%M:%S.000",&ctm);
if (pushStr(cntxt->wlc, p, wlc_time) == NULL)
- throw(MAL, call, MAL_MALLOC_FAIL);
+ throw(MAL, fcn, MAL_MALLOC_FAIL);
return MAL_SUCCEED;
}
-#define WLCstart(P, K, MSG, CALL)\
-{\
- if( cntxt->wlc == NULL){\
- cntxt->wlc_kind = K;\
- if((cntxt->wlc = newMalBlk(STMT_INCREMENT)) == NULL) \
- throw(MAL,CALL, MAL_MALLOC_FAIL); \
- }\
- if( cntxt->wlc->stop == 0){\
_______________________________________________
checkin-list mailing list
checkin-list@monetdb.org
https://www.monetdb.org/mailman/listinfo/checkin-list
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic