[prev in list] [next in list] [prev in thread] [next in thread]
List: mdbtools-dev
Subject: Re: [mdb-dev] mdb-export and embedded \t,\r,\n and quotes
From: "Kyle R. Burton" <mortis () voicenet ! com>
Date: 2003-08-07 0:45:43
[Download RAW message or body]
Whoops! I forgot to attach the patch files. Please excuse the earlier
email.
They should be attached to this one.
Kyle
On Wed, Aug 06, 2003 at 05:40:22PM -0400, Kyle R. Burton wrote:
> Hello again.
>
> In using mdb-export, we've run into a few snags with data that contains
> embedded newlines, carriage returns, tabs and quotes. There are
> options for mdb-export to supress quoting, and to specify an alternate
> delimiter. Unfortunately these options weren't enough to handle the
> data we were trying to dump from our MDB files.
>
> I modified mdb-export and added a few new options:
>
> -q <string> specify a column quoting string (defaults to ")
> -e <string> specify an escape string that will be substituted
> for a double quote in data (defaults to a pair of
> double quotes)
> -d <delimiter> specify a column delimiter (default is a comma)
> -R <eol> specify a record delimiter (default is a single newline)
>
> I also made some changes to the behavior of mdb-export based on
> these options. The changes preserve the original behavior of
> mdb-export with the default values for the new options.
>
> - the code now looks for quote_string instead of a hard-coded double
> quote - emitting escape_string in place of quote_string. This
> means using strstr() instead of single character comparisons.
> - the header row is now quoted unless -Q is specified. We were seeing
> column names with all kinds of special characters in them - commas,
> spaces, etc.
> - escape_string (defaults to ", overrideable via a command line switch)
> is emitted in place of any quote_string values that column data
> contains. It is not emitted before the quote_string, it is emitted
> instead of the quote_string, so a double quote can be replaced
> entirely by another string.
>
> For our data processing, we composed a more complex command line:
>
> [mortis@magenta]$ mdb-export -q "'" -e """ -R "
> ***RECORD SEPARATOR***
> " -d " ||delimiter|| " ~/CREDITS_IMPORT.mdb ALL_CREDS |& less
>
> The record seperator we specified has embedded newlines in it:
>
> "\n***RECORD SEPARATOR***\n"
>
> This way in the Perl code that we're using to wrap the output of mdb-export,
> we can set the input record seperator ($/) to that record delimiter. Doing
> that makes delimiting the records very easy. The pre-existing escaping
> features, combined with the ability to specify the quote character and
> the software looking for that character makes parsing the fields very easy
> as well. Even in the presence of all the embedded meta characters.
>
>
> I've attached two patches, mdbtools-combined.patch includes the changes
> from the perspective of 0.5rc2 for the whole archive, including the
> patch sent by David Mansfield <mdbtools@dm.cobite.com>.
>
> The second patch, mdb-export.patch2, is just my changes to mdb-export
> (assuming David Mansfield's patch as a baseline).
>
>
>
>
> Thanks,
> Kyle
>
>
>
> --
>
> ------------------------------------------------------------------------------
> Wisdom and Compassion are inseparable.
> -- Christmas Humphreys
> mortis@voicenet.com http://www.voicenet.com/~mortis
> ------------------------------------------------------------------------------
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
> _______________________________________________
> mdbtools-dev mailing list
> mdbtools-dev@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mdbtools-dev
--
------------------------------------------------------------------------------
Wisdom and Compassion are inseparable.
-- Christmas Humphreys
mortis@voicenet.com http://www.voicenet.com/~mortis
------------------------------------------------------------------------------
["mdbtools-combined.patch" (text/plain)]
diff -rup mdbtools-0.5rc2/include/mdbtools.h \
mdbtools-0.5rc2-hms-1.1/include/mdbtools.h
--- mdbtools-0.5rc2/include/mdbtools.h 2003-01-12 17:59:42.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/include/mdbtools.h 2003-08-06 15:25:05.000000000 -0400
@@ -327,13 +327,14 @@ extern int mdb_fetch_row(MdbTableDef *ta
extern int mdb_is_fixed_col(MdbColumn *col);
extern char *mdb_col_to_string(MdbHandle *mdb, int start, int datatype, int size);
extern int mdb_find_end_of_row(MdbHandle *mdb, int row);
-
+extern void mdb_set_date_fmt(const char *);
/* dump.c */
extern void buffer_dump(const unsigned char* buf, int start, int end);
/* backend.c */
extern char *mdb_get_coltype_string(MdbBackend *backend, int col_type);
+int mdb_coltype_takes_length(MdbBackend *backend, int col_type);
extern void mdb_init_backends();
extern void mdb_register_backend(MdbBackend *backend, char *backend_name);
extern int mdb_set_default_backend(MdbHandle *mdb, char *backend_name);
Only in mdbtools-0.5rc2-hms-1.1: mdbtools-combined-hms.patch
diff -rup mdbtools-0.5rc2/src/libmdb/backend.c \
mdbtools-0.5rc2-hms-1.1/src/libmdb/backend.c
--- mdbtools-0.5rc2/src/libmdb/backend.c 2002-12-10 18:35:25.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/backend.c 2003-08-06 15:25:05.000000000 -0400
@@ -53,7 +53,7 @@ char *mdb_oracle_types[] =
"NUMBER",
"FLOAT",
"FLOAT",
- "DATE",
+ "-DATE",
"Oracle_Unknown 0x09",
"VARCHAR2",
"BLOB",
@@ -112,15 +112,28 @@ int did_first;
char *mdb_get_coltype_string(MdbBackend *backend, int col_type)
{
-char buf[100];
+static char buf[100];
if (col_type > 0x10) {
// return NULL;
sprintf(buf,"type %04x", col_type);
return buf;
} else {
- return backend->types_table[col_type];
+ char *str = backend->types_table[col_type];
+ if (*str == '-')
+ str++;
+ return str;
}
}
+
+int mdb_coltype_takes_length(MdbBackend *backend, int col_type)
+{
+ if (col_type > 0x10) {
+ return 1;
+ } else {
+ return (*backend->types_table[col_type] != '-');
+ }
+}
+
/*
** mdb_init_backends() initializes the mdb_backends hash and loads the builtin
** backends
diff -rup mdbtools-0.5rc2/src/libmdb/data.c mdbtools-0.5rc2-hms-1.1/src/libmdb/data.c
--- mdbtools-0.5rc2/src/libmdb/data.c 2003-01-12 17:59:43.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/data.c 2003-08-06 15:25:05.000000000 -0400
@@ -22,12 +22,20 @@
#include "math.h"
#define MDB_DEBUG_OLE 1
+#define OFFSET_MASK 0x1fff
char *mdb_money_to_string(MdbHandle *mdb, int start, char *s);
static int _mdb_attempt_bind(MdbHandle *mdb,
MdbColumn *col, unsigned char isnull, int offset, int len);
char *mdb_num_to_string(MdbHandle *mdb, int start, int datatype, int prec, int \
scale);
+static char date_fmt[64] = "%x %X";
+
+void mdb_set_date_fmt(const char * fmt)
+{
+ date_fmt[63] = 0;
+ strncpy(date_fmt, fmt, 63);
+}
void mdb_bind_column(MdbTableDef *table, int col_num, void *bind_ptr)
{
@@ -52,29 +60,30 @@ mdb_find_end_of_row(MdbHandle *mdb, int
MdbFormatConstants *fmt = mdb->fmt;
int row_end;
- /* Search the previous "row start" values for the first non-deleted one.
+ /* Search the previous "row start" values for the first non-'lookupflag' one.
* If we don't find one, then the end of the page is the correct value.
*/
#if 1
if (row==0) {
row_end = fmt->pg_size - 1;
} else {
- row_end = (mdb_get_int16(mdb, ((fmt->row_count_offset + 2) + (row - 1) * 2)) & \
0x0FFF) - 1; + row_end = (mdb_get_int16(mdb, ((fmt->row_count_offset + 2) + (row - \
1) * 2)) & OFFSET_MASK) - 1; }
return row_end;
#else
+ int i, row_start;
for (i = row - 1; i >= 0; i--) {
row_start = mdb_get_int16(mdb, ((fmt->row_count_offset + 2) + i * \
2)); + /* if lookupflag is not set, it's good (deleteflag is ok) */
if (!(row_start & 0x8000)) {
break;
}
}
- row_start &= 0x0FFF;
if (i == -1) {
row_end = fmt->pg_size - 1;
} else {
- row_end = row_start - 1;
+ row_end = (row_start & OFFSET_MASK) - 1;
}
return row_end;
#endif
@@ -177,7 +186,7 @@ unsigned char isnull;
delflag = lookupflag = 0;
if (row_start & 0x8000) lookupflag++;
if (row_start & 0x4000) delflag++;
- row_start &= 0x0FFF; /* remove flags */
+ row_start &= OFFSET_MASK; /* remove flags */
#if MDB_DEBUG
fprintf(stdout,"Row %d bytes %d to %d %s %s\n",
row, row_start, row_end,
@@ -253,7 +262,7 @@ unsigned char isnull;
}
/* if fixed columns add up to more than 256, we need a jump */
- if (col_start >= 256) {
+ if (IS_JET3(mdb) && col_start >= 256) {
num_of_jumps++;
jumps_used++;
row_start = row_start + col_start - (col_start % 256);
@@ -394,24 +403,42 @@ mdb_read_next_dpg_by_map1(MdbTableDef *t
{
MdbCatalogEntry *entry = table->entry;
MdbHandle *mdb = entry->mdb;
-guint32 pgnum, i, j, bitn, map_pg;
+guint32 pgnum, i, j, bitn, map_pg, map_ind, offset, bit_offset;
unsigned char map_byte;
- pgnum = 0;
+ /*
+ * table->cur_phys_pg will tell us where to (re)start the scan
+ * for the next data page. each usage_map entry points to a
+ * 0x05 page which bitmaps (mdb->fmt->pg_size - 4) * 8 pages.
+ *
+ * map_ind gives us the starting usage_map entry
+ * offset gives us a page offset into the bitmap (must be converted
+ * to bytes and bits).
+ */
+ pgnum = table->cur_phys_pg + 1;
+ map_ind = pgnum / ((mdb->fmt->pg_size - 4) * 8);
+ offset = pgnum % ((mdb->fmt->pg_size - 4) * 8);
+ bit_offset = offset % 8;
+
//printf("map size %ld\n", table->map_sz);
- for (i=1;i<table->map_sz-1;i+=4) {
+ for (i = 4 * map_ind + 1; i<table->map_sz-1;i+=4) {
map_pg = _mdb_get_int32(table->usage_map, i);
//printf("loop %d pg %ld %02x%02x%02x%02x\n",i, \
map_pg,table->usage_map[i],table->usage_map[i+1],table->usage_map[i+2],table->usage_map[i+3]);
- if (!map_pg) continue;
+ /* if the usage_map entry is empty, skip it, but advance the page counter */
+ if (!map_pg) {pgnum += (mdb->fmt->pg_size - 4) * 8; continue;}
if(mdb_read_alt_pg(mdb, map_pg) != mdb->fmt->pg_size) {
fprintf(stderr, "Oops! didn't get a full page at %d\n", map_pg);
exit(1);
}
//printf("reading page %ld\n",map_pg);
- for (j=4;j<mdb->fmt->pg_size;j++) {
- for (bitn=0;bitn<8;bitn++) {
+
+ /* first time through, start j, bitn based on the starting offset
+ * after that, reset offset to 0 to start at the beginning of the page/byte
+ */
+ for (j=4 + offset / 8;j<mdb->fmt->pg_size;j++) {
+ for (bitn=bit_offset;bitn<8;bitn++) {
if (mdb->alt_pg_buf[j] & 1 << bitn && pgnum > table->cur_phys_pg) {
table->cur_phys_pg = pgnum;
if (!mdb_read_pg(mdb, pgnum)) {
@@ -423,7 +450,10 @@ unsigned char map_byte;
}
pgnum++;
}
+ bit_offset = 0;
}
+
+ offset = 0;
}
/* didn't find anything */
//printf("returning 0\n");
@@ -476,7 +506,8 @@ int rc;
if (!table->cur_pg_num) {
table->cur_pg_num=1;
table->cur_row=0;
- mdb_read_next_dpg(table);
+ if (!mdb_read_next_dpg(table))
+ return 0;
}
do {
@@ -784,13 +815,39 @@ gint32 l, whole, fraction;
*/
return text;
}
+
+static int trim_trailing_zeros(char * buff, int n)
+{
+ char * p = buff + n - 1;
+ while (p >= buff && *p == '0')
+ *p-- = '\0';
+ if (*p == '.')
+ *p = '\0';
+}
+
+
+/*
+ * how much precision should we ask for from printf and friends?
+ * pass FLT_DIG for floats, DBL_DIG for doubles
+ */
+int max_dec_places(double f, int dig_prec)
+{
+ int ret = dig_prec - (int)ceil(log10(fabs(f)));
+ /* always at least one digit of precision so the trim function works */
+ if (ret < 1)
+ ret = 1;
+ return ret;
+}
+
char *mdb_col_to_string(MdbHandle *mdb, int start, int datatype, int size)
{
/* FIX ME -- not thread safe */
static char text[MDB_BIND_SIZE];
char tmpbuf[10];
time_t t;
-int i,j;
+int i,j,n;
+float tf;
+double td;
switch (datatype) {
case MDB_BOOL:
@@ -810,11 +867,15 @@ int i,j;
return text;
break;
case MDB_FLOAT:
- sprintf(text,"%f",mdb_get_single(mdb, start));
+ tf = mdb_get_single(mdb, start);
+ n = sprintf(text,"%.*f", max_dec_places(tf, FLT_DIG), tf);
+ trim_trailing_zeros(text, n);
return text;
break;
case MDB_DOUBLE:
- sprintf(text,"%f",mdb_get_double(mdb, start));
+ td = mdb_get_double(mdb, start);
+ n = sprintf(text,"%.*f", max_dec_places(td, DBL_DIG), td);
+ trim_trailing_zeros(text, n);
return text;
break;
case MDB_TEXT:
@@ -846,7 +907,7 @@ int i,j;
break;
case MDB_SDATETIME:
t = (long int)((mdb_get_double(mdb, start) - 25569.0) * 86400.0);
- strftime(text, MDB_BIND_SIZE, "%x %X",
+ strftime(text, MDB_BIND_SIZE, date_fmt,
(struct tm*)gmtime(&t));
return text;
diff -rup mdbtools-0.5rc2/src/libmdb/file.c mdbtools-0.5rc2-hms-1.1/src/libmdb/file.c
--- mdbtools-0.5rc2/src/libmdb/file.c 2003-01-12 17:59:43.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/file.c 2003-08-06 15:25:05.000000000 -0400
@@ -26,6 +26,70 @@ MdbFormatConstants MdbJet3Constants = {
2048, 0x08, 12, 25, 27, 31, 35, 36, 43, 8, 13, 16, 1, 18
};
+typedef struct _RC4_KEY
+{
+ unsigned char state[256];
+ unsigned char x;
+ unsigned char y;
+} RC4_KEY;
+
+#define swap_byte(x,y) t = *(x); *(x) = *(y); *(y) = t
+
+void RC4_set_key(RC4_KEY *key, int key_data_len, unsigned char *key_data_ptr)
+{
+ int i;
+ unsigned char t;
+ unsigned char swapByte;
+ unsigned char index1;
+ unsigned char index2;
+ unsigned char* state;
+ short counter;
+
+ state = &key->state[0];
+ for(counter = 0; counter < 256; counter++)
+ state[counter] = counter;
+ key->x = 0;
+ key->y = 0;
+ index1 = 0;
+ index2 = 0;
+ for(counter = 0; counter < 256; counter++)
+ {
+ index2 = (key_data_ptr[index1] + state[counter] + index2) % 256;
+ swap_byte(&state[counter], &state[index2]);
+ index1 = (index1 + 1) % key_data_len;
+ }
+}
+
+/*
+ * this algorithm does 'encrypt in place' instead of inbuff/outbuff
+ * note also: encryption and decryption use same routine
+ * implementation supplied by (Adam Back) at <adam at cypherspace dot org>
+ */
+
+void RC4(RC4_KEY *key, int buffer_len, unsigned char * buff)
+{
+ unsigned char t;
+ unsigned char x;
+ unsigned char y;
+ unsigned char* state;
+ unsigned char xorIndex;
+ short counter;
+
+ x = key->x;
+ y = key->y;
+ state = &key->state[0];
+ for(counter = 0; counter < buffer_len; counter++)
+ {
+ x = (x + 1) % 256;
+ y = (state[x] + y) % 256;
+ swap_byte(&state[x], &state[y]);
+ xorIndex = (state[x] + state[y]) % 256;
+ buff[counter] ^= state[xorIndex];
+ }
+ key->x = x;
+ key->y = y;
+}
+
static size_t _mdb_read_pg(MdbHandle *mdb, unsigned char *pg_buf, unsigned long pg);
static int mdb_find_file(char *file_name, char *file_path, int bufsize)
{
@@ -118,8 +182,13 @@ MdbFile *f;
/* get the db encryption key and xor it back to clear text */
f->db_key = mdb_get_int32(mdb, 0x3e);
- f->db_key ^= 0xe15e01b9;
-
+
+ /* I don't know if this value is valid for some versions?
+ * it doesn't seem to be valid for the databases I have
+ *
+ * f->db_key ^= 0xe15e01b9;
+ */
+ f->db_key ^= 0x4ebc8afb;
/* get the db password located at 0x42 bytes into the file */
for (pos=0;pos<14;pos++) {
@@ -209,6 +278,19 @@ off_t offset = pg * mdb->fmt->pg_size;
/* fprintf(stderr,"EOF reached %d bytes returned.\n",len, mdb->fmt->pg_size); */
return 0;
}
+
+ /*
+ * unencrypt the page if necessary.
+ * it might make sense to cache the unencrypted data blocks?
+ */
+ if (pg != 0 && mdb->f->db_key != 0)
+ {
+ RC4_KEY rc4_key;
+ unsigned int tmp_key = mdb->f->db_key ^ pg;
+ RC4_set_key(&rc4_key, 4, (unsigned char *)&tmp_key);
+ RC4(&rc4_key, mdb->fmt->pg_size, pg_buf);
+ }
+
return len;
}
void mdb_swap_pgbuf(MdbHandle *mdb)
@@ -332,7 +414,7 @@ mdb_get_double(MdbHandle *mdb, int offse
#endif
double d;
- if (offset <0 || offset+4 > mdb->fmt->pg_size) return -1;
+ if (offset <0 || offset+8 > mdb->fmt->pg_size) return -1;
memcpy(&d, &mdb->pg_buf[offset], 8);
diff -rup mdbtools-0.5rc2/src/libmdb/Makefile.am \
mdbtools-0.5rc2-hms-1.1/src/libmdb/Makefile.am
--- mdbtools-0.5rc2/src/libmdb/Makefile.am 2003-01-05 09:58:33.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/Makefile.am 2003-08-06 15:25:05.000000000 \
-0400 @@ -1,4 +1,5 @@
lib_LTLIBRARIES = libmdb.la
libmdb_la_SOURCES= catalog.c mem.c file.c kkd.c table.c data.c dump.c backend.c \
money.c sargs.c index.c like.c write.c stats.c INCLUDES = -I$(top_srcdir)/include \
$(GLIB_CFLAGS)
-LIBS = $(GLIB_LIBS)
+LIBS = $(GLIB_LIBS) -lm
+
diff -rup mdbtools-0.5rc2/src/libmdb/Makefile.in \
mdbtools-0.5rc2-hms-1.1/src/libmdb/Makefile.in
--- mdbtools-0.5rc2/src/libmdb/Makefile.in 2003-01-15 11:39:13.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/Makefile.in 2003-08-06 15:25:05.000000000 \
-0400 @@ -104,7 +104,7 @@ install_sh = @install_sh@
lib_LTLIBRARIES = libmdb.la
libmdb_la_SOURCES = catalog.c mem.c file.c kkd.c table.c data.c dump.c backend.c \
money.c sargs.c index.c like.c write.c stats.c INCLUDES = -I$(top_srcdir)/include \
$(GLIB_CFLAGS)
-LIBS = $(GLIB_LIBS)
+LIBS = $(GLIB_LIBS) -lm
subdir = src/libmdb
mkinstalldirs = $(SHELL) $(top_srcdir)/mkinstalldirs
CONFIG_CLEAN_FILES =
diff -rup mdbtools-0.5rc2/src/libmdb/mem.c mdbtools-0.5rc2-hms-1.1/src/libmdb/mem.c
--- mdbtools-0.5rc2/src/libmdb/mem.c 2003-01-12 17:59:43.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/mem.c 2003-08-06 15:25:05.000000000 -0400
@@ -18,7 +18,7 @@
*/
#include "mdbtools.h"
-
+#include <locale.h>
void mdb_init()
{
mdb_init_backends();
diff -rup mdbtools-0.5rc2/src/libmdb/table.c \
mdbtools-0.5rc2-hms-1.1/src/libmdb/table.c
--- mdbtools-0.5rc2/src/libmdb/table.c 2003-01-12 17:59:43.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/libmdb/table.c 2003-08-06 15:25:05.000000000 -0400
@@ -123,23 +123,29 @@ GSList *slist = NULL;
buffer_dump(mdb->pg_buf, cur_col ,cur_col + 18); */
#endif
memset(&col, 0, sizeof(col));
- col.col_num = mdb->pg_buf[cur_col + fmt->col_num_offset];
read_pg_if(mdb, &cur_col, 0);
col.col_type = mdb->pg_buf[cur_col];
+ read_pg_if(mdb, &cur_col, fmt->col_num_offset); // col_num_offset == 1 or 5
+ col.col_num = mdb->pg_buf[cur_col + fmt->col_num_offset];
+
+ /* FIXME: can this be right in Jet3 and Jet4? */
if (col.col_type == MDB_NUMERIC) {
+ read_pg_if(mdb, &cur_col, 11);
col.col_prec = mdb->pg_buf[cur_col + 11];
+
+ read_pg_if(mdb, &cur_col, 12);
col.col_scale = mdb->pg_buf[cur_col + 12];
}
- read_pg_if(mdb, &cur_col, 13);
- col.is_fixed = mdb->pg_buf[cur_col + fmt->col_fixed_offset] &
+ read_pg_if(mdb, &cur_col, fmt->col_fixed_offset);// col_fixed_offset == 13 or 15
+ col.is_fixed = mdb->pg_buf[cur_col + fmt->col_fixed_offset] &
0x01 ? 1 : 0;
if (col.col_type != MDB_BOOL) {
- read_pg_if(mdb, &cur_col, 17);
- low_byte = mdb->pg_buf[cur_col + fmt->col_size_offset];
- read_pg_if(mdb, &cur_col, 18);
+ read_pg_if(mdb, &cur_col, fmt->col_size_offset); //col_size_offset == 16 or 23
+ low_byte = mdb->pg_buf[cur_col + fmt->col_size_offset];
+ read_pg_if(mdb, &cur_col, fmt->col_size_offset + 1);
high_byte = mdb->pg_buf[cur_col + fmt->col_size_offset + 1];
col.col_size += high_byte * 256 + low_byte;
} else
@@ -179,14 +185,19 @@ GSList *slist = NULL;
if (len < name_sz) {
/* read the next pg */
mdb_read_pg(mdb, mdb_get_int32(mdb,4));
+ /* adjust cur_name to the new page - it will actually
+ * be positioned before the start of valid data, but
+ * since len was not reset, we should start exactly in
+ * the right place. be careful of odd/even bytes (unicode...)
+ */
cur_name = 8 - (fmt->pg_size - cur_name);
- if (len % 2) cur_name++;
/* get the rest of the name */
- for (j=0;j<len;j+=2) {
+ for (j=len;j<name_sz;j++) {
+ if (!(j%2))
+ pcol->name[j/2] = mdb->pg_buf[cur_name + j];
}
- memcpy(&pcol->name[len], &mdb->pg_buf[cur_name], name_sz - len);
}
- pcol->name[name_sz]='\0';
+ pcol->name[name_sz/2]='\0';
cur_name += name_sz;
} else if (IS_JET3(mdb)) {
Only in mdbtools-0.5rc2-hms-1.1/src/sql: lexer.c
diff -rup mdbtools-0.5rc2/src/sql/lexer.l mdbtools-0.5rc2-hms-1.1/src/sql/lexer.l
--- mdbtools-0.5rc2/src/sql/lexer.l 2002-12-27 10:09:02.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/sql/lexer.l 2003-07-31 09:36:11.000000000 -0400
@@ -24,56 +24,62 @@
%}
%%
-select { return SELECT; }
-from { return FROM; }
-connect { return CONNECT; }
-disconnect { return DISCONNECT; }
-to { return TO; }
-list { return LIST; }
-where { return WHERE; }
-and { return AND; }
-tables { return TABLES; }
-table { return TABLE; }
-describe { return DESCRIBE; }
-(<=) { return LTEQ; }
-(>=) { return GTEQ; }
-like { return LIKE; }
-[ \t\r] ;
-\"[A-z][A-z0-9 ]*\" {
- yylval.name = strdup(&yytext[1]);
- yylval.name[strlen(yylval.name)-1]='\0';
- return IDENT;
- }
-[A-z][A-z0-9]* { yylval.name = strdup(yytext); return NAME; }
-
-'.*' { yylval.name = strdup(yytext); return STRING; }
-([0-9]+|([0-9]*\.[0-9+)([eE][-+]?[0-9]+)?) {
- yylval.name = strdup(yytext); return NUMBER;
- }
-~?(\/?[A-z0-9\.]+)+ { yylval.name = strdup(yytext); return PATH; }
-. { return yytext[0]; }
+[sS][eE][lL][eE][cC][tT] { return SELECT; }
+[fF][rR][oO][mM] { return FROM; }
+[cC][oO][nN][nN][eE][cC][tT] { return CONNECT; }
+[dD][iI][sS][cC][oO][nN][nN][eE][cC][tT] { return DISCONNECT; }
+[tT][oO] { return TO; }
+[lL][iI][sS][tT] { return LIST; }
+[wW][hH][eE][rR][eE] { return WHERE; }
+[aA][nN][dD] { return AND; }
+[tT][aA][bB][lL][eE][sS] { return TABLES; }
+[tT][aA][bB][lL][eE] { return TABLE; }
+[dD][eE][sS][cC][rR][iI][bB][eE] { return DESCRIBE; }
+[lL][iI][kK][eE] { return LIKE; }
+
+(<=) { return LTEQ; }
+(>=) { return GTEQ; }
+[ \t\r] ;
+
+\"[[:alpha:]][[:alnum:] _]*\" {
+ yylval.name = strdup(&yytext[1]);
+ yylval.name[strlen(yylval.name)-1]='\0';
+ return IDENT;
+ }
+[[:alpha:]][[:alnum:]_]* { yylval.name = strdup(yytext); return NAME; }
+
+'.*' { yylval.name = strdup(yytext); return STRING; }
+
+([[:digit:]]+|([[:digit:]]*\.[[:digit]]+)([eE][-+]?[[:digit:]]+)?) {
+ yylval.name = strdup(yytext); return NUMBER;
+ }
+
+~?(\/?[[:alnum:]]\.]+)+ { yylval.name = strdup(yytext); return PATH; }
+
+. { return yytext[0]; }
+
%%
int yywrap()
{
- return 1;
+ return 1;
}
void yyerror(char *s)
{
- fprintf(stderr,"Error at Line : %s near %s\n", s, yytext);
+ fprintf(stderr,"Error at Line : %s near %s\n", s, yytext);
}
#if 0
int main(int argc, char **argv)
{
int i;
- g_sql = mdb_sql_init();
- yyin = stdin;
- if (yyparse()) {
- fprintf(stderr, "Couldn't parse SQL\n");
- exit(1);
- }
- mdb_sql_dump(g_sql);
- mdb_sql_exit(g_sql);
+ g_sql = mdb_sql_init();
+ yyin = stdin;
+ if (yyparse()) {
+ fprintf(stderr, "Couldn't parse SQL\n");
+ exit(1);
+ }
+ mdb_sql_dump(g_sql);
+ mdb_sql_exit(g_sql);
}
#endif
Only in mdbtools-0.5rc2-hms-1.1/src/sql: parser.c
Only in mdbtools-0.5rc2-hms-1.1/src/sql: parser.h
diff -rup mdbtools-0.5rc2/src/util/mdb-export.c \
mdbtools-0.5rc2-hms-1.1/src/util/mdb-export.c
--- mdbtools-0.5rc2/src/util/mdb-export.c 2002-12-10 18:35:25.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/util/mdb-export.c 2003-08-06 15:20:15.000000000 -0400
@@ -32,26 +32,42 @@ MdbTableDef *table;
MdbColumn *col;
/* doesn't handle tables > 256 columns. Can that happen? */
char *bound_values[256];
-char *delimiter = ",";
+char *delimiter = strdup(",");
+char *record_delimiter = strdup("\n");
+char *escape_string = strdup("\"\"");
+char *quote_string = strdup("\"");
+char *mdb_file_name = NULL;
+char *mdb_table_name = NULL;
char header_row = 1;
char quote_text = 1;
int opt;
char *s;
- while ((opt=getopt(argc, argv, "HQd:"))!=-1) {
+ while ((opt=getopt(argc, argv, "HQd:D:R:e:q:"))!=-1) {
switch (opt) {
case 'H':
- header_row = 0;
- break;
+ header_row = 0;
+ break;
case 'Q':
- quote_text = 0;
- break;
+ quote_text = 0;
+ break;
+ case 'q':
+ quote_string = strdup(optarg);
+ break;
case 'd':
- delimiter = (char *) malloc(strlen(optarg)+1);
- strcpy(delimiter, optarg);
- break;
+ delimiter = strdup( optarg);
+ break;
+ case 'D':
+ mdb_set_date_fmt(optarg);
+ break;
+ case 'R':
+ record_delimiter = strdup( optarg );
+ break;
+ case 'e':
+ escape_string = strdup( optarg );
+ break;
default:
- break;
+ break;
}
}
@@ -59,19 +75,35 @@ char *s;
** optind is now the position of the first non-option arg,
** see getopt(3)
*/
+
if (argc-optind < 2) {
+ if( argc-optind == 0 ) {
+ fprintf(stderr,"Error, no MDB file specified.\n");
+ }
+ if( argc - optind == 1 ) {
+ fprintf(stderr,"Error, no table name specified.\n");
+ }
fprintf(stderr,"Usage: %s [options] <file> <table>\n",argv[0]);
fprintf(stderr,"where options are:\n");
fprintf(stderr," -H supress header row\n");
fprintf(stderr," -Q don't wrap text-like fields in quotes\n");
- fprintf(stderr," -d <delimiter> specify a column delimiter\n");
- exit(1);
+ fprintf(stderr," -q <string> specify a column quoting string (defaults to \
\")\n"); + fprintf(stderr," -e <string> specify an escape string that will be \
substituted\n"); + fprintf(stderr," for a double quote \
in data (defaults to a pair of\n"); + fprintf(stderr," \
double quotes)\n"); + fprintf(stderr," -d <delimiter> specify a column delimiter \
(default is a comma)\n"); + fprintf(stderr," -R <eol> specify a record \
delimiter (default is a single newline)\n"); + exit(1);
}
+ mdb_file_name = argv[optind];
+ mdb_table_name = argv[optind+1];
+
mdb_init();
- if (!(mdb = mdb_open(argv[optind]))) {
- exit(1);
+ if (!(mdb = mdb_open(mdb_file_name))) {
+ fprintf(stderr,"Error opening MDB file: %s table: \
%s\n",argv[optind],argv[optind+1]); + exit(1);
}
mdb_read_catalog(mdb, MDB_TABLE);
@@ -79,60 +111,74 @@ char *s;
for (i=0;i<mdb->num_catalog;i++) {
entry = g_ptr_array_index(mdb->catalog,i);
if (entry->object_type == MDB_TABLE &&
- !strcmp(entry->object_name,argv[argc-1])) {
+ !strcmp(entry->object_name,mdb_table_name)) {
table = mdb_read_table(entry);
mdb_read_columns(table);
mdb_rewind_table(table);
for (j=0;j<table->num_cols;j++) {
- bound_values[j] = (char *) malloc(MDB_BIND_SIZE);
- bound_values[j][0] = '\0';
- mdb_bind_column(table, j+1, bound_values[j]);
+ bound_values[j] = (char *) malloc(MDB_BIND_SIZE);
+ bound_values[j][0] = '\0';
+ mdb_bind_column(table, j+1, bound_values[j]);
}
if (header_row) {
- col=g_ptr_array_index(table->columns,0);
- fprintf(stdout,"%s",col->name);
- for (j=1;j<table->num_cols;j++) {
- col=g_ptr_array_index(table->columns,j);
- fprintf(stdout,"%s%s",delimiter,col->name);
+ for (j=0;j<table->num_cols;j++) {
+ if( 0 != j ) fprintf(stdout,delimiter);
+ col=g_ptr_array_index(table->columns,j);
+ if( quote_text ) {
+ fprintf(stdout,quote_string);
+ fprintf(stdout,"%s",col->name);
+ fprintf(stdout,quote_string);
+ }
+ else fprintf(stdout,"%s",col->name);
}
- fprintf(stdout,"\n");
+ fprintf(stdout,record_delimiter);
}
while(mdb_fetch_row(table)) {
- if (quote_text && is_text_type(col->col_type)) {
- fprintf(stdout,"\"");
- for (s=bound_values[0];*s;s++) {
- if (*s=='"') fprintf(stdout,"\"\"");
- else fprintf(stdout,"%c",*s);
- }
- fprintf(stdout,"\"");
- /* fprintf(stdout,"\"%s\"",bound_values[0]); */
- } else {
- fprintf(stdout,"%s",bound_values[0]);
- }
- for (j=1;j<table->num_cols;j++) {
- col=g_ptr_array_index(table->columns,j);
- if (quote_text && is_text_type(col->col_type)) {
- fprintf(stdout,"%s",delimiter);
- fprintf(stdout,"\"");
- for (s=bound_values[j];*s;s++) {
- if (*s=='"') fprintf(stdout,"\"\"");
- else fprintf(stdout,"%c",*s);
- }
- fprintf(stdout,"\"");
- } else {
- fprintf(stdout,"%s%s",delimiter,bound_values[j]);
- }
- }
- fprintf(stdout,"\n");
+ for (j=0;j<table->num_cols;j++) {
+ col=g_ptr_array_index(table->columns,j);
+
+ /* If it's not the first field, emit the column \
delimiter */ + if( 0 != j ) fprintf(stdout,delimiter);
+
+ /* If we're supposed to quote the field... */
+ if (quote_text && is_text_type(col->col_type)) {
+ fprintf(stdout,quote_string);
+ /* Search the data for instances of the quote \
string */ + for (s=bound_values[j];*s;s++) {
+ /* If we find the quote string, emit the \
escape string, + * skip past it and finish \
emiting the value + **/
+ if ( s == strstr(s,quote_string) ) {
+ fprintf(stdout,escape_string);
+ s += strlen(quote_string);
+ }
+ /* No delimiter found, emit the character. \
*/ + else fprintf(stdout,"%c",*s);
+ }
+ /* Emit the closing quote. */
+ fprintf(stdout,quote_string);
+ } else { /* Don't quote the field */
+ fprintf(stdout,"%s",bound_values[j]);
+ }
+ }
+ /* Finish off the record. */
+ fprintf(stdout,record_delimiter);
}
+ /* Free the bound values allocated by fetch row. */
for (j=0;j<table->num_cols;j++) {
free(bound_values[j]);
}
}
}
+
+ free(record_delimiter);
+ free(escape_string);
+ free(quote_string);
+ free(delimiter);
+
mdb_free_handle(mdb);
mdb_exit();
diff -rup mdbtools-0.5rc2/src/util/mdb-schema.c \
mdbtools-0.5rc2-hms-1.1/src/util/mdb-schema.c
--- mdbtools-0.5rc2/src/util/mdb-schema.c 2002-12-30 11:38:52.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/util/mdb-schema.c 2003-08-06 15:25:05.000000000 -0400
@@ -17,9 +17,11 @@
*/
/* this utility dumps the schema for an existing database */
-
+#include <ctype.h>
#include "mdbtools.h"
+static char *sanitize_name(char *str, int sanitize);
+
main (int argc, char **argv)
{
int i, j, k;
@@ -29,6 +31,8 @@ MdbTableDef *table;
MdbColumn *col;
char *the_relation;
char *tabname = NULL;
+char *namespace = "";
+int s = 0;
int opt;
if (argc < 2) {
@@ -36,12 +40,19 @@ int opt;
exit (1);
}
- while ((opt=getopt(argc, argv, "T:"))!=-1) {
+ while ((opt=getopt(argc, argv, "T:N:S"))!=-1) {
switch (opt) {
case 'T':
tabname = (char *) malloc(strlen(optarg)+1);
strcpy(tabname, optarg);
break;
+ case 'N':
+ namespace = (char *) malloc(strlen(optarg)+1);
+ strcpy(namespace, optarg);
+ break;
+ case 'S':
+ s = 1;
+ break;
}
}
@@ -50,7 +61,7 @@ int opt;
/* open the database */
mdb = mdb_open (argv[optind]);
- if (argc - optind >2) {
+ if (argc - optind >= 2) {
if (!mdb_set_default_backend(mdb, argv[optind + 1])) {
fprintf(stderr,"Invalid backend type\n");
mdb_exit();
@@ -82,10 +93,10 @@ int opt;
if (!strcmp (mdb_get_objtype_string (entry->object_type), "Table"))
{
/* drop the table if it exists */
- fprintf (stdout, "DROP TABLE %s;\n", entry->object_name);
+ fprintf (stdout, "DROP TABLE %s%s;\n", namespace, \
sanitize_name(entry->object_name, s));
/* create the table */
- fprintf (stdout, "CREATE TABLE %s\n", entry->object_name);
+ fprintf (stdout, "CREATE TABLE %s%s\n", namespace, \
sanitize_name(entry->object_name, s)); fprintf (stdout, " (\n");
table = mdb_read_table (entry);
@@ -99,10 +110,11 @@ int opt;
{
col = g_ptr_array_index (table->columns, k);
- fprintf (stdout, "\t%s\t\t\t%s", col->name,
+ fprintf (stdout, "\t%s\t\t\t%s", sanitize_name(col->name, s),
mdb_get_coltype_string (mdb->default_backend, col->col_type));
- if (col->col_size != 0)
+ if (col->col_size != 0 &&
+ mdb_coltype_takes_length(mdb->default_backend, col->col_type))
fprintf (stdout, " (%d)", col->col_size);
if (k < table->num_cols - 1)
@@ -111,7 +123,7 @@ int opt;
fprintf (stdout, "\n");
}
- fprintf (stdout, "\n);\n");
+ fprintf (stdout, ");\n");
fprintf (stdout, "-- CREATE ANY INDEXES ...\n");
fprintf (stdout, "\n");
}
@@ -133,3 +145,23 @@ int opt;
exit(0);
}
+
+static char *sanitize_name(char *str, int sanitize)
+{
+ static char namebuf[256];
+ char *p = namebuf;
+
+ if (!sanitize)
+ return str;
+
+ while (*str)
+ {
+ *p = isalnum(*str) ? *str : '_';
+ p++;
+ str++;
+ }
+
+ *p = 0;
+
+ return namebuf;
+}
["mdb-export.patch2" (text/plain)]
--- mdbtools-0.5rc2/src/util/mdb-export.c 2002-12-10 18:35:25.000000000 -0500
+++ mdbtools-0.5rc2-hms-1.1/src/util/mdb-export.c 2003-08-06 15:20:15.000000000 -0400
@@ -32,26 +32,42 @@ MdbTableDef *table;
MdbColumn *col;
/* doesn't handle tables > 256 columns. Can that happen? */
char *bound_values[256];
-char *delimiter = ",";
+char *delimiter = strdup(",");
+char *record_delimiter = strdup("\n");
+char *escape_string = strdup("\"\"");
+char *quote_string = strdup("\"");
+char *mdb_file_name = NULL;
+char *mdb_table_name = NULL;
char header_row = 1;
char quote_text = 1;
int opt;
char *s;
- while ((opt=getopt(argc, argv, "HQd:"))!=-1) {
+ while ((opt=getopt(argc, argv, "HQd:D:R:e:q:"))!=-1) {
switch (opt) {
case 'H':
- header_row = 0;
- break;
+ header_row = 0;
+ break;
case 'Q':
- quote_text = 0;
- break;
+ quote_text = 0;
+ break;
+ case 'q':
+ quote_string = strdup(optarg);
+ break;
case 'd':
- delimiter = (char *) malloc(strlen(optarg)+1);
- strcpy(delimiter, optarg);
- break;
+ delimiter = strdup( optarg);
+ break;
+ case 'D':
+ mdb_set_date_fmt(optarg);
+ break;
+ case 'R':
+ record_delimiter = strdup( optarg );
+ break;
+ case 'e':
+ escape_string = strdup( optarg );
+ break;
default:
- break;
+ break;
}
}
@@ -59,19 +75,35 @@ char *s;
** optind is now the position of the first non-option arg,
** see getopt(3)
*/
+
if (argc-optind < 2) {
+ if( argc-optind == 0 ) {
+ fprintf(stderr,"Error, no MDB file specified.\n");
+ }
+ if( argc - optind == 1 ) {
+ fprintf(stderr,"Error, no table name specified.\n");
+ }
fprintf(stderr,"Usage: %s [options] <file> <table>\n",argv[0]);
fprintf(stderr,"where options are:\n");
fprintf(stderr," -H supress header row\n");
fprintf(stderr," -Q don't wrap text-like fields in quotes\n");
- fprintf(stderr," -d <delimiter> specify a column delimiter\n");
- exit(1);
+ fprintf(stderr," -q <string> specify a column quoting string (defaults to \")\n");
+ fprintf(stderr," -e <string> specify an escape string that will be substituted\n");
+ fprintf(stderr," for a double quote in data (defaults to a pair of\n");
+ fprintf(stderr," double quotes)\n");
+ fprintf(stderr," -d <delimiter> specify a column delimiter (default is a comma)\n");
+ fprintf(stderr," -R <eol> specify a record delimiter (default is a single newline)\n");
+ exit(1);
}
+ mdb_file_name = argv[optind];
+ mdb_table_name = argv[optind+1];
+
mdb_init();
- if (!(mdb = mdb_open(argv[optind]))) {
- exit(1);
+ if (!(mdb = mdb_open(mdb_file_name))) {
+ fprintf(stderr,"Error opening MDB file: %s table: %s\n",argv[optind],argv[optind+1]);
+ exit(1);
}
mdb_read_catalog(mdb, MDB_TABLE);
@@ -79,60 +111,74 @@ char *s;
for (i=0;i<mdb->num_catalog;i++) {
entry = g_ptr_array_index(mdb->catalog,i);
if (entry->object_type == MDB_TABLE &&
- !strcmp(entry->object_name,argv[argc-1])) {
+ !strcmp(entry->object_name,mdb_table_name)) {
table = mdb_read_table(entry);
mdb_read_columns(table);
mdb_rewind_table(table);
for (j=0;j<table->num_cols;j++) {
- bound_values[j] = (char *) malloc(MDB_BIND_SIZE);
- bound_values[j][0] = '\0';
- mdb_bind_column(table, j+1, bound_values[j]);
+ bound_values[j] = (char *) malloc(MDB_BIND_SIZE);
+ bound_values[j][0] = '\0';
+ mdb_bind_column(table, j+1, bound_values[j]);
}
if (header_row) {
- col=g_ptr_array_index(table->columns,0);
- fprintf(stdout,"%s",col->name);
- for (j=1;j<table->num_cols;j++) {
- col=g_ptr_array_index(table->columns,j);
- fprintf(stdout,"%s%s",delimiter,col->name);
+ for (j=0;j<table->num_cols;j++) {
+ if( 0 != j ) fprintf(stdout,delimiter);
+ col=g_ptr_array_index(table->columns,j);
+ if( quote_text ) {
+ fprintf(stdout,quote_string);
+ fprintf(stdout,"%s",col->name);
+ fprintf(stdout,quote_string);
+ }
+ else fprintf(stdout,"%s",col->name);
}
- fprintf(stdout,"\n");
+ fprintf(stdout,record_delimiter);
}
while(mdb_fetch_row(table)) {
- if (quote_text && is_text_type(col->col_type)) {
- fprintf(stdout,"\"");
- for (s=bound_values[0];*s;s++) {
- if (*s=='"') fprintf(stdout,"\"\"");
- else fprintf(stdout,"%c",*s);
- }
- fprintf(stdout,"\"");
- /* fprintf(stdout,"\"%s\"",bound_values[0]); */
- } else {
- fprintf(stdout,"%s",bound_values[0]);
- }
- for (j=1;j<table->num_cols;j++) {
- col=g_ptr_array_index(table->columns,j);
- if (quote_text && is_text_type(col->col_type)) {
- fprintf(stdout,"%s",delimiter);
- fprintf(stdout,"\"");
- for (s=bound_values[j];*s;s++) {
- if (*s=='"') fprintf(stdout,"\"\"");
- else fprintf(stdout,"%c",*s);
- }
- fprintf(stdout,"\"");
- } else {
- fprintf(stdout,"%s%s",delimiter,bound_values[j]);
- }
- }
- fprintf(stdout,"\n");
+ for (j=0;j<table->num_cols;j++) {
+ col=g_ptr_array_index(table->columns,j);
+
+ /* If it's not the first field, emit the column delimiter */
+ if( 0 != j ) fprintf(stdout,delimiter);
+
+ /* If we're supposed to quote the field... */
+ if (quote_text && is_text_type(col->col_type)) {
+ fprintf(stdout,quote_string);
+ /* Search the data for instances of the quote string */
+ for (s=bound_values[j];*s;s++) {
+ /* If we find the quote string, emit the escape string,
+ * skip past it and finish emiting the value
+ **/
+ if ( s == strstr(s,quote_string) ) {
+ fprintf(stdout,escape_string);
+ s += strlen(quote_string);
+ }
+ /* No delimiter found, emit the character. */
+ else fprintf(stdout,"%c",*s);
+ }
+ /* Emit the closing quote. */
+ fprintf(stdout,quote_string);
+ } else { /* Don't quote the field */
+ fprintf(stdout,"%s",bound_values[j]);
+ }
+ }
+ /* Finish off the record. */
+ fprintf(stdout,record_delimiter);
}
+ /* Free the bound values allocated by fetch row. */
for (j=0;j<table->num_cols;j++) {
free(bound_values[j]);
}
}
}
+
+ free(record_delimiter);
+ free(escape_string);
+ free(quote_string);
+ free(delimiter);
+
mdb_free_handle(mdb);
mdb_exit();
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
mdbtools-dev mailing list
mdbtools-dev@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mdbtools-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic