[prev in list] [next in list] [prev in thread] [next in thread] 

List:       slony1-general
Subject:    [Slony1-general] Re: updated slon-mkservice (and logrep-mkservice
From:       "Andrew Hammond" <andrew.george.hammond () gmail ! com>
Date:       2007-03-28 21:01:12
Message-ID: 5a0a9d6f0703281401k52bee618md8cf6b60c0f34ed8 () mail ! gmail ! com
[Download RAW message or body]

On 3/16/07, Andrew Hammond <andrew.george.hammond@gmail.com> wrote:
> On 3/9/07, Andrew Hammond <andrew.george.hammond@gmail.com> wrote:
> > On 3/9/07, Vivek Khera <vivek@khera.org> wrote:
> > >
> > > On Mar 9, 2007, at 12:59 PM, Andrew Hammond wrote:
> > >
> > > > I'll give a +1 on using daemontools to run the slons. It's far cleaner
> > > > and easier than anything else I've tried. I've also put some more work
> > > > into polishing the slon-mkservice.sh script, which I would be happy to
> > > > contribute back if there's any interest.
> > >
> > > absolutely!
> >
> > Ok, I'll spit-n-polish then email my version to you.
>
> Attached. Feedback is very welcome. Yes, I realize it creates
> annoyingly verbose directory names for the services.
>
> logrep is a tool intended to allow efficient real-time filtering of
> log files so that you can pull semantically related information from a
> highly verbose log (ie DEBUG2) and put it all in one place. This is
> very handy if you want to implement automated monitoring scripts for
> stuff like nagios. It relies on the tail -F functionality (tested on
> FreeBSD, OSX, Linux).
>
> I've included a multilog filter for use with logrep that plucks out
> everything related to subscribe as an example. I'll eventually get
> around to writing a nagios monitor that does something clever with
> this log output (like say warn when a slon is doing the initial copy
> so you know not to clobber it). I'd love to see other filters that
> pull out other interesting stuff, like specific error conditions, for
> example.

And now with the debugging and other cleanups...


Andrew

["logrep-mkservice.sh" (application/x-sh)]

#!/bin/sh
#
# $Id: logrep-mkservice.sh 12897 2007-03-28 18:30:08Z ahammond $
#
# logrep: use tail -F to pull from another log file for filtering to create
# special purpose log files. Some example filters follow.
#
# For non-interactive use, set the following environment variables.
# BASEDIR SYSUSR SOURCE EXTENSION CRITERIA 
#
# Slony subscribe specific information
# '-*' '+* * ERROR*' '+* * WARN*' '+* * CONFIG enableSubscription*' '+* * DEBUG2 \
remoteWorkerThread_* prepare to copy table*' '+* * DEBUG2 remoteWorkerThread_* all \
tables for set * found on subscriber*' '+* * DEBUG2 remoteWorkerThread_* copy*' '+* * \
DEBUG2 remoteWorkerThread_* Begin COPY of table*' '+* * DEBUG2 remoteWorkerThread_* * \
bytes copied for table*' '+* * DEBUG2 remoteWorkerThread_* * seconds to*' '+* * \
DEBUG2 remoteWorkerThread_* set last_value of sequence*' '+* * DEBUG2 \
remoteWorkerThread_* copy_set*' #
# Errors to trigger nagios
# '-*' '+* * ERROR*'

DEFAULT_BASEDIR='/usr/local/etc'
DEFAULT_SYSUSR='pgsql'                          # FreeBSD-centric. Oh well.
DEFAULT_SOURCE='slon_123'
DEFAULT_EXTENSION='_NODEBUG'
DEFAULT_CRITERIA="'-* [*] DEBUG*'"

if [ -z "$BASEDIR" ]; then
    echo -n "Where do you want the service dir created? Don't want to created this in \
/service or /var/service. Once it's created, either link or move it to the service \
directory (since linking is an atomic filesystem action). If your service directory \
is on a small, relatively static partition, you will almost certainly want to put \
this on a partition that can handle some log files and then link. [$DEFAULT_BASEDIR]: \
"  read BASEDIR
    if [ -z "$BASEDIR" ]; then
        BASEDIR="$DEFAULT_BASEDIR"
    fi
fi
echo "BASEDIR=$BASEDIR"

if [ -z "$SYSUSR" ]; then
    echo -n "System user name for followgrep to run under [$DEFAULT_SYSUSR]: "
    read SYSUSR
    if [ -z "$SYSUSR" ]; then
        SYSUSR="$DEFAULT_SYSUSR"
    fi
fi
echo "SYSUSR=$SYSUSR"

if [ -z "$SOURCE" ]; then
    echo -n "Service to follow [$DEFAULT_SOURCE]: "
    read SOURCE
    if [ -z "$SOURCE" ]; then
        DATABASE="$DEFAULT_SOURCE"
    fi
fi
echo "SOURCE=$SOURCE"

if [ -z "$EXTENSION" ]; then
    echo -n "Name extension [$DEFAULT_EXTENSION]: "
    read EXTENSION
    if [ -z "$EXTENSION" ]; then
        EXTENSION="$DEFAULT_EXTENSION"
    fi
fi
echo "EXTENSION=$EXTENSION"

if [ -z "$CRITERIA" ]; then
    echo -n "Criteria [$DEFAULT_CRITERIA]: "
    read CRITERIA
    if [ -z "$CRITERIA" ]; then
        CRITERIA="$DEFAULT_CRITERIA"
    fi
fi
echo "CRITERIA=$CRITERIA"

DIR="$BASEDIR/logrep_$SOURCE$EXTENSION"
echo "Service dir will be created under $DIR"

mkdir -p "$DIR/supervise" "$DIR/log/supervise" "$DIR/log/main" || exit -1
# Make sure the log file initially exists. This allows others to tail -F it
# before it starts getting populated.
touch "$DIR/log/main/current" || exit -1

# create the run script
cat > "$DIR/run" <<EOF
#!/bin/sh
exec 2>&1
exec setuidgid $SYSUSR tail -F "$BASEDIR/$SOURCE/log/main/current"
EOF
chmod a+x "$DIR/run"
echo "$DIR/run created"

# create the run file for the multilog
cat > "$DIR/log/run" <<EOF
#!/bin/sh
# DO NOT add another timestamp using the t parameter to multilog. Unless
# of course you like being confused.
# Note that size (in bytes) of the log files (before they get rotated) is 
# controlled by the s parameter. You might want to increase this. It goes
# up to a maximum of 16777215 (15MB) and defaults to 99999 (97kB).
# I'm using 10485760 (10MB)
# The n paramter decides how many old log files to keep around. Defaults
# to 10. 
exec setuidgid $SYSUSR multilog s10485760 n99 $CRITERIA ./main
EOF
chmod a+x "$DIR/log/run"
echo "$DIR/log/run created"

# fix permissions and ownership
chown -R "$SYSUSR" "$DIR"
chmod a+rX "$DIR"

cat <<EOF
When you're ready to start logrep, simply link the newly created dir, 
$DIR
under your current supervisor's directory. On FreeBSD this is /var/service
by default. So the command would be:

sudo ln -s $DIR /var/service

The service should be started within 5 seconds.
EOF


["slon-mkservice.sh" (application/x-sh)]

#!/bin/sh
#
# $Id: slon-mkservice.sh 12897 2007-03-28 18:30:08Z ahammond $
#
# Create a slon service directory for use with svscan from deamontools.
# This uses multilog in a pretty basic way. See logrep for cleverness.
# Currently very limited error handling. This probably needs to be fixed...
#
# For non-interactive use, set the following environment variables.
# BASEDIR SYSUSR PASSFILE DBUSER HOST PORT DATABASE CLUSTER SLON_BINARY

DEFAULT_SLON_BINARY=`which slon`                # silly, wild-ass guess
DEFAULT_BASEDIR='/usr/local/etc'
DEFAULT_SYSUSR='pgsql'                          # FreeBSD-centric. Oh well.
DEFAULT_DBUSR='slony'                           # Best Practice...
DEFAULT_SLON_DEBUG_LEVEL=2                      # default to "debug2" level
DEFAULT_HOST='localhost'                        # maybe the unix socket would be \
better? DEFAULT_PORT=5432

if [ -z "$BASEDIR" ]; then
    echo -n "Where do you want the service dir created? Don't want to created this in \
/service or /var/service. Once it's created, either link or move it to the service \
directory (since linking is an atomic filesystem action). If your service directory \
is on a small, relatively static partition, you will almost certainly want to put \
this on a partition that can handle some log files and then link. [$DEFAULT_BASEDIR]: \
"  read BASEDIR
    if [ -z "$BASEDIR" ]; then
        BASEDIR="$DEFAULT_BASEDIR"
    fi
fi
echo "BASEDIR=$BASEDIR"

if [ -z "$SYSUSR" ]; then
    echo -n "System user name for slon to run under [$DEFAULT_SYSUSR]: "
    read SYSUSR
    if [ -z "$SYSUSR" ]; then
        SYSUSR="$DEFAULT_SYSUSR"
    fi
fi
echo "SYSUSR=$SYSUSR"

if [ -z "$PASSFILE" ]; then
    DEFAULT_PASSFILE=`eval echo "~$SYSUSR/.pgpass"`
    echo -n "And $SYSUSR's .pgpass file? [$DEFAULT_PASSFILE]: "
    read PASSFILE
    if [ -z "$PASSFILE" ]; then
        PASSFILE="$DEFAULT_PASSFILE"
    fi
fi
echo "PASSFILE=$PASSFILE"

if [ -z "$DBUSER" ]; then
    echo -n "Database user for slon to connect as [$DEFAULT_DBUSR]: "
    read DBUSER
    if [ -z "$DBUSER" ]; then
        DBUSER="$DEFAULT_DBUSR"
    fi
fi
echo "DBUSER=$DBUSER"

if [ -z "$HOST" ]; then
    echo -n "Host to connect to [$DEFAULT_HOST]: "
    read HOST
    if [ -z "$HOST" ]; then
        HOST="$DEFAULT_HOST"
    fi
fi
echo "HOST=$HOST"
if echo "$HOST" | grep / > /dev/null; then
    echo "Using / in your host name will make things break. Aborting."
    exit -1
fi

if [ -z "$PORT" ]; then
    echo -n "Port [$DEFAULT_PORT]: "
    read PORT
    if [ -z "$PORT" ]; then
        PORT="$DEFAULT_PORT"
    fi
fi
echo "PORT=$PORT"

if [ -z "$DATABASE" ]; then
    echo -n "Database [$DBUSER]: "
    read DATABASE
    if [ -z "$DATABASE" ]; then
        DATABASE="$DBUSER"
    fi
fi
echo "DATABASE=$DATABASE"
if echo "$DATABASE" | grep / > /dev/null; then
    echo "Using / in your database name will make things break. Aborting."
    exit -1
fi

if [ -z "$CLUSTER" ]; then
    echo -n "Cluster name: [$DATABASE]: "
    read CLUSTER
    if [ -z "$CLUSTER" ]; then
        CLUSTER="$DATABASE"
    fi
fi
echo "CLUSTER=$CLUSTER"
if echo "$CLUSTER" | grep / > /dev/null; then
    echo "Using / in your cluster name will make things break. Aborting."
    exit -1
fi

if [ -z "$SLON_BINARY" ]; then
    echo -n "Where is the slon binary? [$DEFAULT_SLON_BINARY]: "
    read SLON_BINARY
    if [ -z "$SLON_BINARY" ]; then
        SLON_BINARY=$DEFAULT_SLON_BINARY
    fi
fi
echo "SLON_BINARY=$SLON_BINARY"

DIR="$BASEDIR/slon_${CLUSTER}_${HOST}_${PORT}_$DATABASE"
CONFIGFILE="$DIR/slon.conf"
echo "CONFIGFILE=$CONFIGFILE"

echo "Service dir will be created under $DIR"

mkdir -p "$DIR/env" "$DIR/supervise" "$DIR/log/supervise" "$DIR/log/main" || exit -1
# Make sure the log file initially exists. This allows others to tail -F it
# before it starts getting populated. go go logrep!
touch "$DIR/log/main/current" || exit -1

# Set up the slon.conf file
cat > "$CONFIGFILE" <<EOF
# $CONFIGFILE
##############################################################################
# Connection settings

# Set the cluster name that this instance of slon is running against.
# The default is to read it off the command line.
cluster_name "$CLUSTER"

# Set slon's connection info; default is to read it off the command line.
conn_info "host=$HOST port=$PORT dbname=$DATABASE user=$DBUSER"

# Execute this SQL on each node at slon connect time. Useful to set logging
# levels, or to tune the planner/memory settings. You can specify multiple
# statements by separating them with a ; 
#sql_on_connection 

##############################################################################
# Logging

# If you want to use syslog then redir output from the slon to /dev/null and
# remove the log dir.

# Sets up logging to syslog. If this parameter is 1, messages go both to
# syslog and the standard output. A value of 2 sends output only to syslog
# (some messages will still go to the standard output/error). 
# Default is 0, which means syslog is off.
#syslog 0

# Sets the syslog "facility" to be used when syslog enabled. Valid values
# are LOCAL0, LOCAL1, LOCAL2, LOCAL3, LOCAL4, LOCAL5, LOCAL6, LOCAL7. 
# Default is LOCAL0.
#syslog_facility LOCAL0

# Sets the program name used to identify slon messages in syslog.
# The default is slon.
#syslog_ident slon

# Debug log level (higher value ==> more output).
# Range: [0,4], default 4
log_level $DEFAULT_SLON_DEBUG_LEVEL     # 3 and up is generally too high for \
production

# Determins, if you would like the pid of the (parent) slon process to appear
# in each log line entry.
# Default 0
log_pid 1           # good to know

# Determines if you would like the timestamp of the event being logged to
# appear in each log line entry.
log_timestamp  0    # multilog will insert a tai64n timestamp

# A strftime()-conformant format string for use if log_timestamp is enabled.
# The default is "%Y-%m-%d %H:%M:%S %Z"
#log_timestamp_format = "%Y-%m-%d %H:%M:%S %Z"

# Location and filename you would like for a file containing the Process ID
# of the slon process.
# The default is not defined in which case no file is written.
#pid_file       # daemontools renders this unnecessary, even undesireable
                # use instead the svc interface to signal the slon.

##############################################################################
# Archive Logging

# This indicates in what directory sync archive files should be stored.
#archive_dir

# This indicates a Unix command to be submitted each time an archive log
# is successfully generated.
#command_on_logarchive

##############################################################################
# Event Tuning

# Check for updates at least this often in milliseconds.
# Range: [10-60000], default 100
#sync_interval 100 

# Maximum amount of time in milliseconds before issuing a SYNC event, 
# This prevents a possible race condition in which the action sequence is
# bumped by the trigger while inserting the log row, which makes this bump
# is immediately visible to the sync thread, but the resulting log rows are
# not visible yet. If the SYNC is picked up by the subscriber, processed
# and finished before the transaction commits, this transaction's changes
# will not be replicated until the next SYNC. But if all application
# activity suddenly stops, there will be no more sequence bumps, so the
# high frequent -s check won't detect that. Thus, the need for
# sync_interval_timeout.
# Range: [0-120000], default 1000
#sync_interval_timeout 1000 

# Maximum number of SYNC events to group together when/if a subscriber falls
# behind. SYNCs are batched only if there are that many available and if they
# are contiguous. Every other event type in between leads to a smaller batch.
# And if there is only one SYNC available, even -g60 will apply just that one.
# As soon as a subscriber catches up, it will apply every single SYNC by itself.
# Range: [0,10000], default: 6
#sync_group_maxsize 6

# Sets how many cleanup cycles to run before a vacuum is done. 0 disables the
# builtin vacuum, intended to be used with the pg_autovacuum daemon.
# Range: [0,100], default: 3
#vac_frequency 3

# Maximum time planned for grouped SYNCs. If replication is behind, slon will
# try to increase numbers of syncs done targetting that they should take this
# quantity of time to process. If the value is set to 0, this logic will be
# ignored. 
# Range [10000,600000] ms, default 60000.
#desired_sync_time 60000 

# This must be used in conjunction with quit_sync_finalsync, and indicates
# which provider node's worker thread should be watched to see if the slon
# should terminate due to reaching some desired "final" event number.
#quit_sync_provider 0

# Final event number to process. This must be used in conjunction with
# quit_sync_finalsync, and allows the slon to terminate itself once it
# reaches a certain event for the specified provider. If the value is set
# to 0, this logic will be ignored. 
#quit_sync_finalsync 0

# Indicates an interval by which this node should lag its providers. If set,
# this is used in the event processing loop to modify what events are to be
# considered for queueing; those events newer than
# now() - lag_interval::interval are left out, to be processed later.
# If the value is left empty, this logic will be ignored. 
#lag_interval

# Size above which an sl_log_? row's log_cmddata is considered large. Up to
# 500 rows of this size are allowed in memory at once. Rows larger than that
# count into the sync_max_largemem space allocated and free()'ed on demand.
# The default value is 8192, meaning that your expected memory consumption
# (for the LOG cursor) should not exceed 8MB.
#sync_max_rowsize 8192

# Maximum memory allocated for large rows, where log_cmddata are larger than
# sync_max_rowsize. Note that the algorithm reads rows until after this value
# is exceeded. Otherwise, a tuple larger than this value would stall
# replication. As a result, don't assume that memory consumption will remain
# smaller than this value.  The default value is 5242880.
#sync_max_largemem 5242880

# How long should the remote listener wait before treating the event selection
# criteria as having timed out?
# Range: [30-30000], default 300 
#remote_listen_timeout 300

EOF

# Set up the envdir contents. Generously.
echo "$SLON_BINARY"                 > $DIR/env/SLON_BINARY
echo "$CONFIGFILE"                  > $DIR/env/CONFIGFILE
echo "$CLUSTER"                     > $DIR/env/CLUSTER
echo "$HOST"                        > $DIR/env/PGHOST
echo "$PORT"                        > $DIR/env/PGPORT
echo "$DATABASE"                    > $DIR/env/PGDATABASE
# The absence of PGPASSWORD is not an oversight. Use .pgpass
# Configure the location of .pgpass file here...
# I'd like a better solution than this for expanding the homedir.
echo "$PASSFILE"                    > $DIR/env/PGPASSFILE 
echo "$DBUSER"                      > $DIR/env/PGUSER
# Change these if you're stupid or unfortunate enough to have to.
echo 'ISO'                          > $DIR/env/PGDATESTYLE
echo 'UTC'                          > $DIR/env/PGTZ

# create the run script for the slon
cat > "$DIR/run" <<EOF
#!/bin/sh
# Note that the slon binary is a variable, so you can edit your envdir
# settings to upgrade slons then restart them.
exec 2>&1
exec envdir ./env sh -c 'exec setuidgid ${SYSUSR} "\${SLON_BINARY}" -f \
"\${CONFIGFILE}"' EOF
chmod a+x "$DIR/run"
echo "$DIR/run created"

# create the run file for the multilog
cat > "$DIR/log/run" <<EOF
#!/bin/sh
# This puts everything in the main log. Unfortunately multilog only allows
# you to select which log you want to write to as opposed to writing each
# line to every log which matches the criteria.
#
# Note that size (in bytes) of the log files (before they get rotated) is 
# controlled by the s parameter. You might want to increase this. It goes
# up to a maximum of 16777215 (15MB) and defaults to 99999 (97kB).
# I'm using 10485760 (10MB)
# The n paramter decides how many old log files to keep around. Defaults
# to 10. 
exec setuidgid $SYSUSR multilog t s10485760 n99 ./main
EOF
chmod a+x "$DIR/log/run"
echo "$DIR/log/run created"

# create and fix permissions for .pgpass appropriately
touch "$PASSFILE"
chmod 600 "$PASSSFILE"
if [ ! -s "$PASSFILE" ]; then
    echo "Populating $PASSFILE"
    cat > "$PASSFILE" <<EOF
#hostname:port:database:username:password
#$HOST:$PORT:*:$DBUSER:secret
EOF
fi

# fix permissions and ownership
chown -R "$SYSUSR" "$DIR"
chmod a+rX "$DIR"

cat <<EOF
When you're ready to start the slon, simply link the newly created dir, 
$DIR
under your current supervisor's directory. On FreeBSD this is /var/service
by default. So the command would be:

sudo ln -s $DIR /var/service

The slon should be started within 5 seconds.

You can further configure the slon by modifying stuff in the 
$DIR/env
directory. Specifically, make sure that the SLON variable points to the right
slon binary. You may also want to edit
$CONFIGFILE

Finally, you need to make sure that $PASSFILE 
is correctly in place. If you didn't have one before (or it was empty), it
has been created and populated with some sample data.

You may also want to set up a logrep to filter out the more intresting
log lines. See logrep-mkservice.sh.
EOF



_______________________________________________
Slony1-general mailing list
Slony1-general@lists.slony.info
http://lists.slony.info/mailman/listinfo/slony1-general


[prev in list] [next in list] [prev in thread] [next in thread]