'Re: KDE Database'

[prev in list] [next in list] [prev in thread] [next in thread] 

List: kde-devel
Subject: Re: KDE Database
From: Craig Howard <kde () choward ! ca>
Date: 2005-11-05 21:27:49
Message-ID: 200511051627.49448.kde () choward ! ca
[Download RAW message or body]

On Saturday 05 November 2005 08:21, Christian Leh wrote:
> currently all settings of kde are stored in .rc files which gives us real
> advantages:
> - easy backup (mostly manually, but it works)
> - easy editing
> - simple usage
> - ...

[snip]

> now, when i want to store my current settings, i just put a date (today)
> into another table (or wherever) which holds these "backups". that would be
> everything need to be done.
>
> loading settings by this date will cause any setting to be loaded valid at
> this date (startdate <= date <= enddate).
>
> another advantage would be that u can set up a settings anywhere from where
> anyone can load his app settings (maybe in a company u can do this, also to
> prevent them to be changed by users). some kind of roles can handle that.
>
> or imagine u go to work and synchronize your settings from home.
> now u can use your kde like at home without copying any configurations
>
> there are still some aspects that have to be resolved:
> - a cache (loading settings everytime from db is not acceptable)
> - compatibility to existing apps / implementation in KConfig?!
> - growing of database (only last 5 changes or something)
> - what about app data files (i.e. kopete styles/smileys)

You mean kinda like a version control system?  Why not use one?  My entire 
home directory is stored in subversion.

I'm moving to Seattle in a few days, so I don't have permanent webspace at the 
moment, but here's the text of a document I've got roughly describing my 
setup.

One thing that I would like to see in the future is the separating of the 
config files into something similar to /etc and /var.  ie read only (which 
can be put into svn) and read/write (things like the number of times you've 
visited a bookmark).  Without this, some config files change constantly 
without having any meaningful changes, which just clutters up the svn st 
listing and increases the sizes of diffs.

Comments on my setup?
-- 

Craig Howard

["svnhome.html" (text/html)]

<html>
<head>
<title>Home Directory in Subversion</title>
</head>
<body>
<h3>Keeping Your Home Directory in Subversion</h3>
<hr>
<h4>Why?</h4>

I've got my entire home directory stored in Subversion. I do this for three
main reasons.

The first reason is that it allows me to move through time in my documents.
It's quite common for me to overwrite a file (source code, document, setting,
etc.) and regret it. Subversion allows me to recover the old version without
and hassle.

The second reason is to keep files synchronized between the systems that I
work on. I've got three systems that I frequently work on and keeping
everything synchronized was a nightmare. Now to keep things synchronized, all
I have to do is an update.

The third, and probably most important reason, is ease of backups. My
laptop hard drive died during a school term, but I was able to checkout all my
work on a different computer and keep working until the drive was replaced, at
which point I moved back to my laptop. I lost almost no time or data!

<h4>Creating the Repository</h4>

The first thing to do is choose a system to host the repository on. A
system with a good network connection and lots of disk space is important.
If you plan on using the subversion home directory on this machine, your
repository cannot be located in your home directory! Mine is located
in /home/svnroot/home. (I have other repositories located in /home/svnroot)

If you're like me and have a small network located behind a router, which
does NAT and you want updates and checkins to work when you're both at home and
away, you'll need a DNS entry. I bought choward.ca. The system that hosts the
repository needs a unique name. I use desktop.choward.ca. From the internet
this is just an alias for choward.ca, but from the inside this resolves to
192.168.0.13. As long as I use the full name whenever subversion needs a URL,
everything works the same from inside the house, as well as outside the
house. As an aside, you'll probably have to do a port forward as well to get
outside access.

<p>To actually create the repository, do:</p>

<p><samp>svnadmin create /home/svnroot/home</samp></p>

<p>Replace /home/svnroot/home with path where you want to create your
repository.</p>

<h4>Repository Layout</h4>

One of the nice things about subversion, in comparison to cvs, is that
directories are movable. I started out with my repository containing exactly
what was in my home directory, but later realized this was a mistake. I now
have the repository laid out as follows:

<p><pre>/</pre></p>
<p><pre>	/trunk</pre></p>
<p><pre>		/all files in home directory go here</pre></p>
<p><pre>	/tags</pre></p>
<p><pre>	/branches</pre></p>
<p><pre>	/clientspecs</pre></p>

<p>The motivation for these directories are explained in the
	<a href="http://svnbook.red-bean.com/">svnbook</a>.
The only difference is the /clientspec directory, which will be explained
below.</p>

<h4>Different Files on Different Computers</h4>

I use my laptop for most of my development and my desktop system typically
sits headless in another room and is just a fileserver. I want different files
on each of my computer, but I still want to synchronize things. What I really
want is Perforce's clientspecs. A clientspec is a map of directories in the
repository and local locations for the directories to be checked out to.

Unfortunately, Subversion does not have clientspecs. So, I decied to
simulate them. What Subversion does have is svn:externals properties.
An external is a mapping of a path in a Subversion repository (not necessarily
the same repository) to a local path. When subversion does a checkout or an
update, it will first do the operation on itself, then it will recurse into
each external definition.

There are two downsides to Subversion's externals. Checkins do not happen
recursively like checkouts do. All paths definitions must be absolute.
Subversion 1.5 has been tentatively targeted as the externals release.

My clientspec directory contains an empty subdirectory for each machine that
I use (or more correctly, each profile I want; there's no reason two machines
couldn't share a profile). I then define the svn:externals property on the
directory by referring to directories under /trunk. For example, my laptop
clientspec includes the following:

<p><pre>bin             svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/bin
doc             svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/doc
src             svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/src
work            svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/work
.dot-files      svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/dot-files
.kde-svn        svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/.kde-svn
.mozilla        svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/.mozilla
.vim            svn+ssh://desktop.choward.ca/home/svnroot/home/trunk/.vim</pre></p>

This allows changes to /src on my laptop to be picked up by any other
systems have have /src in their clientspec, but the files will not appear on
other machines. The .dot-files directory will be explained below.

<h4>Dealing with Dot Files</h4>

A limitation of clientspecs (definately in Subversion and probably in
Perforce too) is that each entry listed must be a directory; files are not
allowed. This creates a problem for simple dot files, like .bashrc. It's
important that these files are shared across systems, so they have to live in
/trunk and be linked to within the clientspecs. My solution is to put all
these files in a directory called .dot-files and I wrote a script to hard link
all the files in that directory into $HOME on each update and checkin.

<h4>Checkout Procedure</h4>

<p>To get my files on a new system, the first thing to do is install
Subversion.</p>

The home directory has to be checked out as your user, but to write to /home
you typically have to be root. My solution is to checkout into a subdirectory
of $HOME as the regular user and then move the directory into /home as
root.

<pre>svn co svn+ssh://desktop.choward.ca/home/svnroot/home craig
su
cd /home
mv craig craig~
mv craig~/craig .
rm -rf craig~</pre>

<h4>Convience Scripts</h4>

<p>I've created a few scripts to help myself out.  
	<a href="dot-files">dot-files</a>
Does the hard linking mentioned above.
	<a href="mysvn">mysvn</a>
Is meant to be run instead of the svn command.  It does some special casing for
a few commands.  For example, when doing a <em>svn st</em> in $HOME, all the
hard linked dot files will appear as unknown.  mysvn detects this and filters
them out with grep.  mysvn also runs dot-files after updates in $HOME.  The
special cases for handling maildirs isn't totally working at the moment
(because my mail was migrated to a different machine for the immediate
future).</p>

<h4>Automated Backups</h4>

<p>There are two easy ways of doing automated backups.  The first is
periodically doing an <em>svnadmin dump</em> of the repository.  This file can
be loaded back into an empty repository with <em>svnadmin load</em>.  I wrote 
	<a href="svnbackup">svnbackup</a>
to help with this.  It does a dump of all repositories it finds in
/home/svnroot to $HOME/backup.  The other method is to mirror the repository 
on another machine using 
	<a href="http://svk.elixus.org/">svk</a>.
I use this method and do an <em>svk sync</em> once a week.  Piece of
cake!</p>

<h4>Disk Usage</h4>

Using Subversion does increase disk usage, but at the price of today's hard
drives, who cares? At the moment, I'm not storing digital camera pictures,
music or videos, but I've considered storing my pictures. Right now my
repository itself takes up 1.1GB. It includes all my configuration settings
for 3 versions of KDE, all the source and data files for my contract work, all
the source and data for my other coding projects and all the school assignments
I did over the last few years. I've now made over 2000 commits.

<h4>Comments/Questions/Suggestions?</h4>

<p>Let me know at
	<a href="mailto:svnhome@choward.ca">svnhome@choward.ca</a>.
</p>

<hr>

<p><em>Last updated: $LastChangedDate$<em></p>

</body>
</html>

["dot-files" (application/x-shellscript)]

#!/bin/sh

CURDIR=`pwd`

cd $HOME

for i in .dot-files/.*; do
	f=`echo $i | cut -d / -f 2`
	if [ $f != "." -a $f != ".." -a $f != ".svn" ]; then
		rm -f $f
		ln .dot-files/$f $f
	fi
done

cd $CURDIR

["mysvn" (application/x-shellscript)]

#!/bin/sh

svn=`which svn`

if [ $# != 1 ]; then
	$svn "$@"
elif [ `pwd` == $HOME -a $1 == 'st' ]; then
	# don't view external definitions in local listing
	dotfiles=`ls -a .dot-files | egrep -v '^.(.|svn|)$'`
	# translate file listing into a regular expression
	# by converting the spaces delimiting filenames into \>)|(\?[ \t]+
	dotfiles="(\?[ \t]+"`echo $dotfiles | sed 's/ /\\\\>)|(\\\\?[ \\\\t]+/g'`"\>)"
	# use the regular expression to filter the svn st output
	$svn st | egrep -v "(^X)|$dotfiles"
elif [ `pwd` == $HOME -a $1 == 'up' ]; then
	# after running update, run dot-files to ensure that config files
	# are updated as well
	$svn up
	$HOME/bin/dot-files

	# set permissions to 600 on .ssh
	chmod 600 .ssh/*
elif [ $1 == 'mailupdate' ]; then
	olddir=`pwd`
	cd $HOME/.maildir

	# first force delete all missing files
	$svn st | egrep "^!" | cut -d ' ' -f 7 | xargs $svn rm --force

	# second add all new files
	$svn st | egrep "^\?" | cut -d ' ' -f 7 | xargs $svn add

	# now commit the dir
	$svn ci -m "Mail automated checkin for `date`"

	cd $olddir
else
	$svn "$@"
fi

["svnbackup" (application/x-shellscript)]

#!/bin/bash

python /home/craig/bin/scripts/svnbackup.py $@

>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

[prev in list] [next in list] [prev in thread] [next in thread] 