[prev in list] [next in list] [prev in thread] [next in thread]
List: subversion-dev
Subject: New version of fsfs-reshard.py for repository maintenance
From: ymartin59 () free ! fr
Date: 2010-02-24 17:02:25
Message-ID: 499746160.681271267030945466.JavaMail.root () zimbra8-e1 ! priv ! proxad ! net
[Download RAW message or body]
Hello,
I have looked for ways to migrate old repositories to latest 1.6.x.
And I let the "dump/import" apart because my window maintenance was too short
and my available disk space too limit to migrate my company's 200 repositories, \
around 27 Gb.
So in a first step, I have just apply "svnadmin upgrade" on repositories.
Now I'm aware the repositories are still in "linear layer" and sharding will become \
soon a requirement, even on a Linux system.
So I have re-worked the fsfs-reshard.py tool to support new 1.6 formats with some \
additional features like shard statistics (current and willing) and also unpacking \
shards.
Usages:
$ fsfs-reshard.py repository
Report current repository version and layer. If sharding, dump also shard \
statistics. No changes.
$ fsfs-reshard.py repository target=n
Report current repository version and layer. Dump shard statistics in case of \
n-revisions-shards. No changes.
$ fsfs-reshard.py repository n
Apply new shard size n-revisions. If n=0, change to linear layer.
This new script version does not upgrade the repository version. It only change layer \
linear/shard. The "svnadmin upgrade repository" command must be used to upgrade a \
repository.
If the repository has packed shards, revisions are unpacked before re-sharding.
After re-sharding, you probably want to run the command "svnadmin pack repository".
Of course, apply the script on a copy first. Do not hesitate to run "svnadmin verify \
repository" before getting it online again.
Hope this job help any large repository administrator.
As it is my first real work with Python, it is necessary to review the source before \
committing - even if I have done a lot of tests. Any volunteer ?
Best regards
Yves Martin
["fsfs-reshard.py" (text/x-python)]
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# fsfs-reshard.py REPOS_PATH
# fsfs-reshard.py REPOS_PATH -target=MAX_FILES_PER_SHARD
#
# Display repository information about fsfs db.
#
# fsfs-reshard.py REPOS_PATH MAX_FILES_PER_SHARD
#
# Perform an offline conversion of an FSFS repository between linear (format
# 2, usable by Subversion 1.4+) and sharded (format 3/4, usable by Subversion
# 1.5+) layouts.
#
# The MAX_FILES_PER_SHARD argument specifies the maximum number of files
# that will be stored in each shard (directory), or zero to specify a linear
# layout. Subversion 1.5 uses a default value of 1000 files per shard.
#
# As the repository will not be valid while the conversion is in progress,
# the repository administrator must ensure that access to the repository is
# blocked for the duration of the conversion.
#
# In the event that the conversion is interrupted, the repository will be in
# an inconsistent state. The repository administrator should then re-run
# this tool to completion.
#
#
# Note that, currently, resharding from one sharded layout to another is
# likely to be an extremely slow process. To reshard, we convert from a
# sharded to linear layout and then to the new sharded layout. The problem
# is that the initial conversion to the linear layout triggers exactly the
# same 'large number of files in a directory' problem that sharding is
# intended to solve.
#
# ====================================================================
# Licensed to the Subversion Corporation (SVN Corp.) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The SVN Corp. licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# ====================================================================
#
# Subversion 1.6 format 4 support, unpack operation and shard file
# computation contributed by Yves Martin (ymartin59 0x40 free 0x2E fr)
#
# $HeadURL: http://svn.collab.net/repos/svn/trunk/tools/server-side/fsfs-reshard.py $
# $LastChangedDate: 2009-07-08 00:03:10 +0200 (Wed, 08 Jul 2009) $
# $LastChangedBy: hwright $
# $LastChangedRevision: 38370 $
import os, stat, sys, shutil
from errno import EEXIST
def usage():
"""Print a usage message and exit."""
print("""usage: %s REPOS_PATH [target=MAX_FILES_PER_SHARD]
Computes shard sizes for current repository or for a target
MAX_FILES_PER_SHARD to tune this parameter according to
performance criteria.
usage: %s REPOS_PATH MAX_FILES_PER_SHARD [START END]
Perform an offline conversion of an FSFS repository between linear
(readable by Subversion 1.4 or later) and sharded (readable by
Subversion 1.5 or later) layouts.
It is recommended to first upgrade your repository to your current
Subversion release with 'svnadmin upgrade REPOS_PATH'.
Packed shards are unpacked before converting. According to your
needs, you may want to invoke 'svnadmin pack REPOS_PATH' after.
The MAX_FILES_PER_SHARD argument specifies the maximum number of
files that will be stored in each shard (directory), or zero to
specify a linear layout. Subversion 1.5 uses a default value of
1000 files per shard.
Convert revisions START through END inclusive if specified, or all
revisions if unspecified.
""" % (sys.argv[0], sys.argv[0]))
sys.exit(1)
def incompatible_repos_format(repos_path, format):
"""Print an error saying that REPOS_PATH is a repository with an
incompatible repository format FORMAT, then exit."""
sys.stderr.write("""error: unable to convert repository '%s'.
This repository is not compatible with this tool. Valid
repository formats are '3' or '5'; this repository is
format '%s'.
""" % (repos_path, format))
sys.stderr.flush()
sys.exit(1)
def incompatible_fs_format(repos_path, format):
"""Print an error saying that REPOS_PATH is a repository with an
incompatible filesystem format FORMAT, then exit."""
sys.stderr.write("""error: unable to convert repository '%s'.
This repository contains a filesystem that is not compatible with
this tool. Valid filesystem formats are '1', '2', or '3'; this
repository contains a filesystem with format '%s'.
""" % (repos_path, format))
sys.stderr.flush()
sys.exit(1)
def unexpected_fs_format_options(repos_path):
"""Print an error saying that REPOS_PATH is a repository with
unexpected filesystem format options, then exit."""
sys.stderr.write("""error: unable to convert repository '%s'.
This repository contains a filesystem that appears to be invalid -
there is unexpected data after the filesystem format number.
""" % repos_path)
sys.stderr.flush()
sys.exit(1)
def incompatible_fs_format_option(repos_path, option):
"""Print an error saying that REPOS_PATH is a repository with an
incompatible filesystem format option OPTION, then exit."""
sys.stderr.write("""error: unable to convert repository '%s'.
This repository contains a filesystem that is not compatible with
this tool. This tool recognises the 'layout' option but the
filesystem uses the '%s' option.
""" % (repos_path, option))
sys.stderr.flush()
sys.exit(1)
def warn_about_fs_format_1(repos_path, format_path):
"""Print a warning saying that REPOS_PATH contains a format 1 FSFS
filesystem that we can't reconstruct, then exit."""
sys.stderr.write("""warning: conversion of '%s' will be one-way.
This repository is currently readable by Subversion 1.1 or later.
This tool can convert this repository to one that is readable by
either Subversion 1.4 (or later) or Subversion 1.5 (or later),
but it is not able to convert it back to the original format - a
separate dump/load step would be required.
If you would like to upgrade this repository anyway, delete the
file '%s' and re-run this tool.
""" % (repos_path, format_path))
sys.stderr.flush()
sys.exit(1)
def check_repos_format(repos_path):
"""Check that REPOS_PATH contains a repository with a suitable format;
print a message and exit if not."""
format_path = os.path.join(repos_path, 'format')
try:
format_file = open(format_path)
format = format_file.readline()
if not format.endswith('\n'):
incompatible_repos_format(repos_path, format + ' <missing newline>')
format = format.rstrip('\n')
if format == '3' or format == '5':
pass
else:
incompatible_repos_format(repos_path, format)
except IOError:
# In all likelihood, the file doesn't exist.
incompatible_repos_format(repos_path, '<unreadable>')
def check_fs_format(repos_path):
"""Check that REPOS_PATH contains a filesystem with a suitable format,
or that it contains no format file; print a message and exit if neither
is true. Return an array [format number, shard size] whether the filesystem is sharded."""
result = [0, 0]
db_path = os.path.join(repos_path, 'db')
format_path = os.path.join(db_path, 'format')
if not(os.path.exists(format_path)):
# Recover from format.bak if interrupted
format_path = os.path.join(db_path, 'format.bak')
if not(os.path.exists(format_path)):
sys.stderr.write("error: db/format and db/format.bak missing.\n")
sys.stderr.flush()
sys.exit(1)
try:
format_file = open(format_path)
format = format_file.readline()
if not format.endswith('\n'):
incompatible_fs_format(repos_path, format + ' <missing newline>')
format = format.rstrip('\n')
if format == '1':
# This is a format 1 (svndiff0 only) filesystem. We can upgrade it,
# but we can't downgrade again (since we can't uncompress any of the
# svndiff1 deltas that may have been written). Warn the user and exit.
warn_about_fs_format_1(repos_path, format_path)
if format == '2':
pass
elif format == '3':
pass
elif format == '4':
pass
else:
incompatible_fs_format(repos_path, format)
result[0] = format;
for line in format_file:
if format == '2':
unexpected_fs_format_options(repos_path)
line = line.rstrip('\n')
if line == 'layout linear':
pass
elif line.startswith('layout sharded '):
result[1] = int(line[15:])
else:
incompatible_fs_format_option(repos_path, line)
format_file.close()
except IOError:
# The format file might not exist if we've previously been interrupted,
# or if the user is following our advice about upgrading a format 1
# repository. In both cases, we'll just assume the format was
# compatible.
pass
return result
def current_file(repos_path):
"""Return triple of (revision, next_node_id, next_copy_id) from
REPOS_PATH/db/current ."""
return open(os.path.join(repos_path, 'db', 'current')).readline().split()
def backup_fs_format(repos_path):
"""Rename the filesystem format file for repository REPOS_PATH.
Do not raise an error if the file is already renamed."""
format_path = os.path.join(repos_path, 'db', 'format')
try:
statinfo = os.stat(format_path)
except OSError:
# The file probably doesn't exist.
return
format_bak_path = os.path.join(repos_path, 'db', 'format.bak')
# On Windows, we need to ensure the file is writable before we can
# rename/remove it.
os.chmod(format_path, statinfo.st_mode | stat.S_IWUSR)
try:
os.rename(format_path, format_bak_path)
except OSError:
# Unexpected but try to go on
os.remove(format_bak_path)
os.rename(format_path, format_bak_path)
def write_fs_format(repos_path, contents):
"""Write a new filesystem format file for repository REPOS_PATH containing
CONTENTS."""
format_path = os.path.join(repos_path, 'db', 'format')
format_bak_path = os.path.join(repos_path, 'db', 'format.bak')
# Permissions and owner/group are preserved with rename
try:
os.rename(format_bak_path, format_path)
except OSError:
# Unexpected but try to go on
os.remove(format_path)
f = open(format_path, 'wb')
f.write(contents)
f.close()
def suffix_unpacked_shard(path):
"""Add '.shard' suffix to unpacked shard number directory."""
for name in os.listdir(path):
if name.endswith('.shard') or name.endswith('.pack'):
continue
subdir_path = os.path.join(path, name)
if not os.path.isdir(subdir_path):
continue
os.rename(subdir_path, subdir_path + '.shard')
def linearise(path):
"""Move all the files in subdirectories of PATH into PATH, and remove the
subdirectories. Handle conflicts between subdirectory names and files
contained in subdirectories by ensuring subdirectories have a '.shard'
suffix prior to moving (the files are assumed not to have this suffix.
Abort if a subdirectory is found to contain another subdirectory."""
suffix_unpacked_shard(path)
# Now move all the subdirectory contents into the parent and remove
# the subdirectories.
for root_path, dirnames, filenames in os.walk(path):
if root_path == path:
continue
if len(dirnames) > 0:
sys.stderr.write("error: directory '%s' contains other unexpected directories.\n" \
% root_path)
sys.stderr.flush()
sys.exit(1)
for name in filenames:
from_path = os.path.join(root_path, name)
to_path = os.path.join(path, name)
os.rename(from_path, to_path)
os.rmdir(root_path)
def shard(path, max_files_per_shard, start, end):
"""Move the files for revisions START to END inclusive in PATH into
subdirectories of PATH named such that subdirectory '0' contains at most
MAX_FILES_PER_SHARD files, those named [0, MAX_FILES_PER_SHARD). Abort if
PATH is found to contain any entries with non-numeric names."""
tmp = path + '.reshard'
try:
os.mkdir(tmp)
except OSError, e:
if e.errno != EEXIST:
raise
# Move all entries into shards named N.shard.
for rev in range(start, end + 1):
name = str(rev)
shard = rev // max_files_per_shard
shard_name = str(shard) + '.shard'
from_path = os.path.join(path, name)
to_path = os.path.join(tmp, shard_name, name)
try:
os.rename(from_path, to_path)
except OSError:
# The most likely explanation is that the shard directory doesn't
# exist. Let's create it and retry the rename.
os.mkdir(os.path.join(tmp, shard_name))
os.rename(from_path, to_path)
# Now rename all the shards to remove the suffix.
skipped = 0
for name in os.listdir(tmp):
if not name.endswith('.shard'):
sys.stderr.write("warning: ignoring unexpected subdirectory '%s'.\n" \
% os.path.join(tmp, name))
sys.stderr.flush()
skipped += 1
continue
from_path = os.path.join(tmp, name)
to_path = os.path.join(path, os.path.basename(from_path)[:-6])
os.rename(from_path, to_path)
skipped == 0 and os.rmdir(tmp)
def unpack_shard(packed_path, unpacklinear, first_rev, revs_size):
"""Compute revision sizes in a packed shard at packed_path
and unpack revision except if unpacklinear is false.
The first revision of the shard has first_rev number.
Revision sizes are stored in rev_sizes dictionnary."""
copy_buffer_size = 4096
manifest = open(os.path.join(packed_path, 'manifest'), 'r')
pack_path = os.path.join(packed_path, 'pack')
end_pack = os.path.getsize(pack_path)
if unpacklinear:
pack = open(pack_path, 'rb')
last_position = int(manifest.readline())
rev_index = first_rev
while last_position < end_pack:
# Read next revision start byte in pack file
try:
byte_position = int(manifest.readline())
except ValueError:
# last revision: end of pack file
byte_position = end_pack
revs_size[rev_index] = byte_position - last_position
if unpacklinear:
# Extract revision from pack file
arev = open(os.path.join(packed_path, os.path.pardir, str(rev_index)), 'wb')
pack.seek(last_position)
while last_position < byte_position:
bytes_tocopy = copy_buffer_size
if (byte_position - last_position) < copy_buffer_size:
bytes_tocopy = byte_position - last_position
rev_buffer = pack.read(bytes_tocopy)
arev.write(rev_buffer)
last_position += len(rev_buffer)
if bytes_tocopy < copy_buffer_size:
break
arev.close()
else:
last_position = byte_position
rev_index += 1
# Close file descriptors
manifest.close()
if unpacklinear:
pack.close()
return revs_size
def compute_rev_sizes(revs_path, current_shard, unpacklinear):
"""Compute revision sizes based on current shard capacity
Support either linear, sharded or packed revisions.
If unpacklinear is True, packed sharded are linearized too."""
revs_size = {}
for root_path, dirnames, filenames in os.walk(revs_path):
if len(filenames) > 0:
for name in filenames:
try:
revnum = int(name)
revs_size[revnum] = os.path.getsize(os.path.join(root_path, name))
except ValueError:
pass
if len(dirnames) > 0:
for name in dirnames:
if (not(name.endswith('.pack'))):
continue
shard_number = int(name[:-5])
shard_path = os.path.join(root_path, name)
# get revision sizes from packed shard [and unpack]
revs_size = unpack_shard(shard_path, unpacklinear, current_shard * shard_number, revs_size)
if unpacklinear:
# remove x.pack structure
shutil.rmtree(shard_path)
return revs_size
def compute_shard_sizes(revs_size, max_files_per_shard):
"""Compute shard sizes based on target max_files_per_shard
and map of revision size."""
current_shard = 0
current_shard_size = 0
min_shard_size = 2**63
max_shard_size = 0
shard_size_sum = 0
for i, size in revs_size.iteritems():
current_shard_size += size
if ((i + 1) % max_files_per_shard) == 0:
print 'Shard %d size: %d' % (current_shard, current_shard_size)
shard_size_sum += current_shard_size
if current_shard_size < min_shard_size:
min_shard_size = current_shard_size
if current_shard_size > max_shard_size:
max_shard_size = current_shard_size
current_shard_size = 0
current_shard += 1
if current_shard_size != 0:
print 'Shard %d size: %d' % (current_shard, current_shard_size)
if current_shard > 0:
print 'Average full-shard size %d. Minimum: %d, Maximum: %d.' \
% ((shard_size_sum / current_shard), min_shard_size, max_shard_size)
def linearise_packed_shards(revs_path, current_shard, min_unpacked_rev_path):
"""Linearise packed shards in revs_path directory based on
current_shard number of revisions per shard.
min-unpacked-rev at min_unpacked_rev_path is reset to 0."""
# Suffix unpacked shard to prevent conflicts
suffix_unpacked_shard(revs_path)
# Linearise packed shards
compute_rev_sizes(revs_path, current_shard, True)
# Reset min-unpacked-rev
min_unpacked_rev_file = open(min_unpacked_rev_path, 'wb')
min_unpacked_rev_file.write('0\n')
min_unpacked_rev_file.close()
def main():
if len(sys.argv) < 2:
usage()
repos_path = sys.argv[1]
# Get [number format, sharded]
fs_format = check_fs_format(repos_path)
# Get minimum unpacked revision, Subversion >= 1.6
min_unpacked_rev = 0
min_unpacked_rev_path = os.path.join(repos_path, 'db', 'min-unpacked-rev')
if os.path.exists(min_unpacked_rev_path):
min_unpacked_rev_file = open(min_unpacked_rev_path)
try:
min_unpacked_rev = int(min_unpacked_rev_file.readline())
except ValueError, OverflowError:
sys.stderr.write("error: repository db/min-unpacked-rev does not contain a valid number.\n")
sys.stderr.flush()
sys.exit(1)
min_unpacked_rev_file.close()
if len(sys.argv) == 2 or (len(sys.argv) == 3 and sys.argv[2].startswith('target=')):
# Print repository information [and computes shard sizes [for the specified target]]
fs_format = check_fs_format(repos_path)
target_shard = fs_format[1]
if len(sys.argv) == 3:
try:
target_shard = int(sys.argv[2][7:])
except ValueError, OverflowError:
sys.stderr.write("error: target maximum files per shard ('%s') is not a valid number.\n" \
% max_files_per_shard)
sys.stderr.flush()
sys.exit(1)
revs_path = os.path.join(repos_path, 'db', 'revs')
sys.stdout.write("Current FSFS db format version ")
sys.stdout.write(fs_format[0])
if fs_format[1] > 0:
sys.stdout.write(" with sharded layout, max files per shard: ")
sys.stdout.write(str(fs_format[1]))
if min_unpacked_rev > 0:
sys.stdout.write(", packed shards: ")
sys.stdout.write(str(min_unpacked_rev / fs_format[1]))
else:
sys.stdout.write(" with linear layout")
if target_shard > 0:
sys.stdout.write(".\nList of shard sizes for max files per shard = ")
sys.stdout.write(str(target_shard))
sys.stdout.write("\n")
revs_size = compute_rev_sizes(revs_path, fs_format[1], False)
compute_shard_sizes(revs_size, target_shard)
else:
sys.stdout.write(".\n")
sys.stdout.flush()
exit(0)
max_files_per_shard = sys.argv[2]
try:
start = int(sys.argv[3])
end = int(sys.argv[4])
except IndexError:
start = 0
end = int(current_file(repos_path)[0])
# Validate the command-line arguments.
db_path = os.path.join(repos_path, 'db')
current_path = os.path.join(db_path, 'current')
if not os.path.exists(current_path):
sys.stderr.write("error: '%s' doesn't appear to be a Subversion FSFS repository.\n" \
% repos_path)
sys.stderr.flush()
sys.exit(1)
try:
max_files_per_shard = int(max_files_per_shard)
except ValueError, OverflowError:
sys.stderr.write("error: maximum files per shard ('%s') is not a valid number.\n" \
% max_files_per_shard)
sys.stderr.flush()
sys.exit(1)
if max_files_per_shard < 0:
sys.stderr.write("error: maximum files per shard ('%d') must not be negative.\n" \
% max_files_per_shard)
sys.stderr.flush()
sys.exit(1)
# Check the format of the repository.
check_repos_format(repos_path)
# Let the user know what's going on.
if max_files_per_shard > 0:
print("Converting '%s' to a sharded structure with %d files per directory" \
% (repos_path, max_files_per_shard))
if fs_format[1]:
print('(will convert to a linear structure first)')
else:
print("Converting '%s' to a linear structure" % repos_path)
# Prevent access to the repository for the duration of the conversion.
# There's no clean way to do this, but since the format of the repository
# is indeterminate, let's remove the format file while we're converting.
print('- marking the repository as invalid')
backup_fs_format(repos_path)
# First, convert to a linear scheme (this makes recovery easier because
# it's easier to reason about the behaviour on restart).
if fs_format[1] > 0:
revs_path = os.path.join(repos_path, 'db', 'revs')
if min_unpacked_rev > 0:
print('- linearising db/revs (unpacking first)')
linearise_packed_shards(revs_path, fs_format[1], min_unpacked_rev_path)
min_unpacked_rev = 0
else:
print('- linearising db/revs')
# Process unpacked shard
linearise(revs_path)
print('- linearising db/revprops')
linearise(os.path.join(repos_path, 'db', 'revprops'))
if max_files_per_shard == 0:
# We're done. Stamp the filesystem with a format 2/3/4 db/format file.
print('- marking the repository as a valid linear repository')
format_layout = '\n'
if fs_format[0] > 2:
format_layout = '\nlayout linear\n'
write_fs_format(repos_path, fs_format[0] + format_layout)
else:
print('- sharding db/revs')
shard(os.path.join(repos_path, 'db', 'revs'), max_files_per_shard,
start, end)
print('- sharding db/revprops')
shard(os.path.join(repos_path, 'db', 'revprops'), max_files_per_shard,
start, end)
# Sharded. Keep original 3/4 format or upgrade format 2 to 3.
target_format = fs_format[0]
if fs_format[0] == 2:
target_format = 3
# We're done. Stamp the filesystem with a format db/format file.
print('- marking the repository as a valid sharded repository')
write_fs_format(repos_path, target_format + '\nlayout sharded %d\n' % max_files_per_shard)
print('- done.')
sys.exit(0)
main()
if __name__ == '__main__':
raise Exception("""This script is unfinished and not ready to be used on live data.
Trust us.""")
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic