[prev in list] [next in list] [prev in thread] [next in thread]
List: ceph-commit
Subject: [ceph-commit] branch wip-logrotate updated. 99339d506dbf4ec4d95b2a5ee4787c5d59764f15
From: ceph-commit () ceph ! com (ceph-qa-suite ! git)
Date: 2015-05-31 23:46:59
Message-ID: 20150531234659.37ABE3F69A () ds3426 ! dreamservers ! com
[Download RAW message or body]
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "".
The branch, wip-logrotate has been updated
discards e45b17b1a6d50f14177602e222826b4b1a244379 (commit)
discards 5c3db762160847dd8ce4cb5835cc768a87939de1 (commit)
via 99339d506dbf4ec4d95b2a5ee4787c5d59764f15 (commit)
via a15316c3fd34ce4a8ef753ba6ac2b0aee8fa70cd (commit)
via d9959b18baaaf1904b3ba27b4651b18cb22ba2f2 (commit)
via 4a79c75daf7e6cc64459c9ce6f447fcb96726683 (commit)
via 90b22c877eea924bfeb3725ce2dff9d2028b4f9f (commit)
via 5ff2743de4bd9389ac670642973860e9af972245 (commit)
via eb6339116cad6a2e3655108cfdbf2e44c99dc68e (commit)
via bd542353f54f200dbc6025ebae21803e5b23d9d1 (commit)
via aa0ffb313c3815d9796462f1e0de4d1ecb9cee4f (commit)
via a7064a7b3d8a865dead3800a4d97bd70da6dec33 (commit)
via fed8837c9863298ab0f11bb3c24b93a665484f3b (commit)
via 8eb3255a4c5c62720cf5c749032cbeccc4f9b264 (commit)
via d9e44336308d192d93d5047178876c2d7f887d2c (commit)
via aaacfb8e4d9539f554c4850b66aec143e1ffeafd (commit)
via c7e8e544a891ac567b238427c106e65462c7db46 (commit)
via f780f1894b82e60a14aadd41dfba83e3a35c5bfc (commit)
via 938db47dc099402e3cc604efb20735b329120a97 (commit)
via 9c36a526629c7c8422e5d6f86b3d23f8ea87391a (commit)
via 13c7f97d894e7f9d166ab1eb62fdfd63a5891056 (commit)
via 38160ecfaaca114dfe2d0e605a4368ef92ec30f6 (commit)
via b10558021fb3ce14627fedcae992453e4a43d458 (commit)
via a6a4aaf3d3cc6677423ee6d2181254ff52e8b76a (commit)
via 3018f4946288520f0ca56c3a12f93f9cd7690d02 (commit)
via 236a93c1e9a4a21a96f497501b4de6621faa16ca (commit)
This update added new revisions after undoing existing revisions. That is
to say, the old revision is not a strict subset of the new revision. This
situation occurs when you --force push a change and generate a repository
containing something like this:
* -- * -- B -- O -- O -- O (e45b17b1a6d50f14177602e222826b4b1a244379)
\
N -- N -- N (99339d506dbf4ec4d95b2a5ee4787c5d59764f15)
When this happens we assume that you've already had alert emails for all
of the O revisions, and so we here report only the revisions in the N
branch from the common base, B.
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
-----------------------------------------------------------------------
Summary of changes:
machine_types/schedule_rados.sh | 7 +
suites/fs/recovery/tasks/mds-full.yaml | 2 +
suites/rados/singleton/all/divergent_priors.yaml | 17 ++
suites/rados/singleton/all/divergent_priors2.yaml | 17 ++
suites/rados/singleton/all/reg11184.yaml | 17 ++
.../basic/2-workload/rbd_api_tests.yaml | 9 +
.../client-upgrade/hammer-client-x/rbd}/% | 0
.../{basic => rbd}/0-cluster/start.yaml | 9 +-
.../{basic => rbd}/1-install/hammer-client-x.yaml | 8 +-
.../rbd/2-workload/rbd_notification_tests.yaml | 21 ++
tasks/ceph.py | 50 ++++-
tasks/cephfs/cephfs_test_case.py | 8 +
tasks/cephfs/filesystem.py | 8 +
tasks/cephfs/fuse_mount.py | 19 ++-
tasks/cephfs/mount.py | 33 ++--
tasks/cephfs/test_client_limits.py | 38 +++-
tasks/cephfs/test_client_recovery.py | 31 +++-
tasks/cephfs/test_full.py | 150 ++++++++----
tasks/divergent_priors.py | 97 +++++---
tasks/divergent_priors2.py | 203 +++++++++++++++++
tasks/logrotate.conf | 10 +-
tasks/reg11184.py | 240 ++++++++++++++++++++
22 files changed, 867 insertions(+), 127 deletions(-)
create mode 100755 machine_types/schedule_rados.sh
create mode 100644 suites/rados/singleton/all/divergent_priors.yaml
create mode 100644 suites/rados/singleton/all/divergent_priors2.yaml
create mode 100644 suites/rados/singleton/all/reg11184.yaml
create mode 100644 suites/upgrade/client-upgrade/hammer-client-x/basic/2-workload/rbd_api_tests.yaml
copy suites/{big/rados-thrash => upgrade/client-upgrade/hammer-client-x/rbd}/% \
(100%) copy suites/upgrade/client-upgrade/hammer-client-x/{basic => \
rbd}/0-cluster/start.yaml (73%) copy \
suites/upgrade/client-upgrade/hammer-client-x/{basic => \
rbd}/1-install/hammer-client-x.yaml (63%) create mode 100644 \
suites/upgrade/client-upgrade/hammer-client-x/rbd/2-workload/rbd_notification_tests.yaml
create mode 100644 tasks/divergent_priors2.py
create mode 100644 tasks/reg11184.py
diff --git a/machine_types/schedule_rados.sh b/machine_types/schedule_rados.sh
new file mode 100755
index 0000000..befb7ac
--- /dev/null
+++ b/machine_types/schedule_rados.sh
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# $1 - part
+# $2 - branch name
+# $3 - machine name
+
+teuthology-suite -v -c $2 -m $3 -k distro -s rados --subset $(echo "(($(date +%U) % \
2) * 7) + $1" | bc)/14
diff --git a/suites/fs/recovery/tasks/mds-full.yaml \
b/suites/fs/recovery/tasks/mds-full.yaml index b9c46a1..a76a849 100644
--- a/suites/fs/recovery/tasks/mds-full.yaml
+++ b/suites/fs/recovery/tasks/mds-full.yaml
@@ -4,8 +4,10 @@ overrides:
log-whitelist:
- OSD full dropping all updates
- OSD near full
+ - is full \(reached quota
conf:
osd:
+ osd mon report interval max: 5
osd objectstore: memstore
memstore device bytes: 100000000
diff --git a/suites/rados/singleton/all/divergent_priors.yaml \
b/suites/rados/singleton/all/divergent_priors.yaml new file mode 100644
index 0000000..a01dd12
--- /dev/null
+++ b/suites/rados/singleton/all/divergent_priors.yaml
@@ -0,0 +1,17 @@
+roles:
+- - mon.0
+ - osd.0
+ - osd.1
+ - osd.2
+ - client.0
+
+overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
+
+tasks:
+- install:
+- ceph:
+- divergent_priors:
diff --git a/suites/rados/singleton/all/divergent_priors2.yaml \
b/suites/rados/singleton/all/divergent_priors2.yaml new file mode 100644
index 0000000..aecbc0a
--- /dev/null
+++ b/suites/rados/singleton/all/divergent_priors2.yaml
@@ -0,0 +1,17 @@
+roles:
+- - mon.0
+ - osd.0
+ - osd.1
+ - osd.2
+ - client.0
+
+overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
+
+tasks:
+- install:
+- ceph:
+- divergent_priors2:
diff --git a/suites/rados/singleton/all/reg11184.yaml \
b/suites/rados/singleton/all/reg11184.yaml new file mode 100644
index 0000000..54361a4
--- /dev/null
+++ b/suites/rados/singleton/all/reg11184.yaml
@@ -0,0 +1,17 @@
+roles:
+- - mon.0
+ - osd.0
+ - osd.1
+ - osd.2
+ - client.0
+
+overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
+
+tasks:
+- install:
+- ceph:
+- reg11184:
diff --git a/suites/upgrade/client-upgrade/hammer-client-x/basic/2-workload/rbd_api_tests.yaml \
b/suites/upgrade/client-upgrade/hammer-client-x/basic/2-workload/rbd_api_tests.yaml \
new file mode 100644 index 0000000..0787fb1
--- /dev/null
+++ b/suites/upgrade/client-upgrade/hammer-client-x/basic/2-workload/rbd_api_tests.yaml
@@ -0,0 +1,9 @@
+tasks:
+- workunit:
+ branch: hammer
+ clients:
+ client.0:
+ - rbd/test_librbd_api.sh
+ env:
+ RBD_FEATURES: "13"
+- print: "**** done rbd/test_librbd_api.sh"
diff --git a/suites/big/rados-thrash/% \
b/suites/upgrade/client-upgrade/hammer-client-x/rbd/% similarity index 100%
copy from suites/big/rados-thrash/%
copy to suites/upgrade/client-upgrade/hammer-client-x/rbd/%
diff --git a/suites/upgrade/client-upgrade/hammer-client-x/basic/0-cluster/start.yaml \
b/suites/upgrade/client-upgrade/hammer-client-x/rbd/0-cluster/start.yaml similarity \
index 73% copy from suites/upgrade/client-upgrade/hammer-client-x/basic/0-cluster/start.yaml
copy to suites/upgrade/client-upgrade/hammer-client-x/rbd/0-cluster/start.yaml
index db6f5e2..0cdce92 100644
--- a/suites/upgrade/client-upgrade/hammer-client-x/basic/0-cluster/start.yaml
+++ b/suites/upgrade/client-upgrade/hammer-client-x/rbd/0-cluster/start.yaml
@@ -1,13 +1,12 @@
roles:
- - mon.a
- - mds.a
+ - mon.b
+ - mon.c
- osd.0
- osd.1
-- - mon.b
- - mon.c
- osd.2
- - osd.3
-- - client.0
+ - client.0
+- - client.1
overrides:
ceph:
log-whitelist:
diff --git a/suites/upgrade/client-upgrade/hammer-client-x/basic/1-install/hammer-client-x.yaml \
b/suites/upgrade/client-upgrade/hammer-client-x/rbd/1-install/hammer-client-x.yaml \
similarity index 63% copy from \
suites/upgrade/client-upgrade/hammer-client-x/basic/1-install/hammer-client-x.yaml \
copy to suites/upgrade/client-upgrade/hammer-client-x/rbd/1-install/hammer-client-x.yaml
index c6dd4ed..b261e54 100644
--- a/suites/upgrade/client-upgrade/hammer-client-x/basic/1-install/hammer-client-x.yaml
+++ b/suites/upgrade/client-upgrade/hammer-client-x/rbd/1-install/hammer-client-x.yaml
@@ -1,10 +1,10 @@
tasks:
- install:
- branch: hammer
+ branch: hammer
- print: "**** done install hammer"
- install.upgrade:
exclude_packages: ['ceph-test', 'ceph-test-dbg']
- client.0:
-- print: "**** done install.upgrade client.0"
-- ceph:
+ client.1:
+- print: "**** done install.upgrade client.1"
+- ceph:
- print: "**** done ceph"
diff --git a/suites/upgrade/client-upgrade/hammer-client-x/rbd/2-workload/rbd_notification_tests.yaml \
b/suites/upgrade/client-upgrade/hammer-client-x/rbd/2-workload/rbd_notification_tests.yaml
new file mode 100644
index 0000000..984dfa0
--- /dev/null
+++ b/suites/upgrade/client-upgrade/hammer-client-x/rbd/2-workload/rbd_notification_tests.yaml
@@ -0,0 +1,21 @@
+tasks:
+- workunit:
+ branch: hammer
+ clients:
+ client.0:
+ - rbd/notify_master.sh
+ client.1:
+ - rbd/notify_slave.sh
+ env:
+ RBD_FEATURES: "13"
+- print: "**** done rbd: old librbd -> new librbd"
+- workunit:
+ branch: hammer
+ clients:
+ client.0:
+ - rbd/notify_slave.sh
+ client.1:
+ - rbd/notify_master.sh
+ env:
+ RBD_FEATURES: "13"
+- print: "**** done rbd: new librbd -> old librbd"
diff --git a/tasks/ceph.py b/tasks/ceph.py
index 2feeae9..3b9a353 100644
--- a/tasks/ceph.py
+++ b/tasks/ceph.py
@@ -94,27 +94,55 @@ def ceph_log(ctx, config):
self.stopping = True
self.thread.get()
- def write_rotate_conf(ctx):
+ def write_rotate_conf(ctx, daemons):
+ testdir = teuthology.get_testdir(ctx)
rotate_conf_path = os.path.join(os.path.dirname(__file__), 'logrotate.conf')
- conf = file(rotate_conf_path, 'rb').read() # does this leak an fd or \
anything?
- for remote in ctx.cluster.iteritems():
- teuthology.write_file(remote=remote,
- path='/etc/logrotate.d/ceph-test.conf',
- data=conf
- )
-
- if ctx.config.get('mds-log-rotate'):
- write_rotate_conf(ctx)
+ with file(rotate_conf_path, 'rb') as f:
+ conf = ""
+ for daemon,size in daemons.iteritems():
+ log.info('writing logrotate stanza for \
{daemon}'.format(daemon=daemon)) + conf += \
f.read().format(daemon_type=daemon,max_size=size) + f.seek(0, 0)
+
+ for remote in ctx.cluster.remotes.iterkeys():
+ teuthology.write_file(remote=remote,
+ \
path='{tdir}/logrotate.ceph-test.conf'.format(tdir=testdir), + \
data=StringIO(conf) + )
+ remote.run(
+ args=[
+ 'sudo',
+ 'mv',
+ '{tdir}/logrotate.ceph-test.conf'.format(tdir=testdir),
+ '/etc/logrotate.d/ceph-test.conf',
+ run.Raw('&&'),
+ 'sudo',
+ 'chmod',
+ '0644',
+ '/etc/logrotate.d/ceph-test.conf',
+ run.Raw('&&'),
+ 'sudo',
+ 'chown',
+ 'root.root',
+ '/etc/logrotate.d/ceph-test.conf'
+ ]
+ )
+
+ if ctx.config.get('log-rotate'):
+ daemons = ctx.config.get('log-rotate')
+ log.info('Setting up log rotation with ' + str(daemons))
+ write_rotate_conf(ctx, daemons)
logrotater = Rotater()
logrotater.begin()
try:
yield
finally:
- if ctx.config.get('mds-log-rotate'):
+ if ctx.config.get('log-rotate'):
+ log.info('Shutting down logrotate')
logrotater.end()
ctx.cluster.run(
- args=['rm', '/etc/logrotate.d/ceph-test.conf'
+ args=['sudo', 'rm', '/etc/logrotate.d/ceph-test.conf'
]
)
if ctx.archive is not None and \
diff --git a/tasks/cephfs/cephfs_test_case.py b/tasks/cephfs/cephfs_test_case.py
index b146154..23a4653 100644
--- a/tasks/cephfs/cephfs_test_case.py
+++ b/tasks/cephfs/cephfs_test_case.py
@@ -1,3 +1,4 @@
+import json
import logging
import unittest
from unittest import case
@@ -96,6 +97,13 @@ class CephFSTestCase(unittest.TestCase):
self.wait_until_true(lambda: not self.fs.is_full(),
timeout=osd_mon_report_interval_max * 5)
+ # In case anything is in the OSD blacklist list, clear it out. This is to \
avoid + # the OSD map changing in the background (due to blacklist expiry) \
while tests run. + blacklist = \
json.loads(self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json-pretty"))['flags'] + log.info("Removing {0} blacklist \
entries".format(len(blacklist))) + for addr, blacklisted_at in \
blacklist.items(): + self.fs.mon_manager.raw_cluster_cmd("osd", \
"blacklist", "rm", addr) +
self.fs.mds_restart()
self.fs.wait_for_daemons()
if not self.mount_a.is_mounted():
diff --git a/tasks/cephfs/filesystem.py b/tasks/cephfs/filesystem.py
index 6367203..bf3a739 100644
--- a/tasks/cephfs/filesystem.py
+++ b/tasks/cephfs/filesystem.py
@@ -422,6 +422,14 @@ class Filesystem(object):
flags = json.loads(self.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json-pretty"))['flags'] return 'full' in flags
+ def is_pool_full(self, pool_name):
+ pools = json.loads(self.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json-pretty"))['pools'] + for pool in pools:
+ if pool['pool_name'] == pool_name:
+ return 'full' in pool['flags_names'].split(",")
+
+ raise RuntimeError("Pool not found '{0}'".format(pool_name))
+
def wait_for_state(self, goal_state, reject=None, timeout=None, mds_id=None):
"""
Block until the MDS reaches a particular state, or a failure condition
diff --git a/tasks/cephfs/fuse_mount.py b/tasks/cephfs/fuse_mount.py
index c0fbcde..5517f5f 100644
--- a/tasks/cephfs/fuse_mount.py
+++ b/tasks/cephfs/fuse_mount.py
@@ -68,10 +68,17 @@ class FuseMount(CephFSMount):
run_cmd.extend(run_cmd_tail)
def list_connections():
+ self.client_remote.run(
+ args=["sudo", "mount", "-t", "fusectl", "/sys/fs/fuse/connections", \
"/sys/fs/fuse/connections"], + check_status=False
+ )
p = self.client_remote.run(
args=["ls", "/sys/fs/fuse/connections"],
- stdout=StringIO()
+ stdout=StringIO(),
+ check_status=False
)
+ if p.exitstatus != 0:
+ return []
ls_str = p.stdout.getvalue().strip()
if ls_str:
@@ -340,3 +347,13 @@ print find_socket("{client_name}")
"""
status = self._admin_socket(['status'])
return status['osd_epoch'], status['osd_epoch_barrier']
+
+ def get_dentry_count(self):
+ """
+ Return 2-tuple of dentry_count, dentry_pinned_count
+ """
+ status = self._admin_socket(['status'])
+ return status['dentry_count'], status['dentry_pinned_count']
+
+ def set_cache_size(self, size):
+ return self._admin_socket(['config', 'set', 'client_cache_size', str(size)])
diff --git a/tasks/cephfs/mount.py b/tasks/cephfs/mount.py
index 1d42120..cb8c0cc 100644
--- a/tasks/cephfs/mount.py
+++ b/tasks/cephfs/mount.py
@@ -167,7 +167,7 @@ class CephFSMount(object):
raise RuntimeError("Timed out after {0}s waiting for {1} to become visible \
from {2}".format( i, basename, self.client_id))
- def lock_background(self, basename="background_file"):
+ def lock_background(self, basename="background_file", do_flock=True):
"""
Open and lock a files for writing, hold the lock in a background process
"""
@@ -175,36 +175,40 @@ class CephFSMount(object):
path = os.path.join(self.mountpoint, basename)
- pyscript = dedent("""
+ script_builder = """
import time
import fcntl
- import struct
-
+ import struct"""
+ if do_flock:
+ script_builder += """
f1 = open("{path}-1", 'w')
- fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB)
-
+ fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB)"""
+ script_builder += """
f2 = open("{path}-2", 'w')
lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0)
fcntl.fcntl(f2, fcntl.F_SETLK, lockdata)
while True:
time.sleep(1)
- """).format(path=path)
+ """
+
+ pyscript = dedent(script_builder).format(path=path)
log.info("lock file {0}".format(basename))
rproc = self._run_python(pyscript)
self.background_procs.append(rproc)
return rproc
- def check_filelock(self, basename="background_file"):
+ def check_filelock(self, basename="background_file", do_flock=True):
assert(self.is_mounted())
path = os.path.join(self.mountpoint, basename)
- pyscript = dedent("""
+ script_builder = """
import fcntl
import errno
- import struct
-
+ import struct"""
+ if do_flock:
+ script_builder += """
f1 = open("{path}-1", 'r')
try:
fcntl.flock(f1, fcntl.LOCK_EX | fcntl.LOCK_NB)
@@ -212,8 +216,8 @@ class CephFSMount(object):
if e.errno == errno.EAGAIN:
pass
else:
- raise RuntimeError("flock on file {path}-1 not found")
-
+ raise RuntimeError("flock on file {path}-1 not found")"""
+ script_builder += """
f2 = open("{path}-2", 'r')
try:
lockdata = struct.pack('hhllhh', fcntl.F_WRLCK, 0, 0, 0, 0, 0)
@@ -223,7 +227,8 @@ class CephFSMount(object):
pass
else:
raise RuntimeError("posix lock on file {path}-2 not found")
- """).format(path=path)
+ """
+ pyscript = dedent(script_builder).format(path=path)
log.info("check lock on file {0}".format(basename))
self.client_remote.run(args=[
diff --git a/tasks/cephfs/test_client_limits.py b/tasks/cephfs/test_client_limits.py
index decc6a8..f415ed8 100644
--- a/tasks/cephfs/test_client_limits.py
+++ b/tasks/cephfs/test_client_limits.py
@@ -5,11 +5,13 @@ exceed the limits of how many caps/inodes they should hold.
"""
import logging
+from textwrap import dedent
from unittest import SkipTest
from teuthology.orchestra.run import CommandFailedError
from tasks.cephfs.cephfs_test_case import CephFSTestCase
-
from tasks.cephfs.fuse_mount import FuseMount
+import os
+import time
log = logging.getLogger(__name__)
@@ -169,3 +171,37 @@ class TestClientLimits(CephFSTestCase):
# Wait for the health warnings. Assume mds can handle 10 request per second \
at least
self.wait_for_health("failing to advance its oldest_client_tid", \
max_requests / 10) +
+ def test_client_cache_size(self):
+ """
+ check if client invalidate kernel dcache according to its cache size config
+ """
+
+ # The debug hook to inject the failure only exists in the fuse client
+ if not isinstance(self.mount_a, FuseMount):
+ raise SkipTest("Require FUSE client to inject client release failure")
+
+ dir_path = os.path.join(self.mount_a.mountpoint, "testdir")
+
+ mkdir_script = dedent("""
+ import os
+ os.mkdir("{path}")
+ for n in range(0, {num_dirs}):
+ os.mkdir("{path}/dir{{0}}".format(n))
+ """);
+
+ num_dirs = 1000
+ self.mount_a.run_python(mkdir_script.format(path=dir_path, \
num_dirs=num_dirs)) + self.mount_a.run_shell(["sync"])
+
+ dentry_count, dentry_pinned_count = self.mount_a.get_dentry_count();
+ self.assertGreaterEqual(dentry_count, num_dirs);
+ self.assertGreaterEqual(dentry_pinned_count, num_dirs);
+
+ cache_size = num_dirs / 10;
+ self.mount_a.set_cache_size(cache_size);
+ time.sleep(30);
+
+ dentry_count, dentry_pinned_count = self.mount_a.get_dentry_count();
+ self.assertLessEqual(dentry_count, cache_size);
+ self.assertLessEqual(dentry_pinned_count, cache_size);
diff --git a/tasks/cephfs/test_client_recovery.py \
b/tasks/cephfs/test_client_recovery.py index adfc471..1c45102 100644
--- a/tasks/cephfs/test_client_recovery.py
+++ b/tasks/cephfs/test_client_recovery.py
@@ -5,9 +5,12 @@ Teuthology task for exercising CephFS client recovery
import logging
import time
+import distutils.version as version
+import re
from teuthology.orchestra.run import CommandFailedError, ConnectionLostError
from tasks.cephfs.cephfs_test_case import CephFSTestCase
+from teuthology.packaging import get_package_version
log = logging.getLogger(__name__)
@@ -307,19 +310,41 @@ class TestClientRecovery(CephFSTestCase):
self.assertLess(recovery_time, self.ms_max_backoff * 2)
self.assert_session_state(client_id, "open")
+
def test_filelock(self):
"""
Check that file lock doesn't get lost after an MDS restart
"""
- lock_holder = self.mount_a.lock_background()
+ a_version_str = get_package_version(self.mount_a.client_remote, "fuse")
+ b_version_str = get_package_version(self.mount_b.client_remote, "fuse")
+ flock_version_str = "2.9"
+
+ version_regex = re.compile(r"[0-9\.]+")
+ a_result = version_regex.match(a_version_str)
+ self.assertTrue(a_result)
+ b_result = version_regex.match(b_version_str)
+ self.assertTrue(b_result)
+ a_version = version.StrictVersion(a_result.group())
+ b_version = version.StrictVersion(b_result.group())
+ flock_version=version.StrictVersion(flock_version_str)
+
+ flockable = False
+ if (a_version >= flock_version and b_version >= flock_version):
+ log.info("testing flock locks")
+ flockable = True
+ else:
+ log.info("not testing flock locks, machines have versions {av} and \
{bv}".format( + av=a_version_str,bv=b_version_str))
+
+ lock_holder = self.mount_a.lock_background(do_flock=flockable)
self.mount_b.wait_for_visible("background_file-2")
- self.mount_b.check_filelock()
+ self.mount_b.check_filelock(do_flock=flockable)
self.fs.mds_fail_restart()
self.fs.wait_for_state('up:active', timeout=MDS_RESTART_GRACE)
- self.mount_b.check_filelock()
+ self.mount_b.check_filelock(do_flock=flockable)
# Tear down the background process
lock_holder.stdin.close()
diff --git a/tasks/cephfs/test_full.py b/tasks/cephfs/test_full.py
index ec7d993..3bba361 100644
--- a/tasks/cephfs/test_full.py
+++ b/tasks/cephfs/test_full.py
@@ -13,29 +13,38 @@ from tasks.cephfs.cephfs_test_case import CephFSTestCase
log = logging.getLogger(__name__)
-class TestClusterFull(CephFSTestCase):
- """
- Exercise the MDS and Client behaviour when the cluster fills up.
- """
- # Persist-between-tests constants
+class FullnessTestCase(CephFSTestCase):
+ CLIENTS_REQUIRED = 2
+
+ # Subclasses define whether they're filling whole cluster or just data pool
+ data_only = False
+
+ # Subclasses define how many bytes should be written to achieve fullness
pool_capacity = None
+ fill_mb = None
- CLIENTS_REQUIRED = 2
+ # Subclasses define what fullness means to them
+ def is_full(self):
+ raise NotImplementedError()
def setUp(self):
- super(TestClusterFull, self).setUp()
+ CephFSTestCase.setUp(self)
- if self.pool_capacity is None:
- # This is a hack to overcome weird fluctuations in the reported
- # `max_avail` attribute of pools that sometimes occurs in between
- # tests (reason as yet unclear, but this dodges the issue)
- TestClusterFull.pool_capacity = \
self.fs.get_pool_df(self._data_pool_name())['max_avail'] + # These tests just \
use a single active MDS throughout, so remember its ID + # for use in mds_asok \
calls + self.active_mds_id = self.fs.get_active_names()[0]
- objectstore = self.fs.get_config("osd_objectstore", "osd")
- if objectstore != "memstore":
- # You certainly *could* run this on a real OSD, but you don't want to \
sit
- # here for hours waiting for the test to fill up a 1TB drive!
- raise case.SkipTest("Require `memstore` OSD backend to simulate full \
drives") + # Capture the initial OSD map epoch for later use
+ self.initial_osd_epoch = json.loads(
+ self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json").strip() + )['epoch']
+
+ # Check the initial barrier epoch on the MDS: this should be
+ # set to the latest map at MDS startup. We do this check in
+ # setUp to get in there before subclasses might touch things
+ # in their own setUp functions.
+ self.assertGreaterEqual(self.fs.mds_asok(["status"], \
mds_id=self.active_mds_id)['osdmap_epoch_barrier'], + \
self.initial_osd_epoch)
def test_barrier(self):
"""
@@ -44,13 +53,6 @@ class TestClusterFull(CephFSTestCase):
epoch.
"""
- # Check the initial barrier epoch on the MDS: this should be
- # set to the latest map at MDS startup
- initial_osd_epoch = json.loads(
- self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json").strip()
- )['epoch']
- self.assertGreaterEqual(self.fs.mds_asok(["status"])['osdmap_epoch_barrier'], \
initial_osd_epoch)
-
# Sync up clients with initial MDS OSD map barrier
self.mount_a.open_no_data("foo")
self.mount_b.open_no_data("bar")
@@ -60,8 +62,8 @@ class TestClusterFull(CephFSTestCase):
mount_a_initial_epoch = self.mount_a.get_osd_epoch()[0]
# Freshly mounted at start of test, should be up to date with OSD map
- self.assertGreaterEqual(mount_a_initial_epoch, initial_osd_epoch)
- self.assertGreaterEqual(self.mount_b.get_osd_epoch()[0], initial_osd_epoch)
+ self.assertGreaterEqual(mount_a_initial_epoch, self.initial_osd_epoch)
+ self.assertGreaterEqual(self.mount_b.get_osd_epoch()[0], \
self.initial_osd_epoch)
# Set and unset a flag to cause OSD epoch to increment
self.fs.mon_manager.raw_cluster_cmd("osd", "set", "pause")
@@ -69,7 +71,7 @@ class TestClusterFull(CephFSTestCase):
out = self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json").strip() new_epoch = json.loads(out)['epoch']
- self.assertNotEqual(initial_osd_epoch, new_epoch)
+ self.assertNotEqual(self.initial_osd_epoch, new_epoch)
# Do a metadata operation on client A, witness that it ends up with
# the old OSD map from startup time (nothing has prompted it
@@ -85,7 +87,7 @@ class TestClusterFull(CephFSTestCase):
self.assertEqual(mount_a_epoch, mount_a_initial_epoch)
# Set a barrier on the MDS
- self.fs.mds_asok(["osdmap", "barrier", new_epoch.__str__()])
+ self.fs.mds_asok(["osdmap", "barrier", new_epoch.__str__()], \
mds_id=self.active_mds_id)
# Do an operation on client B, witness that it ends up with
# the latest OSD map from the barrier
@@ -127,29 +129,25 @@ class TestClusterFull(CephFSTestCase):
"""
osd_mon_report_interval_max = \
int(self.fs.get_config("osd_mon_report_interval_max", \
service_type='osd'))
- mon_osd_full_ratio = float(self.fs.get_config("mon_osd_full_ratio"))
-
- pool_capacity = self.pool_capacity
- fill_mb = int(1.05 * mon_osd_full_ratio * (pool_capacity / (1024.0 * \
1024.0))) + 2
- log.info("Writing {0}MB should fill this cluster".format(fill_mb))
+ log.info("Writing {0}MB should fill this cluster".format(self.fill_mb))
# Fill up the cluster. This dd may or may not fail, as it depends on
# how soon the cluster recognises its own fullness
- self.mount_a.write_n_mb("large_file_a", fill_mb / 2)
+ self.mount_a.write_n_mb("large_file_a", self.fill_mb / 2)
try:
- self.mount_a.write_n_mb("large_file_b", fill_mb / 2)
+ self.mount_a.write_n_mb("large_file_b", self.fill_mb / 2)
except CommandFailedError:
log.info("Writing file B failed (full status happened already)")
- assert self.fs.is_full()
+ assert self.is_full()
else:
log.info("Writing file B succeeded (full status will happen soon)")
- self.wait_until_true(lambda: self.fs.is_full(),
+ self.wait_until_true(lambda: self.is_full(),
timeout=osd_mon_report_interval_max * 5)
# Attempting to write more data should give me ENOSPC
with self.assertRaises(CommandFailedError) as ar:
- self.mount_a.write_n_mb("large_file_b", 50, seek=fill_mb / 2)
+ self.mount_a.write_n_mb("large_file_b", 50, seek=self.fill_mb / 2)
self.assertEqual(ar.exception.exitstatus, 1) # dd returns 1 on "No space"
# Wait for the MDS to see the latest OSD map so that it will reliably
@@ -157,11 +155,12 @@ class TestClusterFull(CephFSTestCase):
# while in the full state.
osd_epoch = json.loads(self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json-pretty"))['epoch'] self.wait_until_true(
- lambda: self.fs.mds_asok(['status'])['osdmap_epoch'] >= osd_epoch,
+ lambda: self.fs.mds_asok(['status'], \
mds_id=self.active_mds_id)['osdmap_epoch'] >= osd_epoch, timeout=10)
- with self.assertRaises(CommandFailedError):
- self.mount_a.write_n_mb("small_file_1", 0)
+ if not self.data_only:
+ with self.assertRaises(CommandFailedError):
+ self.mount_a.write_n_mb("small_file_1", 0)
# Clear out some space
if easy_case:
@@ -178,14 +177,14 @@ class TestClusterFull(CephFSTestCase):
# Here we are waiting for two things to happen:
# * The MDS to purge the stray folder and execute object deletions
# * The OSDs to inform the mon that they are no longer full
- self.wait_until_true(lambda: not self.fs.is_full(),
+ self.wait_until_true(lambda: not self.is_full(),
timeout=osd_mon_report_interval_max * 5)
# Wait for the MDS to see the latest OSD map so that it will reliably
# be applying the free space policy
osd_epoch = json.loads(self.fs.mon_manager.raw_cluster_cmd("osd", "dump", \
"--format=json-pretty"))['epoch'] self.wait_until_true(
- lambda: self.fs.mds_asok(['status'])['osdmap_epoch'] >= osd_epoch,
+ lambda: self.fs.mds_asok(['status'], \
mds_id=self.active_mds_id)['osdmap_epoch'] >= osd_epoch, timeout=10)
# Now I should be able to write again
@@ -208,15 +207,16 @@ class TestClusterFull(CephFSTestCase):
# Enough to trip the full flag
osd_mon_report_interval_max = \
int(self.fs.get_config("osd_mon_report_interval_max", \
service_type='osd'))
- mon_osd_full_ratio = float(self.fs.get_config("mon_osd_full_ratio"))
- pool_capacity = self.pool_capacity
+ mon_tick_interval = int(self.fs.get_config("mon_tick_interval", \
service_type="mon"))
# Sufficient data to cause RADOS cluster to go 'full'
- fill_mb = int(1.05 * mon_osd_full_ratio * (pool_capacity / (1024.0 * \
1024.0)))
- log.info("pool capacity {0}, {1}MB should be enough to fill \
it".format(pool_capacity, fill_mb)) + log.info("pool capacity {0}, {1}MB \
should be enough to fill it".format(self.pool_capacity, self.fill_mb))
# Long enough for RADOS cluster to notice it is full and set flag on mons
- full_wait = osd_mon_report_interval_max * 1.5
+ # (report_interval for mon to learn PG stats, tick interval for it to update \
OSD map, + # factor of 1.5 for I/O + network latency in committing OSD map \
and distributing it + # to the OSDs)
+ full_wait = (osd_mon_report_interval_max + mon_tick_interval) * 1.5
# Configs for this test should bring this setting down in order to
# run reasonably quickly
@@ -225,7 +225,7 @@ class TestClusterFull(CephFSTestCase):
"osd_mon_report_interval_max (5 is a good setting)!")
self.mount_a.run_python(template.format(
- fill_mb=fill_mb,
+ fill_mb=self.fill_mb,
file_path=file_path,
full_wait=full_wait
))
@@ -339,3 +339,55 @@ class TestClusterFull(CephFSTestCase):
""")
self._remote_write_test(remote_script)
+
+
+class TestQuotaFull(FullnessTestCase):
+ """
+ Test per-pool fullness, which indicates quota limits exceeded
+ """
+ pool_capacity = 1024 * 1024 * 32 # arbitrary low-ish limit
+ fill_mb = pool_capacity / (1024 * 1024)
+
+ # We are only testing quota handling on the data pool, not the metadata
+ # pool.
+ data_only = True
+
+ def setUp(self):
+ super(TestQuotaFull, self).setUp()
+
+ pool_name = self.fs.get_data_pool_name()
+ self.fs.mon_manager.raw_cluster_cmd("osd", "pool", "set-quota", pool_name,
+ "max_bytes", \
"{0}".format(self.pool_capacity)) +
+ def is_full(self):
+ return self.fs.is_pool_full(self.fs.get_data_pool_name())
+
+
+class TestClusterFull(FullnessTestCase):
+ """
+ Test cluster-wide fullness, which indicates that an OSD has become too full
+ """
+ pool_capacity = None
+
+ def setUp(self):
+ super(TestClusterFull, self).setUp()
+
+ if self.pool_capacity is None:
+ # This is a hack to overcome weird fluctuations in the reported
+ # `max_avail` attribute of pools that sometimes occurs in between
+ # tests (reason as yet unclear, but this dodges the issue)
+ TestClusterFull.pool_capacity = \
self.fs.get_pool_df(self._data_pool_name())['max_avail'] + \
mon_osd_full_ratio = float(self.fs.get_config("mon_osd_full_ratio")) + \
TestClusterFull.fill_mb = int(1.05 * mon_osd_full_ratio * (self.pool_capacity / \
(1024.0 * 1024.0))) +
+ objectstore = self.fs.get_config("osd_objectstore", "osd")
+ if objectstore != "memstore":
+ # You certainly *could* run this on a real OSD, but you don't want to \
sit + # here for hours waiting for the test to fill up a 1TB drive!
+ raise case.SkipTest("Require `memstore` OSD backend to simulate full \
drives") +
+ def is_full(self):
+ return self.fs.is_full()
+
+# Hide the parent class so that unittest.loader doesn't try to run it.
+del globals()['FullnessTestCase']
diff --git a/tasks/divergent_priors.py b/tasks/divergent_priors.py
index e5bcc8f..1fff21b 100644
--- a/tasks/divergent_priors.py
+++ b/tasks/divergent_priors.py
@@ -10,14 +10,19 @@ from util.rados import rados
log = logging.getLogger(__name__)
+
def task(ctx, config):
"""
Test handling of divergent entries with prior_version
prior to log_tail
- config: none
+ overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
- Requires 3 osds.
+ Requires 3 osds on a single test node.
"""
if config is None:
config = {}
@@ -44,19 +49,21 @@ def task(ctx, config):
osds = [0, 1, 2]
for i in osds:
- ctx.manager.set_config(i, osd_min_pg_log_entries=1)
+ ctx.manager.set_config(i, osd_min_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_max_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_pg_log_trim_min=5)
# determine primary
divergent = ctx.manager.get_pg_primary('foo', 0)
log.info("primary and soon to be divergent is %d", divergent)
- non_divergent = [0,1,2]
+ non_divergent = list(osds)
non_divergent.remove(divergent)
log.info('writing initial objects')
first_mon = teuthology.get_first_mon(ctx, config)
(mon,) = ctx.cluster.only(first_mon).remotes.iterkeys()
- # write 1000 objects
- for i in range(1000):
+ # write 100 objects
+ for i in range(100):
rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i, dummyfile])
ctx.manager.wait_for_clean()
@@ -64,26 +71,33 @@ def task(ctx, config):
# blackhole non_divergent
log.info("blackholing osds %s", str(non_divergent))
for i in non_divergent:
- ctx.manager.set_config(i, filestore_blackhole='')
-
- # write 1 (divergent) object
- log.info('writing divergent object existing_0')
- rados(
- ctx, mon, ['-p', 'foo', 'put', 'existing_0', dummyfile2],
- wait=False)
+ ctx.manager.set_config(i, filestore_blackhole=1)
+
+ DIVERGENT_WRITE = 5
+ DIVERGENT_REMOVE = 5
+ # Write some soon to be divergent
+ log.info('writing divergent objects')
+ for i in range(DIVERGENT_WRITE):
+ rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i,
+ dummyfile2], wait=False)
+ # Remove some soon to be divergent
+ log.info('remove divergent objects')
+ for i in range(DIVERGENT_REMOVE):
+ rados(ctx, mon, ['-p', 'foo', 'rm',
+ 'existing_%d' % (i + DIVERGENT_WRITE)], wait=False)
time.sleep(10)
mon.run(
args=['killall', '-9', 'rados'],
wait=True,
check_status=False)
- # kill all the osds
+ # kill all the osds but leave divergent in
log.info('killing all the osds')
for i in osds:
ctx.manager.kill_osd(i)
for i in osds:
ctx.manager.mark_down_osd(i)
- for i in osds:
+ for i in non_divergent:
ctx.manager.mark_out_osd(i)
# bring up non-divergent
@@ -93,48 +107,61 @@ def task(ctx, config):
for i in non_divergent:
ctx.manager.mark_in_osd(i)
- log.info('making log long to prevent backfill')
- for i in non_divergent:
- ctx.manager.set_config(i, osd_min_pg_log_entries=100000)
-
# write 1 non-divergent object (ensure that old divergent one is divergent)
- log.info('writing non-divergent object existing_1')
- rados(ctx, mon, ['-p', 'foo', 'put', 'existing_1', dummyfile2])
+ objname = "existing_%d" % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ log.info('writing non-divergent object ' + objname)
+ rados(ctx, mon, ['-p', 'foo', 'put', objname, dummyfile2])
ctx.manager.wait_for_recovery()
- # ensure no recovery
+ # ensure no recovery of up osds first
log.info('delay recovery')
for i in non_divergent:
- ctx.manager.set_config(i, osd_recovery_delay_start=100000)
+ ctx.manager.wait_run_admin_socket(
+ 'osd', i, ['set_recovery_delay', '100000'])
# bring in our divergent friend
log.info("revive divergent %d", divergent)
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noup')
ctx.manager.revive_osd(divergent)
+ log.info('delay recovery divergent')
+ ctx.manager.wait_run_admin_socket(
+ 'osd', divergent, ['set_recovery_delay', '100000'])
+
+ ctx.manager.raw_cluster_cmd('osd', 'unset', 'noup')
while len(ctx.manager.get_osd_status()['up']) < 3:
time.sleep(10)
- log.info('delay recovery divergent')
- ctx.manager.set_config(divergent, osd_recovery_delay_start=100000)
- log.info('mark divergent in')
- ctx.manager.mark_in_osd(divergent)
-
log.info('wait for peering')
rados(ctx, mon, ['-p', 'foo', 'put', 'foo', dummyfile])
+ # At this point the divergent_priors should have been detected
+
log.info("killing divergent %d", divergent)
ctx.manager.kill_osd(divergent)
log.info("reviving divergent %d", divergent)
ctx.manager.revive_osd(divergent)
log.info('allowing recovery')
- for i in non_divergent:
- ctx.manager.set_config(i, osd_recovery_delay_start=0)
+ # Set osd_recovery_delay_start back to 0 and kick the queue
+ for i in osds:
+ ctx.manager.raw_cluster_cmd('tell', 'osd.%d' % i, 'debug',
+ 'kick_recovery_wq', ' 0')
+
+ log.info('reading divergent objects')
+ for i in range(DIVERGENT_WRITE + DIVERGENT_REMOVE):
+ exit_status = rados(ctx, mon, ['-p', 'foo', 'get', 'existing_%d' % i,
+ '-o', '/tmp/existing'])
+ assert exit_status is 0
+
+ (remote,) = ctx.\
+ cluster.only('osd.{o}'.format(o=divergent)).remotes.iterkeys()
+ msg = "dirty_divergent_priors: true, divergent_priors: %d" \
+ % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ cmd = 'grep "{msg}" /var/log/ceph/ceph-osd.{osd}.log'\
+ .format(msg=msg, osd=divergent)
+ proc = remote.run(args=cmd, wait=True, check_status=False)
+ assert proc.exitstatus == 0
- log.info('reading existing_0')
- exit_status = rados(ctx, mon,
- ['-p', 'foo', 'get', 'existing_0',
- '-o', '/tmp/existing'])
- assert exit_status is 0
log.info("success")
diff --git a/tasks/divergent_priors2.py b/tasks/divergent_priors2.py
new file mode 100644
index 0000000..991ddca
--- /dev/null
+++ b/tasks/divergent_priors2.py
@@ -0,0 +1,203 @@
+"""
+Special case divergence test with ceph-objectstore-tool export/remove/import
+"""
+import logging
+import time
+from cStringIO import StringIO
+
+from teuthology import misc as teuthology
+from util.rados import rados
+import os
+
+
+log = logging.getLogger(__name__)
+
+
+def task(ctx, config):
+ """
+ Test handling of divergent entries with prior_version
+ prior to log_tail and a ceph-objectstore-tool export/import
+
+ overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
+
+ Requires 3 osds on a single test node.
+ """
+ if config is None:
+ config = {}
+ assert isinstance(config, dict), \
+ 'divergent_priors task only accepts a dict for configuration'
+
+ while len(ctx.manager.get_osd_status()['up']) < 3:
+ time.sleep(10)
+ ctx.manager.raw_cluster_cmd('tell', 'osd.0', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('tell', 'osd.1', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('tell', 'osd.2', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noout')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noin')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'nodown')
+ ctx.manager.wait_for_clean()
+
+ # something that is always there
+ dummyfile = '/etc/fstab'
+ dummyfile2 = '/etc/resolv.conf'
+ testdir = teuthology.get_testdir(ctx)
+
+ # create 1 pg pool
+ log.info('creating foo')
+ ctx.manager.raw_cluster_cmd('osd', 'pool', 'create', 'foo', '1')
+
+ osds = [0, 1, 2]
+ for i in osds:
+ ctx.manager.set_config(i, osd_min_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_max_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_pg_log_trim_min=5)
+
+ # determine primary
+ divergent = ctx.manager.get_pg_primary('foo', 0)
+ log.info("primary and soon to be divergent is %d", divergent)
+ non_divergent = list(osds)
+ non_divergent.remove(divergent)
+
+ log.info('writing initial objects')
+ first_mon = teuthology.get_first_mon(ctx, config)
+ (mon,) = ctx.cluster.only(first_mon).remotes.iterkeys()
+ # write 100 objects
+ for i in range(100):
+ rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i, dummyfile])
+
+ ctx.manager.wait_for_clean()
+
+ # blackhole non_divergent
+ log.info("blackholing osds %s", str(non_divergent))
+ for i in non_divergent:
+ ctx.manager.set_config(i, filestore_blackhole=1)
+
+ DIVERGENT_WRITE = 5
+ DIVERGENT_REMOVE = 5
+ # Write some soon to be divergent
+ log.info('writing divergent objects')
+ for i in range(DIVERGENT_WRITE):
+ rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i,
+ dummyfile2], wait=False)
+ # Remove some soon to be divergent
+ log.info('remove divergent objects')
+ for i in range(DIVERGENT_REMOVE):
+ rados(ctx, mon, ['-p', 'foo', 'rm',
+ 'existing_%d' % (i + DIVERGENT_WRITE)], wait=False)
+ time.sleep(10)
+ mon.run(
+ args=['killall', '-9', 'rados'],
+ wait=True,
+ check_status=False)
+
+ # kill all the osds but leave divergent in
+ log.info('killing all the osds')
+ for i in osds:
+ ctx.manager.kill_osd(i)
+ for i in osds:
+ ctx.manager.mark_down_osd(i)
+ for i in non_divergent:
+ ctx.manager.mark_out_osd(i)
+
+ # bring up non-divergent
+ log.info("bringing up non_divergent %s", str(non_divergent))
+ for i in non_divergent:
+ ctx.manager.revive_osd(i)
+ for i in non_divergent:
+ ctx.manager.mark_in_osd(i)
+
+ # write 1 non-divergent object (ensure that old divergent one is divergent)
+ objname = "existing_%d" % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ log.info('writing non-divergent object ' + objname)
+ rados(ctx, mon, ['-p', 'foo', 'put', objname, dummyfile2])
+
+ ctx.manager.wait_for_recovery()
+
+ # ensure no recovery of up osds first
+ log.info('delay recovery')
+ for i in non_divergent:
+ ctx.manager.wait_run_admin_socket(
+ 'osd', i, ['set_recovery_delay', '100000'])
+
+ # bring in our divergent friend
+ log.info("revive divergent %d", divergent)
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noup')
+ ctx.manager.revive_osd(divergent)
+
+ log.info('delay recovery divergent')
+ ctx.manager.wait_run_admin_socket(
+ 'osd', divergent, ['set_recovery_delay', '100000'])
+
+ ctx.manager.raw_cluster_cmd('osd', 'unset', 'noup')
+ while len(ctx.manager.get_osd_status()['up']) < 3:
+ time.sleep(10)
+
+ log.info('wait for peering')
+ rados(ctx, mon, ['-p', 'foo', 'put', 'foo', dummyfile])
+
+ # At this point the divergent_priors should have been detected
+
+ log.info("killing divergent %d", divergent)
+ ctx.manager.kill_osd(divergent)
+
+ # Export a pg
+ (exp_remote,) = ctx.\
+ cluster.only('osd.{o}'.format(o=divergent)).remotes.iterkeys()
+ FSPATH = ctx.manager.get_filepath()
+ JPATH = os.path.join(FSPATH, "journal")
+ prefix = ("sudo adjust-ulimits ceph-objectstore-tool "
+ "--data-path {fpath} --journal-path {jpath} "
+ "--log-file="
+ "/var/log/ceph/objectstore_tool.$$.log ".
+ format(fpath=FSPATH, jpath=JPATH))
+ pid = os.getpid()
+ expfile = os.path.join(testdir, "exp.{pid}.out".format(pid=pid))
+ cmd = ((prefix + "--op export --pgid 1.0 --file {file}").
+ format(id=divergent, file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ cmd = ((prefix + "--op remove --pgid 1.0").
+ format(id=divergent, file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ cmd = ((prefix + "--op import --file {file}").
+ format(id=divergent, file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ log.info("reviving divergent %d", divergent)
+ ctx.manager.revive_osd(divergent)
+
+ log.info('allowing recovery')
+ # Set osd_recovery_delay_start back to 0 and kick the queue
+ for i in osds:
+ ctx.manager.raw_cluster_cmd('tell', 'osd.%d' % i, 'debug',
+ 'kick_recovery_wq', ' 0')
+
+ log.info('reading divergent objects')
+ for i in range(DIVERGENT_WRITE + DIVERGENT_REMOVE):
+ exit_status = rados(ctx, mon, ['-p', 'foo', 'get', 'existing_%d' % i,
+ '-o', '/tmp/existing'])
+ assert exit_status is 0
+
+ (remote,) = ctx.\
+ cluster.only('osd.{o}'.format(o=divergent)).remotes.iterkeys()
+ msg = "dirty_divergent_priors: true, divergent_priors: %d" \
+ % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ cmd = 'grep "{msg}" /var/log/ceph/ceph-osd.{osd}.log'\
+ .format(msg=msg, osd=divergent)
+ proc = remote.run(args=cmd, wait=True, check_status=False)
+ assert proc.exitstatus == 0
+
+ cmd = 'rm {file}'.format(file=expfile)
+ remote.run(args=cmd, wait=True)
+ log.info("success")
diff --git a/tasks/logrotate.conf b/tasks/logrotate.conf
index cec252a..b0cb801 100644
--- a/tasks/logrotate.conf
+++ b/tasks/logrotate.conf
@@ -1,11 +1,13 @@
-/var/log/ceph/*mds*.log {
+/var/log/ceph/*{daemon_type}*.log {{
rotate 100
- size 1M
+ size {max_size}
compress
sharedscripts
postrotate
- killall ceph-mds -1
+ killall {daemon_type} -1 || true
endscript
missingok
notifempty
-}
\ No newline at end of file
+ su root root
+}}
+
diff --git a/tasks/reg11184.py b/tasks/reg11184.py
new file mode 100644
index 0000000..4bb8cae
--- /dev/null
+++ b/tasks/reg11184.py
@@ -0,0 +1,240 @@
+"""
+Special regression test for tracker #11184
+
+Synopsis: osd/SnapMapper.cc: 282: FAILED assert(check(oid))
+
+This is accomplished by moving a pg that wasn't part of split and still include
+divergent priors.
+"""
+import logging
+import time
+from cStringIO import StringIO
+
+from teuthology import misc as teuthology
+from util.rados import rados
+import os
+
+
+log = logging.getLogger(__name__)
+
+
+def task(ctx, config):
+ """
+ Test handling of divergent entries during export / import
+ to regression test tracker #11184
+
+ overrides:
+ ceph:
+ conf:
+ osd:
+ debug osd: 5
+
+ Requires 3 osds on a single test node.
+ """
+ if config is None:
+ config = {}
+ assert isinstance(config, dict), \
+ 'divergent_priors task only accepts a dict for configuration'
+
+ while len(ctx.manager.get_osd_status()['up']) < 3:
+ time.sleep(10)
+ ctx.manager.raw_cluster_cmd('tell', 'osd.0', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('tell', 'osd.1', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('tell', 'osd.2', 'flush_pg_stats')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noout')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noin')
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'nodown')
+ ctx.manager.wait_for_clean()
+
+ # something that is always there
+ dummyfile = '/etc/fstab'
+ dummyfile2 = '/etc/resolv.conf'
+ testdir = teuthology.get_testdir(ctx)
+
+ # create 1 pg pool
+ log.info('creating foo')
+ ctx.manager.raw_cluster_cmd('osd', 'pool', 'create', 'foo', '1')
+
+ osds = [0, 1, 2]
+ for i in osds:
+ ctx.manager.set_config(i, osd_min_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_max_pg_log_entries=10)
+ ctx.manager.set_config(i, osd_pg_log_trim_min=5)
+
+ # determine primary
+ divergent = ctx.manager.get_pg_primary('foo', 0)
+ log.info("primary and soon to be divergent is %d", divergent)
+ non_divergent = list(osds)
+ non_divergent.remove(divergent)
+
+ log.info('writing initial objects')
+ first_mon = teuthology.get_first_mon(ctx, config)
+ (mon,) = ctx.cluster.only(first_mon).remotes.iterkeys()
+ # write 100 objects
+ for i in range(100):
+ rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i, dummyfile])
+
+ ctx.manager.wait_for_clean()
+
+ # blackhole non_divergent
+ log.info("blackholing osds %s", str(non_divergent))
+ for i in non_divergent:
+ ctx.manager.set_config(i, filestore_blackhole=1)
+
+ DIVERGENT_WRITE = 5
+ DIVERGENT_REMOVE = 5
+ # Write some soon to be divergent
+ log.info('writing divergent objects')
+ for i in range(DIVERGENT_WRITE):
+ rados(ctx, mon, ['-p', 'foo', 'put', 'existing_%d' % i,
+ dummyfile2], wait=False)
+ # Remove some soon to be divergent
+ log.info('remove divergent objects')
+ for i in range(DIVERGENT_REMOVE):
+ rados(ctx, mon, ['-p', 'foo', 'rm',
+ 'existing_%d' % (i + DIVERGENT_WRITE)], wait=False)
+ time.sleep(10)
+ mon.run(
+ args=['killall', '-9', 'rados'],
+ wait=True,
+ check_status=False)
+
+ # kill all the osds but leave divergent in
+ log.info('killing all the osds')
+ for i in osds:
+ ctx.manager.kill_osd(i)
+ for i in osds:
+ ctx.manager.mark_down_osd(i)
+ for i in non_divergent:
+ ctx.manager.mark_out_osd(i)
+
+ # bring up non-divergent
+ log.info("bringing up non_divergent %s", str(non_divergent))
+ for i in non_divergent:
+ ctx.manager.revive_osd(i)
+ for i in non_divergent:
+ ctx.manager.mark_in_osd(i)
+
+ # write 1 non-divergent object (ensure that old divergent one is divergent)
+ objname = "existing_%d" % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ log.info('writing non-divergent object ' + objname)
+ rados(ctx, mon, ['-p', 'foo', 'put', objname, dummyfile2])
+
+ ctx.manager.wait_for_recovery()
+
+ # ensure no recovery of up osds first
+ log.info('delay recovery')
+ for i in non_divergent:
+ ctx.manager.wait_run_admin_socket(
+ 'osd', i, ['set_recovery_delay', '100000'])
+
+ # bring in our divergent friend
+ log.info("revive divergent %d", divergent)
+ ctx.manager.raw_cluster_cmd('osd', 'set', 'noup')
+ ctx.manager.revive_osd(divergent)
+
+ log.info('delay recovery divergent')
+ ctx.manager.wait_run_admin_socket(
+ 'osd', divergent, ['set_recovery_delay', '100000'])
+
+ ctx.manager.raw_cluster_cmd('osd', 'unset', 'noup')
+ while len(ctx.manager.get_osd_status()['up']) < 3:
+ time.sleep(10)
+
+ log.info('wait for peering')
+ rados(ctx, mon, ['-p', 'foo', 'put', 'foo', dummyfile])
+
+ # At this point the divergent_priors should have been detected
+
+ log.info("killing divergent %d", divergent)
+ ctx.manager.kill_osd(divergent)
+
+ # Split pgs for pool foo
+ ctx.manager.raw_cluster_cmd('osd', 'pool', 'set', 'foo', 'pg_num', '2')
+ time.sleep(5)
+
+ # Export a pg
+ (exp_remote,) = ctx.\
+ cluster.only('osd.{o}'.format(o=divergent)).remotes.iterkeys()
+ FSPATH = ctx.manager.get_filepath()
+ JPATH = os.path.join(FSPATH, "journal")
+ prefix = ("sudo adjust-ulimits ceph-objectstore-tool "
+ "--data-path {fpath} --journal-path {jpath} "
+ "--log-file="
+ "/var/log/ceph/objectstore_tool.$$.log ".
+ format(fpath=FSPATH, jpath=JPATH))
+ pid = os.getpid()
+ expfile = os.path.join(testdir, "exp.{pid}.out".format(pid=pid))
+ cmd = ((prefix + "--op export --pgid 1.0 --file {file}").
+ format(id=divergent, file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ # Remove the same pg that was exported
+ cmd = ((prefix + "--op remove --pgid 1.0").
+ format(id=divergent, file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ # Kill one of non-divergent OSDs
+ log.info('killing osd.%d' % non_divergent[1])
+ ctx.manager.kill_osd(non_divergent[1])
+ ctx.manager.mark_down_osd(non_divergent[1])
+ # ctx.manager.mark_out_osd(non_divergent[1])
+
+ cmd = ((prefix + "--op import --file {file}").
+ format(id=non_divergent[1], file=expfile))
+ proc = exp_remote.run(args=cmd, wait=True,
+ check_status=False, stdout=StringIO())
+ assert proc.exitstatus == 0
+
+ # bring in our divergent friend and other node
+ log.info("revive divergent %d", divergent)
+ ctx.manager.revive_osd(divergent)
+ ctx.manager.mark_in_osd(divergent)
+ log.info("revive %d", non_divergent[1])
+ ctx.manager.revive_osd(non_divergent[1])
+
+ while len(ctx.manager.get_osd_status()['up']) < 3:
+ time.sleep(10)
+
+ log.info('delay recovery divergent')
+ ctx.manager.set_config(divergent, osd_recovery_delay_start=100000)
+ log.info('mark divergent in')
+ ctx.manager.mark_in_osd(divergent)
+
+ log.info('wait for peering')
+ rados(ctx, mon, ['-p', 'foo', 'put', 'foo', dummyfile])
+
+ log.info("killing divergent %d", divergent)
+ ctx.manager.kill_osd(divergent)
+ log.info("reviving divergent %d", divergent)
+ ctx.manager.revive_osd(divergent)
+ time.sleep(3)
+
+ log.info('allowing recovery')
+ # Set osd_recovery_delay_start back to 0 and kick the queue
+ for i in osds:
+ ctx.manager.raw_cluster_cmd('tell', 'osd.%d' % i, 'debug',
+ 'kick_recovery_wq', ' 0')
+
+ log.info('reading divergent objects')
+ for i in range(DIVERGENT_WRITE + DIVERGENT_REMOVE):
+ exit_status = rados(ctx, mon, ['-p', 'foo', 'get', 'existing_%d' % i,
+ '-o', '/tmp/existing'])
+ assert exit_status is 0
+
+ (remote,) = ctx.\
+ cluster.only('osd.{o}'.format(o=divergent)).remotes.iterkeys()
+ msg = "dirty_divergent_priors: true, divergent_priors: %d" \
+ % (DIVERGENT_WRITE + DIVERGENT_REMOVE)
+ cmd = 'grep "{msg}" /var/log/ceph/ceph-osd.{osd}.log'\
+ .format(msg=msg, osd=divergent)
+ proc = remote.run(args=cmd, wait=True, check_status=False)
+ assert proc.exitstatus == 0
+
+ cmd = 'rm {file}'.format(file=expfile)
+ remote.run(args=cmd, wait=True)
+ log.info("success")
hooks/post-receive
--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic