[prev in list] [next in list] [prev in thread] [next in thread]
List: ceph-users
Subject: Re: [ceph-users] Upgrading and lost OSDs
From: Alfredo Deza <adeza () redhat ! com>
Date: 2019-07-26 16:48:29
Message-ID: CAC-Np1zUMimGBXYq3XPhSyY52E_YChMwWwmwxCfy6sMF=2RNSQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
On Thu, Jul 25, 2019 at 7:00 PM Bob R <bobr@drinksbeer.org> wrote:
> I would try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume simple scan'
> again. We had some problems upgrading due to OSDs (perhaps initially
> installed as firefly?) missing the 'type' attribute and iirc the
> 'ceph-volume simple scan' command refused to overwrite existing json files
> after I made some changes to ceph-volume.
>
Ooof. I could swear that this issue was fixed already and it took me a
while to find out that it wasn't at all. We saw this a few months ago in
our Long Running Cluster used for dogfooding.
I've created a ticket to track this work at
http://tracker.ceph.com/issues/40987
But what you've done is exactly why we chose to persist the JSON files in
/etc/ceph/osd/*.json, so that an admin could tell if anything is missing
(or incorrect like in this case) and make the changes needed.
> Bob
>
> On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza <adeza@redhat.com> wrote:
>
> >
> >
> > On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch <peter.eisch@virginpulse.com>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > I appreciate the insistency that the directions be followed. I wholly
> > > agree. The only liberty I took was to do a ‘yum update' instead of just
> > > ‘yum update ceph-osd' and then reboot. (Also my MDS runs on the MON hosts,
> > > so it got update a step early.)
> > >
> > >
> > >
> > > As for the logs:
> > >
> > >
> > >
> > > [2019-07-24 15:07:22,713][ceph_volume.main][INFO ] Running command:
> > > ceph-volume simple scan
> > >
> > > [2019-07-24 15:07:22,714][ceph_volume.process][INFO ] Running command:
> > > /bin/systemctl show --no-pager --property=Id --state=running ceph-osd@*
> > >
> > > [2019-07-24 15:07:27,574][ceph_volume.main][INFO ] Running command:
> > > ceph-volume simple activate --all
> > >
> > > [2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO ]
> > > activating OSD specified in
> > > /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
> > >
> > > [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
> > > Required devices (block and data) not present for bluestore
> > >
> > > [2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ]
> > > bluestore devices found: [u'data']
> > >
> > > [2019-07-24 15:07:27,576][ceph_volume][ERROR ] exception caught by
> > > decorator
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py",
> > > line 59, in newfunc
> > >
> > > return f(*a, **kw)
> > >
> > > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148,
> > > in main
> > >
> > > terminal.dispatch(self.mapper, subcommand_args)
> > >
> > > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> > > 182, in dispatch
> > >
> > > instance.main()
> > >
> > > File
> > > "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line
> > > 33, in main
> > >
> > > terminal.dispatch(self.mapper, self.argv)
> > >
> > > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line
> > > 182, in dispatch
> > >
> > > instance.main()
> > >
> > > File
> > > "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> > > line 272, in main
> > >
> > > self.activate(args)
> > >
> > > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py",
> > > line 16, in is_root
> > >
> > > return func(*a, **kw)
> > >
> > > File
> > > "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> > > line 131, in activate
> > >
> > > self.validate_devices(osd_metadata)
> > >
> > > File
> > > "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py",
> > > line 62, in validate_devices
> > >
> > > raise RuntimeError('Unable to activate bluestore OSD due to missing
> > > devices')
> > >
> > > RuntimeError: Unable to activate bluestore OSD due to missing devices
> > >
> > >
> > >
> > > (this is repeated for each of the 16 drives)
> > >
> > >
> > >
> > > Any other thoughts? (I'll delete/create the OSDs with ceph-deply
> > > otherwise.)
> > >
> >
> > Try using `ceph-volume simple scan --stdout` so that it doesn't persist
> > data onto /etc/ceph/osd/ and inspect that the JSON produced is capturing
> > all the necessary details for OSDs.
> >
> > Alternatively, I would look into the JSON files already produced in
> > /etc/ceph/osd/ and check if the details are correct. The `scan` sub-command
> > does a tremendous effort to cover all cases where ceph-disk
> > created an OSD (filestore, bluestore, dmcrypt, etc...) but it is possible
> > that it may be hitting a problem. This is why the tool made these JSON
> > files available, so that they could be inspected and corrected if anything.
> >
> > The details of the scan sub-command can be found at
> > http://docs.ceph.com/docs/master/ceph-volume/simple/scan/ and the JSON
> > structure is described in detail below at
> > http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents
> >
> > In this particular case the tool is refusing to activate what seems to be
> > a bluestore OSD. Is it really a bluestore OSD? if so, then it can't find
> > where is the data partition. What does that partition look like (for any of
> > the failing OSDs) ? Does it use dmcrypt, how was it created? (hopefully
> > with ceph-disk!)
> >
> > If you know the data partition for a given OSD, try and pass it onto
> > 'scan'. For example if it is /dev/sda1 you could do `ceph-volume simple
> > scan /dev/sda1` and check its output.
> >
> >
> >
> > >
> > > peter
> > >
> > >
> > >
> > >
> > > Peter Eisch
> > > Senior Site Reliability Engineer
> > > T *1.612.659.3228* <1.612.659.3228>
> > > [image: Facebook] <https://www.facebook.com/VirginPulse>
> > > [image: LinkedIn] <https://www.linkedin.com/company/virgin-pulse>
> > > [image: Twitter] <https://twitter.com/virginpulse>
> > > *virginpulse.com* <https://www.virginpulse.com/>
> > > > *virginpulse.com/global-challenge*
> > > <https://www.virginpulse.com/en-gb/global-challenge/>
> > >
> > > Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland \
> > > | United Kingdom | USA Confidentiality Notice: The information contained in \
> > > this e-mail, including any attachment(s), is intended solely for use by the \
> > > designated recipient(s). Unauthorized use, dissemination, distribution, or
> > > reproduction of this message by anyone other than the intended
> > > recipient(s), or a person designated as responsible for delivering such
> > > messages to the intended recipient, is strictly prohibited and may be
> > > unlawful. This e-mail may contain proprietary, confidential or privileged
> > > information. Any views or opinions expressed are solely those of the author
> > > and do not necessarily represent those of Virgin Pulse, Inc. If you have
> > > received this message in error, or are not the named recipient(s), please
> > > immediately notify the sender and delete this e-mail message.
> > > v2.59
> > >
> > > *From: *Alfredo Deza <adeza@redhat.com>
> > > *Date: *Wednesday, July 24, 2019 at 3:02 PM
> > > *To: *Peter Eisch <peter.eisch@virginpulse.com>
> > > *Cc: *Paul Emmerich <paul.emmerich@croit.io>, "ceph-users@lists.ceph.com"
> > > <ceph-users@lists.ceph.com>
> > > *Subject: *Re: [ceph-users] Upgrading and lost OSDs
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 24, 2019 at 3:49 PM Peter Eisch <peter.eisch@virginpulse.com>
> > > wrote:
> > >
> > >
> > >
> > > I'm at step 6. I updated/rebooted the host to complete "installing the
> > > new packages and restarting the ceph-osd daemon" on the first OSD host.
> > > All the systemctl definitions to start the OSDs were deleted, all the
> > > properties in /var/lib/ceph/osd/ceph-* directories were deleted. All the
> > > files in /var/lib/ceph/osd-lockbox, for comparison, were untouched and
> > > still present.
> > >
> > >
> > >
> > > Peeking into step 7 I can run ceph-volume:
> > >
> > >
> > >
> > > # ceph-volume simple scan /dev/sda1
> > >
> > > Running command: /usr/sbin/cryptsetup status /dev/sda1
> > >
> > > Running command: /usr/sbin/cryptsetup status
> > > 93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > Running command: /bin/mount -v /dev/sda5 /tmp/tmpF5F8t2
> > >
> > > stdout: mount: /dev/sda5 mounted on /tmp/tmpF5F8t2.
> > >
> > > Running command: /usr/sbin/cryptsetup status /dev/sda5
> > >
> > > Running command: /bin/ceph --cluster ceph --name
> > > client.osd-lockbox.93fb5f2f-0273-4c87-a718-886d7e6db983 --keyring
> > > /tmp/tmpF5F8t2/keyring config-key get
> > > dm-crypt/osd/93fb5f2f-0273-4c87-a718-886d7e6db983/luks
> > >
> > > Running command: /bin/umount -v /tmp/tmpF5F8t2
> > >
> > > stderr: umount: /tmp/tmpF5F8t2 (/dev/sda5) unmounted
> > >
> > > Running command: /usr/sbin/cryptsetup --key-file - --allow-discards
> > > luksOpen /dev/sda1 93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > Running command: /bin/mount -v
> > > /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 /tmp/tmpYK0WEV
> > >
> > > stdout: mount: /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 mounted
> > > on /tmp/tmpYK0WEV.
> > >
> > > --> broken symlink found /tmp/tmpYK0WEV/block ->
> > > /dev/mapper/a05b447c-c901-4690-a249-cc1a2d62a110
> > >
> > > Running command: /usr/sbin/cryptsetup status /tmp/tmpYK0WEV/block_dmcrypt
> > >
> > > Running command: /usr/sbin/cryptsetup status
> > > /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > Running command: /bin/umount -v /tmp/tmpYK0WEV
> > >
> > > stderr: umount: /tmp/tmpYK0WEV
> > > (/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983) unmounted
> > >
> > > Running command: /usr/sbin/cryptsetup remove
> > > /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > --> OSD 0 got scanned and metadata persisted to file:
> > > /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json
> > >
> > > --> To take over management of this scanned OSD, and disable ceph-disk
> > > and udev, run:
> > >
> > > --> ceph-volume simple activate 0
> > > 93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > #
> > >
> > > #
> > >
> > > # ceph-volume simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983
> > >
> > > --> Required devices (block and data) not present for bluestore
> > >
> > > --> bluestore devices found: [u'data']
> > >
> > > --> RuntimeError: Unable to activate bluestore OSD due to missing
> > > devices
> > >
> > > #
> > >
> > >
> > >
> > > The tool detected bluestore, or rather, it failed to find a journal
> > > associated with /dev/sda1. Scanning a single partition can cause that.
> > > There is a flag to spit out the findings to STDOUT instead of persisting
> > > them in /etc/ceph/osd/
> > >
> > >
> > >
> > > Since this is a "whole system" upgrade, then the upgrade documentation
> > > instructions need to be followed:
> > >
> > >
> > >
> > > ceph-volume simple scan
> > > ceph-volume simple activate --all
> > >
> > >
> > >
> > > If the `scan` command doesn't display any information (not even with the
> > > --stdout flag) then the logs at /var/log/ceph/ceph-volume.log need to be
> > > inspected. It would be useful to check any findings in there
> > >
> > >
> > >
> > >
> > > Okay, this created /etc/ceph/osd/*.json. This is cool. Is there a
> > > command or option which will read these files and mount the devices?
> > >
> > >
> > >
> > > peter
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *Peter Eisch*
> > >
> > > Senior Site Reliability Engineer
> > >
> > > *T*
> > >
> > > 1.612.659.3228
> > >
> > > [image: Facebook]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook. \
> > > com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad22462 \
> > > 5423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227707952&sdata=B2JiNp12z7gsfF2i5T2l%2FSjfg6Fhg8E85OpdyGpEMHg%3D&reserved=0>
> > >
> > > [image: LinkedIn]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin. \
> > > com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25ce \
> > > a362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995 \
> > > 953227717942&sdata=z3Ii%2BGgPKe7fCOhNGXw%2BlD9j28YCY4gH81is%2BJoiSJU%3D&reserved=0>
> > >
> > > [image: Twitter]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2 \
> > > Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad2246254233 \
> > > 08d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=zUaGW2Fm16sdyJdHUPtDN6CzaMXtxMOHvmNDi9VshCw%3D&reserved=0>
> > >
> > > virginpulse.com
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpul \
> > > se.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d \
> > > 71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=n3m9S%2Bt8fYzGY%2BqPw2wT433TQhf2oPXp9wAum9s9%2BUk%3D&reserved=0>
> > >
> > > >
> > >
> > > virginpulse.com/global-challenge
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpul \
> > > se.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.co \
> > > m%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0% \
> > > 7C636995953227727937&sdata=5wn%2F%2FkY1IL0d4BxXNpqCJUHG09gUFRTr2S9KWv1mVG4%3D&reserved=0>
> > >
> > >
> > > Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland \
> > > | United Kingdom | USA
> > > Confidentiality Notice: The information contained in this e-mail,
> > > including any attachment(s), is intended solely for use by the designated
> > > recipient(s). Unauthorized use, dissemination, distribution, or
> > > reproduction of this message by anyone other than the intended
> > > recipient(s), or a person designated as responsible for delivering such
> > > messages to the intended recipient, is strictly prohibited and may be
> > > unlawful. This e-mail may contain proprietary, confidential or privileged
> > > information. Any views or opinions expressed are solely those of the author
> > > and do not necessarily represent those of Virgin Pulse, Inc. If you have
> > > received this message in error, or are not the named recipient(s), please
> > > immediately notify the sender and delete this e-mail message.
> > >
> > > v2.59
> > >
> > > *From: *Alfredo Deza <adeza@redhat.com>
> > > *Date: *Wednesday, July 24, 2019 at 2:20 PM
> > > *To: *Peter Eisch <peter.eisch@virginpulse.com>
> > > *Cc: *Paul Emmerich <paul.emmerich@croit.io>, "ceph-users@lists.ceph.com"
> > > <ceph-users@lists.ceph.com>
> > > *Subject: *Re: [ceph-users] Upgrading and lost OSDs
> > >
> > >
> > >
> > > On Wed, Jul 24, 2019 at 2:56 PM Peter Eisch <peter.eisch@virginpulse.com>
> > > wrote:
> > >
> > > Hi Paul,
> > >
> > > To do better to answer you question, I'm following:
> > > http://docs.ceph.com/docs/nautilus/releases/nautilus/
> > > <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com% \
> > > 2Fdocs%2Fnautilus%2Freleases%2Fnautilus%2F&data=02%7C01%7Cpeter.eisch%40virginpu \
> > > lse.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C \
> > > 0%7C0%7C636995953227727937&sdata=D1Q0Hrg9mq2tTtfTt1kp4Ts3vBc7mertKvYy8vBWNF8%3D&reserved=0>
> > >
> > > At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum
> > > update' for patching the host and rebooted to pick up the current centos
> > > kernel.
> > >
> > >
> > >
> > > If you are at Step 6 then it is *crucial* to understand that the tooling
> > > used to create the OSDs is no longer available and Step 7 *is absolutely
> > > required*.
> > >
> > >
> > >
> > > ceph-volume has to scan the system and give you the output of all OSDs
> > > found so that it can persist them in /etc/ceph/osd/*.json files and then
> > > can later be
> > >
> > > "activated".
> > >
> > >
> > >
> > >
> > > I didn't do anything to specific commands for just updating the ceph
> > > RPMs in this process.
> > >
> > >
> > >
> > > It is not clear if you are at Step 6 and wondering why OSDs are not up,
> > > or you are past that and ceph-volume wasn't able to detect anything.
> > >
> > >
> > >
> > > peter
> > >
> > > *Peter Eisch*
> > >
> > > Senior Site Reliability Engineer
> > >
> > > *T*
> > >
> > > 1.612.659.3228
> > >
> > > [image: Facebook]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook. \
> > > com%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad22462 \
> > > 5423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=uncHzIOSXt25%2F0NydXSJaLAf6E3Ad05N%2BJLBKYYJQ%2Fw%3D&reserved=0>
> > >
> > > [image: LinkedIn]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin. \
> > > com%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25ce \
> > > a362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=NxQ3BmhgWo93uoQfJ1W7lLcDdSUQHgoXu1I49vzibwE%3D&reserved=0>
> > >
> > > [image: Twitter]
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2 \
> > > Fvirginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad2246254233 \
> > > 08d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=sYUapxFqHq0LVyxO4I7kkwN1y9PG5ZLHd83gseRIxvM%3D&reserved=0>
> > >
> > > virginpulse.com
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpul \
> > > se.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d \
> > > 71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=aoXH94QOjdngJXkAPcz9kmJAK5BA6c9rR5BP01lX0bw%3D&reserved=0>
> > >
> > > >
> > >
> > > virginpulse.com/global-challenge
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpul \
> > > se.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.co \
> > > m%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0% \
> > > 7C636995953227757921&sdata=aUbx8uIlqldr5JtKT1PL05nbzcNPHONpouVkj1qXRdM%3D&reserved=0>
> > >
> > >
> > > Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland \
> > > | United Kingdom | USA
> > > Confidentiality Notice: The information contained in this e-mail,
> > > including any attachment(s), is intended solely for use by the designated
> > > recipient(s). Unauthorized use, dissemination, distribution, or
> > > reproduction of this message by anyone other than the intended
> > > recipient(s), or a person designated as responsible for delivering such
> > > messages to the intended recipient, is strictly prohibited and may be
> > > unlawful. This e-mail may contain proprietary, confidential or privileged
> > > information. Any views or opinions expressed are solely those of the author
> > > and do not necessarily represent those of Virgin Pulse, Inc. If you have
> > > received this message in error, or are not the named recipient(s), please
> > > immediately notify the sender and delete this e-mail message.
> > >
> > > v2.59
> > >
> > >
> > > From: Paul Emmerich <paul.emmerich@croit.io>
> > > Date: Wednesday, July 24, 2019 at 1:39 PM
> > > To: Peter Eisch <peter.eisch@virginpulse.com>
> > > Cc: Xavier Trilla <xavier.trilla@clouding.io>, "
> > > ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com>
> > > Subject: Re: [ceph-users] Upgrading and lost OSDs
> > >
> > > On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto:
> > > peter.eisch@virginpulse.com> wrote:
> > > # lsblk
> > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > > sda 8:0 0 1.7T 0 disk
> > > ├─sda1 8:1 0 100M 0 part
> > > ├─sda2 8:2 0 1.7T 0 part
> > > └─sda5 8:5 0 10M 0 part
> > > sdb 8:16 0 1.7T 0 disk
> > > ├─sdb1 8:17 0 100M 0 part
> > > ├─sdb2 8:18 0 1.7T 0 part
> > > └─sdb5 8:21 0 10M 0 part
> > > sdc 8:32 0 1.7T 0 disk
> > > ├─sdc1 8:33 0 100M 0 part
> > >
> > > That's ceph-disk which was removed, run "ceph-volume simple scan"
> > >
> > >
> > > --
> > > Paul Emmerich
> > >
> > > Looking for help with your Ceph cluster? Contact us at
> > > https://nam02.safelinks.protection.outlook.com/?url=https://croit.io&data=02|01| \
> > > peter.eisch@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6a55 \
> > > a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0
> > > <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io&dat \
> > > a=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb \
> > > 123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=gAFSElTqBRhu1zDIQYvWQ2WZtqqHoW%2FLa3stBfqXXHQ%3D&reserved=0>
> > >
> > > croit GmbH
> > > Freseniusstr. 31h
> > > 81247 München
> > >
> > > https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.io&data=02| \
> > > 01|peter.eisch@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e892b4cf6 \
> > > a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0
> > > <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.croit.io& \
> > > data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968% \
> > > 7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=LWJKCZ4VxuwoHMJGSWiEWAcawFPw7pDGC48%2B6bnXk6A%3D&reserved=0>
> > >
> > > Tel: +49 89 1896585 90
> > >
> > >
> > > ...
> > > I'm thinking the OSD would start (I can recreate the .service
> > > definitions in systemctl) if the above were mounted in a way like they are
> > > on another of my hosts:
> > > # lsblk
> > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> > > sda 8:0 0 1.7T 0 disk
> > > ├─sda1 8:1 0 100M 0 part
> > > │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt
> > > /var/lib/ceph/osd/ceph-16
> > > ├─sda2 8:2 0 1.7T 0 part
> > > │ └─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt
> > > └─sda5 8:5 0 10M 0 part
> > > /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5
> > > sdb 8:16 0 1.7T 0 disk
> > > ├─sdb1 8:17 0 100M 0 part
> > > │ └─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt
> > > /var/lib/ceph/osd/ceph-17
> > > ├─sdb2 8:18 0 1.7T 0 part
> > > │ └─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt
> > > └─sdb5 8:21 0 10M 0 part
> > > /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2
> > > sdc 8:32 0 1.7T 0 disk
> > > ├─sdc1 8:33 0 100M 0 part
> > > │ └─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt
> > > /var/lib/ceph/osd/ceph-18
> > > ├─sdc2 8:34 0 1.7T 0 part
> > > │ └─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt
> > > ...
> > >
> > > Thank you for your time on this,
> > >
> > > peter
> > >
> > > From: Xavier Trilla <mailto:xavier.trilla@clouding.io>
> > > Date: Wednesday, July 24, 2019 at 1:25 PM
> > > To: Peter Eisch <mailto:peter.eisch@virginpulse.com>
> > > Cc: "mailto:ceph-users@lists.ceph.com" <mailto:ceph-users@lists.ceph.com
> > > >
> > > Subject: Re: [ceph-users] Upgrading and lost OSDs
> > >
> > > Hi Peter,
> > >
> > > Im not sure but maybe after some changes the OSDs are not being
> > > recongnized by ceph scripts.
> > >
> > > Ceph used to use udev to detect the OSDs and then moved to lvm, which
> > > kind of OSDs are you running? Blustore or filestore? Which version did you
> > > use to create them?
> > >
> > > Cheers!
> > >
> > > El 24 jul 2019, a les 20:04, Peter Eisch <mailto:mailto:
> > > peter.eisch@virginpulse.com> va escriure:
> > > Hi,
> > >
> > > I'm working through updating from 12.2.12/luminious to 14.2.2/nautilus
> > > on centos 7.6. The managers are updated alright:
> > >
> > > # ceph -s
> > > cluster:
> > > id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec
> > > health: HEALTH_WARN
> > > Degraded data redundancy: 24177/9555955 objects degraded
> > > (0.253%), 7 pgs degraded, 1285 pgs undersized
> > > 3 monitors have not enabled msgr2
> > > ...
> > >
> > > I updated ceph on a OSD host with 'yum update' and then rebooted to grab
> > > the current kernel. Along the way, the contents of all the directories in
> > > /var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this.
> > > I can manage the undersized but I'd like to get these drives working again
> > > without deleting each OSD and recreating them.
> > >
> > > So far I've pulled the respective cephx key into the 'keyring' file and
> > > populated 'bluestore' into the 'type' files but I'm unsure how to get the
> > > lockboxes mounted to where I can get the OSDs running. The osd-lockbox
> > > directory is otherwise untouched from when the OSDs were deployed.
> > >
> > > Is there a way to run ceph-deploy or some other tool to rebuild the
> > > mounts for the drives?
> > >
> > > peter
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com \
> > > %2Flistinfo.cgi%2Fceph-users-ceph.com&data=02%7C01%7Cpeter.eisch%40virginpulse.c \
> > > om%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0 \
> > > %7C636995953227767913&sdata=jeAj%2FfzN%2BOG1NPFeZwqYnQiB4mgbgLGqq85tz99xBz8%3D&reserved=0>
> > >
> > > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
[Attachment #7 (text/html)]
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Thu, Jul 25, 2019 at 7:00 PM Bob R <<a \
href="mailto:bobr@drinksbeer.org">bobr@drinksbeer.org</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I would \
try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume simple scan' \
again. We had some problems upgrading due to OSDs (perhaps initially installed as \
firefly?) missing the 'type' attribute and iirc the 'ceph-volume simple \
scan' command refused to overwrite existing json files after I made some changes \
to ceph-volume. <br></div></blockquote><div><br></div><div>Ooof. I could swear that \
this issue was fixed already and it took me a while to find out that it wasn't at \
all. We saw this a few months ago in our Long Running Cluster used for dogfooding. \
<br></div><div><br></div><div>I've created a ticket to track this work at <a \
href="http://tracker.ceph.com/issues/40987">http://tracker.ceph.com/issues/40987</a></div><div><br></div><div>But \
what you've done is exactly why we chose to persist the JSON files in \
/etc/ceph/osd/*.json, so that an admin could tell if anything is missing (or \
incorrect like in this case) and make the changes \
needed.</div><div><br></div><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div \
dir="ltr"><div><br></div><div>Bob</div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza <<a \
href="mailto:adeza@redhat.com" target="_blank">adeza@redhat.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div \
dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Wed, Jul 24, 2019 at 4:15 PM Peter Eisch <<a \
href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171WordSection1">
<p class="MsoNormal">Hi,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I appreciate the insistency that the directions be followed. I \
wholly agree. The only liberty I took was to do a ‘yum update' instead of just \
‘yum update ceph-osd' and then reboot. (Also my MDS runs on the MON hosts, so it \
got update a step early.)<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">As for the logs:<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:22,713][ceph_volume.main][INFO ] Running command: ceph-volume simple \
scan<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:22,714][ceph_volume.process][INFO ] Running command: /bin/systemctl show \
--no-pager --property=Id --state=running ceph-osd@*<u></u><u></u></span></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:27,574][ceph_volume.main][INFO ] Running command: ceph-volume simple \
activate --all<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:27,575][ceph_volume.devices.simple.activate][INFO ] activating OSD specified \
in /etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json<u></u><u></u></span></p> \
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] Required devices (block \
and data) not present for bluestore<u></u><u></u></span></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] bluestore devices found: \
[u'data']<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">[2019-07-24 \
15:07:27,576][ceph_volume][ERROR ] exception caught by \
decorator<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">Traceback (most recent call \
last):<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in \
newfunc<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> return f(*a, \
**kw)<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in \
main<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> terminal.dispatch(self.mapper, \
subcommand_args)<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in \
dispatch<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> \
instance.main()<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", line \
33, in main<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> terminal.dispatch(self.mapper, \
self.argv)<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in \
dispatch<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> \
instance.main()<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", \
line 272, in main<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> \
self.activate(args)<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in \
is_root<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> return func(*a, \
**kw)<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", \
line 131, in activate<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> \
self.validate_devices(osd_metadata)<u></u><u></u></span></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas"> File \
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", \
line 62, in validate_devices<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas"> raise RuntimeError('Unable to \
activate bluestore OSD due to missing devices')<u></u><u></u></span></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">RuntimeError: \
Unable to activate bluestore OSD due to missing devices<u></u><u></u></span></p> <p \
class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p> <p \
class="MsoNormal"><span style="font-size:12pt">(this is repeated for each of the 16 \
drives)<u></u><u></u></span></p> <p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Any other thoughts? (I'll delete/create the OSDs with \
ceph-deply otherwise.)</p></div></div></blockquote><div><br></div><div>Try using \
`ceph-volume simple scan --stdout` so that it doesn't persist data onto \
/etc/ceph/osd/ and inspect that the JSON produced is capturing all the necessary \
details for OSDs.</div><div><br></div><div>Alternatively, I would look into the JSON \
files already produced in /etc/ceph/osd/ and check if the details are correct. The \
`scan` sub-command does a tremendous effort to cover all cases where \
ceph-disk</div><div>created an OSD (filestore, bluestore, dmcrypt, etc...) but it is \
possible that it may be hitting a problem. This is why the tool made these JSON files \
available, so that they could be inspected and corrected if \
anything.</div><div><br></div><div>The details of the scan sub-command can be found \
at <a href="http://docs.ceph.com/docs/master/ceph-volume/simple/scan/" \
target="_blank">http://docs.ceph.com/docs/master/ceph-volume/simple/scan/</a> and the \
JSON structure is described in detail below at <a \
href="http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents" \
target="_blank">http://docs.ceph.com/docs/master/ceph-volume/simple/scan/#json-contents</a></div><div><br></div><div>In \
this particular case the tool is refusing to activate what seems to be a bluestore \
OSD. Is it really a bluestore OSD? if so, then it can't find where is the data \
partition. What does that partition look like (for any of the failing OSDs) ? Does it \
use dmcrypt, how was it created? (hopefully with \
ceph-disk!)</div><div><br></div><div>If you know the data partition for a given OSD, \
try and pass it onto 'scan'. For example if it is /dev/sda1 you could do \
`ceph-volume simple scan /dev/sda1` and check its \
output.</div><div><br></div><div><br></div><blockquote class="gmail_quote" \
style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div \
class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171WordSection1"><p \
class="MsoNormal"><u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">peter<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div><table style="width:100%" cellspacing="0" cellpadding="0" border="0"><tbody><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="font-size:0px" cellspacing="0" cellpadding="0" border="0"><tbody><tr \
style="font-size:0px"><td style="padding:8px 0px 0px;vertical-align:top" \
align="left"><table style="font-size:0px;color:rgb(84,84,84);font-style:normal;font-weight:700;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="vertical-align:top;font-family:Arial" align="left">Peter Eisch<span \
style="font-family:remialcxesans;font-size:1px;color:rgb(255,255,255);line-height:1px"></span></td></tr><tr \
style="font-size:13.33px;font-weight:400"><td \
style="vertical-align:top;font-family:Arial" align="left">Senior Site Reliability \
Engineer</td></tr></tbody></table></td></tr></tbody></table></td></tr><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="width:100%;font-size:0px" cellspacing="0" cellpadding="0" \
border="0"><tbody><tr style="font-size:0px"><td style="padding:12px 0px \
0px;vertical-align:top" align="left"><table style="width:100%;font-size:0px" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:0px"><td \
style="vertical-align:top" align="left"><table style="width:100%;font-size:0px" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:0px"><td \
style="vertical-align:top" align="left"><table \
style="font-size:0px;color:rgb(84,84,84);font-style:normal;font-weight:400;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="vertical-align:top;font-size:0px" align="left"><table \
style="white-space:nowrap;color:rgb(84,84,84);font-size:13.33px;font-family:Arial;font-weight:700;font-style:normal;text-align:left;width:15px" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="font-family:Arial">T</td></tr></tbody></table></td><td \
style="vertical-align:top;font-family:Arial" align="left"><a \
href="tel:1.612.659.3228" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
style="text-decoration:none;color:rgb(84,84,84)" target="_blank"><b \
style="font-weight:400">1.612.659.3228</b></a></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="font-size:0px" cellspacing="0" cellpadding="0" border="0"><tbody><tr \
style="font-size:0px"><td style="padding:12px 0px 0px;vertical-align:top" \
align="left"><table style="font-size:0px" cellspacing="0" cellpadding="0" \
border="0"><tbody><tr style="font-size:0px"><td style="vertical-align:top" \
align="left"><table style="font-size:0px;line-height:normal" cellspacing="0" \
cellpadding="0" border="0"><tbody><tr style="font-size:0px"><td style="padding:0px \
8px 3px 0px;vertical-align:top" align="left"><a \
href="https://www.facebook.com/VirginPulse" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
style="text-decoration:none" target="_blank"><img src="cid:16c25a202ca7914647a7" \
title="Facebook" alt="Facebook" style="width: 20px; min-width: 20px; max-width: 20px; \
font-size: 12px;" width="20" border="0"></a></td></tr></tbody></table></td><td \
style="vertical-align:top" align="left"><table \
style="font-size:0px;line-height:normal" cellspacing="0" cellpadding="0" \
border="0"><tbody><tr style="font-size:0px"><td style="padding:0px 8px 3px \
0px;vertical-align:top" align="left"><a \
href="https://www.linkedin.com/company/virgin-pulse" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
style="text-decoration:none" target="_blank"><img src="cid:16c25a202cacad0e5528" \
title="LinkedIn" alt="LinkedIn" style="width: 20px; min-width: 20px; max-width: 20px; \
font-size: 12px;" width="20" border="0"></a></td></tr></tbody></table></td><td \
style="vertical-align:top" align="left"><table \
style="font-size:0px;line-height:normal" cellspacing="0" cellpadding="0" \
border="0"><tbody><tr style="font-size:0px"><td style="padding:0px 8px 3px \
0px;vertical-align:top" align="left"><a href="https://twitter.com/virginpulse" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
style="text-decoration:none" target="_blank"><img src="cid:16c25a202ca4afd5c2d9" \
title="Twitter" alt="Twitter" style="width: 20px; min-width: 20px; max-width: 20px; \
font-size: 12px;" width="20" \
border="0"></a></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="font-size:0px" cellspacing="0" cellpadding="0" border="0"><tbody><tr \
style="font-size:0px"><td style="padding:8px 0px 0px;vertical-align:top" \
align="left"><table style="font-size:0px;color:rgb(84,84,84);font-style:normal;font-weight:400;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="vertical-align:top;font-family:Arial" align="left"><a \
href="https://www.virginpulse.com/" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
style="text-decoration:none;color:rgb(84,84,84)" target="_blank"><b \
style="font-weight:400">virginpulse.com</b></a><br></td><td \
style="vertical-align:top;font-size:0px" align="left"><table \
style="font-size:0px;color:rgb(84,84,84);font-style:normal;font-weight:400;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="padding:0px 4px;vertical-align:top;font-family:Arial" \
align="left">|</td></tr></tbody></table></td><td \
style="vertical-align:top;font-family:Arial" align="left"><a \
href="https://www.virginpulse.com/en-gb/global-challenge/" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171LPlnk689713" \
title="Global Challenge" style="text-decoration:none;color:rgb(84,84,84)" \
target="_blank"><b style="font-weight:400">virginpulse.com/global-challenge</b></a><br></td></tr></tbody></table></td></tr></tbody></table></td></tr><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="font-size:0px;color:rgb(84,84,84);font-style:normal;font-weight:400;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:13.33px"><td \
style="padding:3px 0px 0px;vertical-align:top;font-family:Arial" \
align="left">Australia | Bosnia and Herzegovina | Brazil | Canada | \
Singapore | Switzerland | United Kingdom | \
USA<br></td></tr></tbody></table></td></tr><tr style="font-size:0px"><td \
style="vertical-align:top" align="left"><table style="width:100%;font-size:0px" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:0px"><td \
style="padding:12px 0px 10px;vertical-align:top" align="left"><table \
style="white-space:normal;color:rgb(186,186,186);font-size:13.33px;font-family:Arial;font-weight:400;font-style:normal;text-align:justify;line-height:10px;width:100%" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:9px"><td \
style="font-family:Arial">Confidentiality Notice: The information contained in this \
e-mail, including any attachment(s), is intended solely for use by the designated \
recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this \
message by anyone other than the intended recipient(s), or a person designated as \
responsible for delivering such messages to the intended recipient, is strictly \
prohibited and may be unlawful. This e-mail may contain proprietary, confidential or \
privileged information. Any views or opinions expressed are solely those of the \
author and do not necessarily represent those of Virgin Pulse, Inc. If you have \
received this message in error, or are not the named recipient(s), please immediately \
notify the sender and delete this e-mail \
message.</td></tr></tbody></table></td></tr></tbody></table></td></tr><tr \
style="font-size:0px"><td style="vertical-align:top" align="left"><table \
style="font-size:0px;color:rgb(186,186,186);font-style:normal;font-weight:400;white-space:nowrap" \
cellspacing="0" cellpadding="0" border="0"><tbody><tr style="font-size:10px"><td \
style="padding:0px 0px 12px;vertical-align:top;font-family:Arial" \
align="left">v2.59<br></td></tr></tbody></table></td></tr></tbody></table></div><div \
style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid \
none none;border-width:1pt medium medium;padding:3pt 0in 0in"> <p \
class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span \
style="font-size:12pt;color:black">Alfredo Deza <<a href="mailto:adeza@redhat.com" \
target="_blank">adeza@redhat.com</a>><br> <b>Date: </b>Wednesday, July 24, 2019 at \
3:02 PM<br> <b>To: </b>Peter Eisch <<a href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>><br> <b>Cc: </b>Paul Emmerich \
<<a href="mailto:paul.emmerich@croit.io" \
target="_blank">paul.emmerich@croit.io</a>>, "<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>" <<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>><br> <b>Subject: </b>Re: \
[ceph-users] Upgrading and lost OSDs<u></u><u></u></span></p> </div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jul 24, 2019 at 3:49 PM Peter Eisch <<a \
href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>> wrote:<u></u><u></u></p> </div>
<blockquote style="border-color:currentcolor currentcolor currentcolor \
rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium \
1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"> <div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I'm at step 6. I updated/rebooted the host to complete \
"installing the new packages and restarting the ceph-osd daemon" on the first OSD \
host. All the systemctl definitions to start the OSDs were deleted, all the \
properties in /var/lib/ceph/osd/ceph-* directories were deleted. All the files in \
/var/lib/ceph/osd-lockbox, for comparison, were untouched and still \
present.<u></u><u></u></p> <p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Peeking into step 7 I can run ceph-volume:<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas"># ceph-volume \
simple scan /dev/sda1</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">Running command: /usr/sbin/cryptsetup \
status /dev/sda1</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">Running command: /usr/sbin/cryptsetup \
status 93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/bin/mount -v /dev/sda5 /tmp/tmpF5F8t2</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">stdout: mount: \
/dev/sda5 mounted on /tmp/tmpF5F8t2.</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/usr/sbin/cryptsetup status /dev/sda5</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/bin/ceph --cluster ceph --name \
client.osd-lockbox.93fb5f2f-0273-4c87-a718-886d7e6db983 --keyring \
/tmp/tmpF5F8t2/keyring config-key get \
dm-crypt/osd/93fb5f2f-0273-4c87-a718-886d7e6db983/luks</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/bin/umount -v /tmp/tmpF5F8t2</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">stderr: umount: /tmp/tmpF5F8t2 \
(/dev/sda5) unmounted</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">Running command: /usr/sbin/cryptsetup \
--key-file - --allow-discards luksOpen /dev/sda1 \
93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/bin/mount -v /dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 \
/tmp/tmpYK0WEV</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">stdout: mount: \
/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983 mounted on \
/tmp/tmpYK0WEV.</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">--> broken symlink found \
/tmp/tmpYK0WEV/block -> \
/dev/mapper/a05b447c-c901-4690-a249-cc1a2d62a110</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/usr/sbin/cryptsetup status /tmp/tmpYK0WEV/block_dmcrypt</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/usr/sbin/cryptsetup status \
/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running command: \
/bin/umount -v /tmp/tmpYK0WEV</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">stderr: umount: /tmp/tmpYK0WEV \
(/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983) unmounted</span><u></u><u></u></p> \
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">Running \
command: /usr/sbin/cryptsetup remove \
/dev/mapper/93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">--> OSD 0 got \
scanned and metadata persisted to file: \
/etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">--> To take \
over management of this scanned OSD, and disable ceph-disk and udev, \
run:</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">--> ceph-volume simple \
activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">#</span><u></u><u></u></p> <p \
class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">#</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas"># ceph-volume \
simple activate 0 93fb5f2f-0273-4c87-a718-886d7e6db983</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">--> Required \
devices (block and data) not present for bluestore</span><u></u><u></u></p> <p \
class="MsoNormal"><span style="font-size:10pt;font-family:Consolas">--> bluestore \
devices found: [u'data']</span><u></u><u></u></p> <p class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">--> RuntimeError: Unable to activate \
bluestore OSD due to missing devices</span><u></u><u></u></p> <p \
class="MsoNormal"><span \
style="font-size:10pt;font-family:Consolas">#</span><u></u><u></u></p> </div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The tool detected bluestore, or rather, it failed to find a \
journal associated with /dev/sda1. Scanning a single partition can cause that. There \
is a flag to spit out the findings to STDOUT instead of persisting them in \
/etc/ceph/osd/<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Since this is a "whole system" upgrade, then the \
upgrade documentation instructions need to be followed:<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<pre><span class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-c1">ceph-volume \
simple scan</span><br><span \
class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-c1">ceph-volume \
simple activate --all</span><u></u><u></u></pre> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If the `scan` command doesn't display any information (not \
even with the --stdout flag) then the logs at /var/log/ceph/ceph-volume.log need to \
be inspected. It would be useful to check any findings in there<u></u><u></u></p> \
</div> <p class="MsoNormal"><br>
<u></u><u></u></p>
<blockquote style="border-color:currentcolor currentcolor currentcolor \
rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium \
1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"> <div>
<div>
<p class="MsoNormal">Okay, this created
<span class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m4668523972764021174s1">/etc/ceph/osd/*.json. \
This is cool. Is there a command or option which will read these files and mount \
the devices?</span><u></u><u></u></p> <p class="MsoNormal"><span \
class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m4668523972764021174s1"> \
</span><u></u><u></u></p> <p class="MsoNormal"><span \
class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m4668523972764021174s1">peter</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:6pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><b><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Peter \
Eisch<u></u><u></u></span></b></p> </td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Senior Site \
Reliability Engineer<u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:11.25pt" width="15" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in">
<p class="MsoNormal"><b><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">T<u></u><u></u></span></b></p>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="tel:1.612.659.3228" target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">1.612.659.3228</span></a><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.c \
om%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad22462542 \
3308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227707952&sdata=B2JiNp12z7gsfF2i5T2l%2FSjfg6Fhg8E85OpdyGpEMHg%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171_x0000_i1030" \
src="cid:16c25a202c84cff311" alt="Facebook" width="20" height="20" \
border="0"></span></span></a><u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.c \
om%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea36 \
2ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717 \
942&sdata=z3Ii%2BGgPKe7fCOhNGXw%2BlD9j28YCY4gH81is%2BJoiSJU%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171_x0000_i1029" \
src="cid:16c25a202c95b16b22" alt="LinkedIn" width="20" height="20" \
border="0"></span></span></a><u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2F \
virginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d \
71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=zUaGW2Fm16sdyJdHUPtDN6CzaMXtxMOHvmNDi9VshCw%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171_x0000_i1028" \
src="cid:16c25a202c9692e333" alt="Twitter" width="20" height="20" \
border="0"></span></span></a><u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:6pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpuls \
e.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d710 \
71c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227717942&sdata=n3m9S%2Bt8fYzGY%2BqPw2wT433TQhf2oPXp9wAum9s9%2BUk%3D&reserved=0" \
target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">virginpulse.com</span></a><u></u><u></u></span></p>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 3pt" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">|<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpuls \
e.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7 \
C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C6369959 \
53227727937&sdata=5wn%2F%2FkY1IL0d4BxXNpqCJUHG09gUFRTr2S9KWv1mVG4%3D&reserved=0" \
title="Global Challenge" target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">virginpulse.com/global-challenge</span></a><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:2.25pt 0in 0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Australia | \
Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | \
United Kingdom | USA<u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 7.5pt" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in">
<p class="MsoNormal" style="text-align:justify;line-height:7.5pt"><span \
style="font-size:7pt;font-family:Arial,sans-serif;color:rgb(186,186,186)">Confidentiality \
Notice: The information contained in this e-mail, including any attachment(s), is \
intended solely for use by the designated recipient(s). Unauthorized use, \
dissemination, distribution, or reproduction of this message by anyone other than the \
intended recipient(s), or a person designated as responsible for delivering such \
messages to the intended recipient, is strictly prohibited and may be unlawful. This \
e-mail may contain proprietary, confidential or privileged information. Any views or \
opinions expressed are solely those of the author and do not necessarily represent \
those of Virgin Pulse, Inc. If you have received this message in error, or are not \
the named recipient(s), please immediately notify the sender and delete this e-mail \
message.<u></u><u></u></span></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 0in 9pt" valign="top">
<p class="MsoNormal"><span \
style="font-size:7.5pt;font-family:Arial,sans-serif;color:rgb(186,186,186)">v2.59<u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt \
0in 0in;border-color:currentcolor"> <p class="MsoNormal"><b><span \
style="font-size:12pt;color:black">From: </span></b><span \
style="font-size:12pt;color:black">Alfredo Deza <<a href="mailto:adeza@redhat.com" \
target="_blank">adeza@redhat.com</a>><br> <b>Date: </b>Wednesday, July 24, 2019 at \
2:20 PM<br> <b>To: </b>Peter Eisch <<a href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>><br> <b>Cc: </b>Paul Emmerich \
<<a href="mailto:paul.emmerich@croit.io" \
target="_blank">paul.emmerich@croit.io</a>>, "<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>" <<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>><br> <b>Subject: </b>Re: \
[ceph-users] Upgrading and lost OSDs</span><u></u><u></u></p> </div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jul 24, 2019 at 2:56 PM Peter Eisch <<a \
href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>> wrote:<u></u><u></u></p> </div>
<blockquote style="border-style:none none none solid;border-width:medium medium \
medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor \
currentcolor currentcolor rgb(204,204,204)"> <div>
<div>
<p class="MsoNormal">Hi Paul,<br>
<br>
To do better to answer you question, I'm following: <a \
href="https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2 \
Fdocs%2Fnautilus%2Freleases%2Fnautilus%2F&data=02%7C01%7Cpeter.eisch%40virginpulse \
.com%7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C6 \
36995953227727937&sdata=D1Q0Hrg9mq2tTtfTt1kp4Ts3vBc7mertKvYy8vBWNF8%3D&reserved=0" \
target="_blank"> http://docs.ceph.com/docs/nautilus/releases/nautilus/</a><br>
<br>
At step 6, upgrade OSDs, I jumped on an OSD host and did a full 'yum update' \
for patching the host and rebooted to pick up the current centos \
kernel.<u></u><u></u></p> </div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If you are at Step 6 then it is *crucial* to understand that the \
tooling used to create the OSDs is no longer available and Step 7 *is absolutely \
required*.<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ceph-volume has to scan the system and give you the output of \
all OSDs found so that it can persist them in /etc/ceph/osd/*.json files and then can \
later be <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">"activated".<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium \
medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor \
currentcolor currentcolor rgb(204,204,204)"> <div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
I didn't do anything to specific commands for just updating the ceph RPMs in this \
process.<u></u><u></u></p> </div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">It is not clear if you are at Step 6 and wondering why OSDs are \
not up, or you are past that and ceph-volume wasn't able to detect \
anything.<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium \
medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor \
currentcolor currentcolor rgb(204,204,204)"> <div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">peter<u></u><u></u></p>
<div>
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:6pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><b><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Peter \
Eisch</span></b><u></u><u></u></p> </td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Senior Site \
Reliability Engineer</span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:11.25pt" width="15" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in">
<p class="MsoNormal"><b><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">T</span></b><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="tel:1.612.659.3228" target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">1.612.659.3228</span></a></span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.c \
om%2FVirginPulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad22462542 \
3308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=uncHzIOSXt25%2F0NydXSJaLAf6E3Ad05N%2BJLBKYYJQ%2Fw%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m_4668523972764021174_x005f_x0000_i1027" \
src="cid:16c25a202c97745b44" alt="Facebook" width="20" height="20" \
border="0"></span></span></a></span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.c \
om%2Fcompany%2Fvirgin-pulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea36 \
2ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227737929&sdata=NxQ3BmhgWo93uoQfJ1W7lLcDdSUQHgoXu1I49vzibwE%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m_4668523972764021174_x005f_x0000_i1026" \
src="cid:16c25a202c9855d355" alt="LinkedIn" width="20" height="20" \
border="0"></span></span></a></span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 6pt 2.25pt 0in" valign="top">
<p class="MsoNormal"><span style="font-size:1pt"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2F \
virginpulse&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d \
71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=sYUapxFqHq0LVyxO4I7kkwN1y9PG5ZLHd83gseRIxvM%3D&reserved=0" \
target="_blank"><span style="color:windowtext;text-decoration:none"><span \
style="color:blue"><img style="width: 0.2083in; height: 0.2083in;" \
id="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171gmail-m_4668523972764021174_x005f_x0000_i1025" \
src="cid:16c25a202c99374b66" alt="Twitter" width="20" height="20" \
border="0"></span></span></a></span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:6pt 0in 0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpuls \
e.com%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d710 \
71c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227747924&sdata=aoXH94QOjdngJXkAPcz9kmJAK5BA6c9rR5BP01lX0bw%3D&reserved=0" \
target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">virginpulse.com</span></a></span><u></u><u></u></p>
</td>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 3pt" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">|</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)"><a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.virginpuls \
e.com%2Fen-gb%2Fglobal-challenge%2F&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7 \
C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C6369959 \
53227757921&sdata=aUbx8uIlqldr5JtKT1PL05nbzcNPHONpouVkj1qXRdM%3D&reserved=0" \
title="Global Challenge" target="_blank"><span \
style="color:rgb(84,84,84);text-decoration:none">virginpulse.com/global-challenge</span></a></span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:2.25pt 0in 0in" valign="top">
<p class="MsoNormal"><span \
style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(84,84,84)">Australia | \
Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | \
United Kingdom | USA</span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:9pt 0in 7.5pt" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in">
<p class="MsoNormal" style="text-align:justify;line-height:7.5pt">
<span style="font-size:7pt;font-family:Arial,sans-serif;color:rgb(186,186,186)">Confidentiality \
Notice: The information contained in this e-mail, including any attachment(s), is \
intended solely for use by the designated recipient(s). Unauthorized use, \
dissemination, distribution, or reproduction of this message by anyone other than \
the intended recipient(s), or a person designated as responsible for delivering such \
messages to the intended recipient, is strictly prohibited and may be unlawful. This \
e-mail may contain proprietary, confidential or privileged information. Any views or \
opinions expressed are solely those of the author and do not necessarily represent \
those of Virgin Pulse, Inc. If you have received this message in error, or are not \
the named recipient(s), please immediately notify the sender and delete this e-mail \
message.</span><u></u><u></u></p> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td style="padding:0in" valign="top">
<table class="gmail-m_-5428940315049701298gmail-m_-7353443747219366048gmail-m_-2220776925344054171MsoNormalTable" \
cellspacing="0" cellpadding="0" border="0"> <tbody>
<tr>
<td style="padding:0in 0in 9pt" valign="top">
<p class="MsoNormal"><span \
style="font-size:7.5pt;font-family:Arial,sans-serif;color:rgb(186,186,186)">v2.59</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
From: Paul Emmerich <<a href="mailto:paul.emmerich@croit.io" \
target="_blank">paul.emmerich@croit.io</a>><br>
Date: Wednesday, July 24, 2019 at 1:39 PM<br>
To: Peter Eisch <<a href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>><br>
Cc: Xavier Trilla <<a href="mailto:xavier.trilla@clouding.io" \
target="_blank">xavier.trilla@clouding.io</a>>, "<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>" <<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>><br>
Subject: Re: [ceph-users] Upgrading and lost OSDs<br>
<br>
On Wed, Jul 24, 2019 at 8:36 PM Peter Eisch <mailto:<a \
href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>> wrote:<br> # lsblk<br>
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT<br>
sda 8:0 0 1.7T 0 disk <br>
<span style="font-family:"MS Gothic"">├</span>─sda1 8:1 0 100M 0 part \
<br> <span style="font-family:"MS Gothic"">├</span>─sda2 8:2 0 1.7T 0 \
part <br> └─sda5 8:5 0 10M 0 part <br>
sdb 8:16 0 1.7T 0 disk <br>
<span style="font-family:"MS Gothic"">├</span>─sdb1 8:17 0 100M 0 part \
<br> <span style="font-family:"MS Gothic"">├</span>─sdb2 8:18 0 1.7T 0 \
part <br> └─sdb5 8:21 0 10M 0 part <br>
sdc 8:32 0 1.7T 0 disk <br>
<span style="font-family:"MS Gothic"">├</span>─sdc1 8:33 0 100M 0 part \
<br> <br>
That's ceph-disk which was removed, run "ceph-volume simple scan"<br>
<br>
<br>
-- <br>
Paul Emmerich<br>
<br>
Looking for help with your Ceph cluster? Contact us at <a \
href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcroit.io&d \
ata=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7Cb123a \
16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227757921&sdata=gAFSElTqBRhu1zDIQYvWQ2WZtqqHoW%2FLa3stBfqXXHQ%3D&reserved=0" \
target="_blank"> https://nam02.safelinks.protection.outlook.com/?url=https://croit.io& \
amp;data=02|01|peter.eisch@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e89 \
2b4cf6a55a6f8c7606a035|0|0|636995903843215231&sdata=YEQI+UvikVPVeOFNSB2ikqVRiul8ElD3JEZDVOQI+NY=&reserved=0</a><br>
<br>
croit GmbH<br>
Freseniusstr. 31h<br>
81247 München<br>
<a href="https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.croit.io \
&data=02%7C01%7Cpeter.eisch%40virginpulse.com%7C25cea362ad224625423308d71071c968%7 \
Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995953227767913&sdata=LWJKCZ4VxuwoHMJGSWiEWAcawFPw7pDGC48%2B6bnXk6A%3D&reserved=0" \
target="_blank">https://nam02.safelinks.protection.outlook.com/?url=http://www.croit.i \
o&data=02|01|peter.eisch@virginpulse.com|93235ab7971a4beceab708d710664a14|b123a16e \
892b4cf6a55a6f8c7606a035|0|0|636995903843225224&sdata=83sD9wJHxE5W0renuDE7RGR/cPznR6jl9rEfl1AO+oA=&reserved=0</a><br>
Tel: +49 89 1896585 90<br>
<br>
<br>
...<br>
I'm thinking the OSD would start (I can recreate the .service definitions in \
systemctl) if the above were mounted in a way like they are on another of my \
hosts:<br> # lsblk<br>
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT<br>
sda 8:0 0 1.7T 0 disk <br>
<span style="font-family:"MS Gothic"">├</span>─sda1 8:1 0 100M 0 part \
<br> │ └─97712be4-1234-4acc-8102-2265769053a5 253:17 0 98M 0 crypt \
/var/lib/ceph/osd/ceph-16<br> <span style="font-family:"MS \
Gothic"">├</span>─sda2 8:2 0 1.7T 0 part <br> │ \
└─049b7160-1234-4edd-a5dc-fe00faca8d89 253:16 0 1.7T 0 crypt <br> └─sda5 8:5 \
0 10M 0 part /var/lib/ceph/osd-lockbox/97712be4-9674-4acc-1234-2265769053a5<br> sdb \
8:16 0 1.7T 0 disk <br> <span style="font-family:"MS \
Gothic"">├</span>─sdb1 8:17 0 100M 0 part <br> │ \
└─f03f0298-1234-42e9-8b28-f3016e44d1e2 253:26 0 98M 0 crypt \
/var/lib/ceph/osd/ceph-17<br> <span style="font-family:"MS \
Gothic"">├</span>─sdb2 8:18 0 1.7T 0 part <br> │ \
└─51177019-1234-4963-82d1-5006233f5ab2 253:30 0 1.7T 0 crypt <br> └─sdb5 8:21 \
0 10M 0 part /var/lib/ceph/osd-lockbox/f03f0298-1234-42e9-8b28-f3016e44d1e2<br> sdc \
8:32 0 1.7T 0 disk <br> <span style="font-family:"MS \
Gothic"">├</span>─sdc1 8:33 0 100M 0 part <br> │ \
└─0184df0c-1234-404d-92de-cb71b1047abf 253:8 0 98M 0 crypt \
/var/lib/ceph/osd/ceph-18<br> <span style="font-family:"MS \
Gothic"">├</span>─sdc2 8:34 0 1.7T 0 part <br> │ \
└─fdad7618-1234-4021-a63e-40d973712e7b 253:13 0 1.7T 0 crypt<br>
...<br>
<br>
Thank you for your time on this,<br>
<br>
peter<br>
<br>
From: Xavier Trilla <mailto:<a href="mailto:xavier.trilla@clouding.io" \
target="_blank">xavier.trilla@clouding.io</a>><br>
Date: Wednesday, July 24, 2019 at 1:25 PM<br>
To: Peter Eisch <mailto:<a href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>><br>
Cc: "mailto:<a href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>" <mailto:<a \
href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a>><br>
Subject: Re: [ceph-users] Upgrading and lost OSDs<br>
<br>
Hi Peter, <br>
<br>
Im not sure but maybe after some changes the OSDs are not being recongnized by ceph \
scripts.<br> <br>
Ceph used to use udev to detect the OSDs and then moved to lvm, which kind of OSDs \
are you running? Blustore or filestore? Which version did you use to create them?<br> \
<br> Cheers!<br>
<br>
El 24 jul 2019, a les 20:04, Peter Eisch <mailto:<a href="mailto:mailto" \
target="_blank">mailto</a>:<a href="mailto:peter.eisch@virginpulse.com" \
target="_blank">peter.eisch@virginpulse.com</a>> va escriure:<br> Hi,<br>
<br>
I'm working through updating from 12.2.12/luminious to 14.2.2/nautilus on centos 7.6. \
The managers are updated alright:<br> <br>
# ceph -s<br>
cluster:<br>
id: 2fdb5976-1234-4b29-ad9c-1ca74a9466ec<br>
health: HEALTH_WARN<br>
Degraded data redundancy: 24177/9555955 objects degraded (0.253%), \
7 pgs degraded, 1285 pgs undersized<br> 3 monitors have not enabled msgr2<br>
...<br>
<br>
I updated ceph on a OSD host with 'yum update' and then rebooted to grab the \
current kernel. Along the way, the contents of all the directories in \
/var/lib/ceph/osd/ceph-*/ were deleted. Thus I have 16 OSDs down from this. I can \
manage the undersized but I'd like to get these drives working again without \
deleting each OSD and recreating them.<br> <br>
So far I've pulled the respective cephx key into the 'keyring' file and \
populated 'bluestore' into the 'type' files but I'm unsure how to \
get the lockboxes mounted to where I can get the OSDs running. The osd-lockbox \
directory is otherwise untouched from when the OSDs were deployed.<br>
<br>
Is there a way to run ceph-deploy or some other tool to rebuild the mounts for the \
drives?<br> <br>
peter<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
ceph-users mailing list<br>
<a href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a><br> <a \
href="https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.ceph.com% \
2Flistinfo.cgi%2Fceph-users-ceph.com&data=02%7C01%7Cpeter.eisch%40virginpulse.com% \
7C25cea362ad224625423308d71071c968%7Cb123a16e892b4cf6a55a6f8c7606a035%7C0%7C0%7C636995 \
953227767913&sdata=jeAj%2FfzN%2BOG1NPFeZwqYnQiB4mgbgLGqq85tz99xBz8%3D&reserved=0" \
target="_blank">http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>
_______________________________________________<br>
ceph-users mailing list<br>
<a href="mailto:ceph-users@lists.ceph.com" \
target="_blank">ceph-users@lists.ceph.com</a><br> <a \
href="http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com" rel="noreferrer" \
target="_blank">http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com</a><br> \
</blockquote></div> </blockquote></div></div>
["image001.png" (image/png)]
["image002.png" (image/png)]
["image003.png" (image/png)]
["image004.png" (image/png)]
["image005.png" (image/png)]
["image006.png" (image/png)]
["image224118.png" (image/png)]
["image510473.png" (image/png)]
["image314360.png" (image/png)]
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic