[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] Re: Multipath and cephadm
From:       Thomas Roth <t.roth () gsi ! de>
Date:       2022-01-31 7:55:58
Message-ID: f481bc13-bcbc-b6f7-4934-0252f253c368 () gsi ! de
[Download RAW message or body]

Thanks, Peter, this works.

Before, I had the impression cephadm would only accept 'bare' disks as osd devices, \
but indeed it will swallow any kind of block device or LV that you  prepare for it on \
the osd host.

Regards,
Thomas

On 1/25/22 20:21, Peter Childs wrote:
> This came from a previous thread that I started last year, so you may want
> to look in the archive.
> 
> https://www.mail-archive.com/ceph-users@ceph.io/msg11572.html
> 
> Although the doc page it refers to looks to have disappeared :(
> 
> You can use "ceph orch daemon add osd <host>:<path to multipath device>"
> 
> I've been using
> 
> 1=mpatha
> 
> pvcreate /dev/mapper/$1
> vgcreate $1-vg /dev/mapper/$1
> lvcreate -l 100%FREE -n $1-lv $1-vg
> ceph orch daemon add osd dampwood48:$1-vg/$1-lv
> 
> to create osd's on the multipath devices terrible dev opt script but it
> works.
> 
> I've not currently got a method to use the yaml device description method
> (which would be much more ideal),  hence there is no obvious way to use
> separate db_devices, but this does look to work for me as far as it goes.
> 
> Hope that helps
> 
> Peter Childs
> 
> 
> 
> 
> On Tue, 25 Jan 2022, 17:53 Thomas Roth, <t.roth@gsi.de> wrote:
> 
> > Would like to know that as well.
> > 
> > I have the same setup - cephadm, Pacific, CentOS8, and a host with a
> > number of HDDs which are all connect by 2 paths.
> > No way to use these without multipath
> > 
> > > ceph orch daemon add osd serverX:/dev/sdax
> > 
> > > Cannot update volume group ceph-51f8b9b0-2917-431d-8a6d-8ff90440641b
> > with duplicate PV devices
> > 
> > (because sdax == sdce, etc.)
> > 
> > and with multipath, it fails with
> > 
> > > ceph orch daemon add osd serverX:/dev/mapper/mpathbq
> > 
> > > podman: stderr -->  IndexError: list index out of range
> > 
> > 
> > Quite strange that the 'future of storage' does not know how to handle
> > multipath devices?
> > 
> > Regrads,
> > Thomas
> > 
> > 
> > On 12/23/21 18:40, Michal Strnad wrote:
> > > Hi all.
> > > 
> > > We have problem using disks accessible via multipath. We are using
> > cephadm for deployment, Pacific version for containers, CentOS 8 Stream on
> > servers
> > > and following LVM configuration.
> > > 
> > > devices {
> > > multipath_component_detection = 1
> > > }
> > > 
> > > 
> > > 
> > > We tried several methods.
> > > 
> > > 1.) Direct approach.
> > > 
> > > cephadm shell
> > 
> > /mapper/mpatha
> > > 
> > > Errors are attached in 1.output file.
> > > 
> > > 
> > > 
> > > 2.  With the help of OSD specifications where they are mpathX devices
> > used.
> > > 
> > > service_type: osd
> > > service_id: osd-spec-serverX
> > > placement:
> > > host_pattern: 'serverX'
> > > spec:
> > > data_devices:
> > > paths:
> > > - /dev/mapper/mpathaj
> > > - /dev/mapper/mpathan
> > > - /dev/mapper/mpatham
> > > db_devices:
> > > paths:
> > > - /dev/sdc
> > > encrypted: true
> > > 
> > > Errors are attached in 2.output file.
> > > 
> > > 
> > > 2.  With the help of OSD specifications where they are dm-X devices used.
> > > 
> > > service_type: osd
> > > service_id: osd-spec-serverX
> > > placement:
> > > host_pattern: 'serverX'
> > > spec:
> > > data_devices:
> > > paths:
> > > - /dev/dm-1
> > > - /dev/dm-2
> > > - /dev/dm-3
> > > - /dev/dm-X
> > > db_devices:
> > > size: ':2TB'
> > > encrypted: true
> > > 
> > > Errors are attached in 3.output file.
> > > 
> > > What is the right method for multipath deployments? I didn't find much
> > on this topic.
> > > 
> > > Thank you
> > > 
> > > Michal
> > > 
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-leave@ceph.io
> > 
> > 
> > --
> > --------------------------------------------------------------------
> > Thomas Roth
> > HPC Department
> > 
> > GSI Helmholtzzentrum für Schwerionenforschung GmbH
> > Planckstr. 1, 64291 Darmstadt, http://www.gsi.de/
> > 
> > Gesellschaft mit beschraenkter Haftung
> > 
> > Sitz der Gesellschaft / Registered Office:                    Darmstadt
> > Handelsregister       / Commercial Register:
> > Amtsgericht Darmstadt, HRB 1528
> > 
> > Geschaeftsfuehrung    / Managing Directors:
> > Professor Dr. Paolo Giubellino, Ursula Weyrich, Jörg Blaurock
> > 
> > Vorsitzender des GSI-Aufsichtsrates /
> > Chairman of the Supervisory Board:
> > Staatssekretaer / State Secretary Dr. Georg Schütte
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-leave@ceph.io
> > 
> 
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io


[prev in list] [next in list] [prev in thread] [next in thread]