[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] Re: how to handle rgw leaked data (aka data that is not available via buckets but eats 
From:       Boris Behrens <bb () kervyn ! de>
Date:       2021-04-27 14:41:15
Message-ID: CAHjiyFQHnPK8fMccuwaRSW6=1Ki2V1ABz4SfkyXKtkwNdoZNLA () mail ! gmail ! com
[Download RAW message or body]

So,
maybe somebody can answer me the following question:

I have ~150m objects in the ceph cluster (ceph status shows (objects:
152.68M objects, 316 TiB).
How can
radosgw-admin bucket --bucket BUCKET radoslist
create an output that is 252677729 and still growing?

Am Di., 27. Apr. 2021 um 06:59 Uhr schrieb Boris Behrens <bb@kervyn.de>:

> Hi Anthony,
>
> yes we are using replication, the lost space is calculated before it's
> replicated.
> RAW STORAGE:
>     CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
>     hdd       1.1 PiB     191 TiB     968 TiB      968 TiB         83.55
>     TOTAL     1.1 PiB     191 TiB     968 TiB      968 TiB         83.55
>
> POOLS:
>     POOL                                ID     PGS      STORED
>  OBJECTS     USED        %USED     MAX AVAIL
>     rbd                                  0       64         0 B
> 0         0 B         0        13 TiB
>     .rgw.root                            1       64      99 KiB
> 119      99 KiB         0        13 TiB
>     eu-central-1.rgw.control             2       64         0 B
> 8         0 B         0        13 TiB
>     eu-central-1.rgw.data.root           3       64     947 KiB
> 2.82k     947 KiB         0        13 TiB
>     eu-central-1.rgw.gc                  4       64     101 MiB
> 128     101 MiB         0        13 TiB
>     eu-central-1.rgw.log                 5       64     267 MiB
> 500     267 MiB         0        13 TiB
>     eu-central-1.rgw.users.uid           6       64     2.9 MiB
> 6.91k     2.9 MiB         0        13 TiB
>     eu-central-1.rgw.users.keys          7       64     263 KiB
> 6.73k     263 KiB         0        13 TiB
>     eu-central-1.rgw.meta                8       64     384 KiB
>  1k     384 KiB         0        13 TiB
>     eu-central-1.rgw.users.email         9       64        40 B
> 1        40 B         0        13 TiB
>     eu-central-1.rgw.buckets.index      10       64      10 GiB
>  67.28k      10 GiB      0.03        13 TiB
>     eu-central-1.rgw.buckets.data       11     2048     313 TiB
> 151.71M     313 TiB     89.25        13 TiB
> ...
>
> EC profile is pretty standard
> [root@s3db16 ~]# ceph osd erasure-code-profile ls
> default
> [root@s3db16 ~]# ceph osd erasure-code-profile get default
> k=2
> m=1
> plugin=jerasure
> technique=reed_sol_van
>
> We use mainly ceph 14.2.18. There is an OSD host with 14.2.19 and one with
> 14.2.20
>
> Object populations is mixed, but the most amount of data is in huge files.
> We store our platforms RBD snapshots in it.
>
> Cheers
>  Boris
>
>
> Am Di., 27. Apr. 2021 um 06:49 Uhr schrieb Anthony D'Atri <
> anthony.datri@gmail.com>:
>
>> Are you using Replication?  EC? How many copies / which profile?
>> On which Ceph release were your OSDs built?  BlueStore? Filestore?
>> What is your RGW object population like?  Lots of small objects?  Mostly
>> large objects?  Average / median object size?
>>
>> > On Apr 26, 2021, at 9:32 PM, Boris Behrens <bb@kervyn.de> wrote:
>> >
>> > HI,
>> >
>> > we still have the problem that our rgw eats more diskspace than it
>> should.
>> > Summing up the "size_kb_actual" of all buckets show only half of the
>> used
>> > diskspace.
>> >
>> > There are 312TiB stored acording to "ceph df" but we only need around
>> 158TB.
>> >
>> > I've already wrote to this ML with the problem, but there were no
>> solutions
>> > that would help.
>> > I've doug through the ML archive and found some interesting threads
>> > regarding orphan objects and these kind of issues.
>> >
>> > Did someone ever solved this problem?
>> > Or do you just add more disk space.
>> >
>> > we tried to:
>> > * use the "radosgw-admin orphan find/finish" tool (didn't work)
>> > * manually triggering the GC (didn't work)
>> >
>> > currently running (since yesterday evening):
>> > * rgw-orphan-list, which procused 270GB of text output, and it's not
>> done
>> > yet (I have 60GB diskspace left)
>> >
>> > --
>> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
>> > groà ¼en Saal.
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-leave@ceph.io
>>
>>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groà ¼en Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groà ¼en Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io

[prev in list] [next in list] [prev in thread] [next in thread]