[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] OST still has inodes and size after deleting all files
From: Pavlo Khmel via lustre-discuss <lustre-discuss () lists ! lustre ! org>
Date: 2024-01-20 17:01:01
Message-ID: 7BFAB39D-C5BC-4EC2-ACC8-3404199B838C () gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thank you. I checked all those files with "ll_decode_filter_fid" and "lfs fid2path":
On OSS servers:
# umount /lustre/ost01
# mount -t ldiskfs /dev/mapper/ost01 /mnt/
# find /mnt/O/0/d*/ -type f
/mnt/O/0/d11/450605003
/mnt/O/0/d12/450605484
. . .
# ll_decode_filter_fid /mnt/O/0/d11/450605003
/mnt/O/0/d11/450605003: parent=[0x200019425:0x733f:0x0] stripe=0 stripe_size=1048576 \
stripe_count=1 layout_version=0 range=0
# umount /mnt
# mount -t lustre /dev/mapper/ost01 /lustre/ost01
On client:
# lfs fid2path /cluster [0x200019425:0x733f:0x0]
lfs fid2path: cannot find /cluster [0x200019425:0x733f:0x0]: No such file or \
directory
I got a reply "No such file or directory" for all those unknown files.
So I can assume those are "stray objects from deleted files".
Best regards,
Pavlo Khmel
> On 20 Jan 2024, at 04:45, Andreas Dilger <adilger@whamcloud.com> wrote:
>
>
>
> > On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss \
> > <lustre-discuss@lists.lustre.org <mailto:lustre-discuss@lists.lustre.org>> wrote: \
> > Hi,
> >
> > I'm trying to remove 4 OSTs.
> >
> > # lfs osts
> > OBDS:
> > 0: cluster-OST0000_UUID ACTIVE
> > 1: cluster-OST0001_UUID ACTIVE
> > 2: cluster-OST0002_UUID ACTIVE
> > 3: cluster-OST0003_UUID ACTIVE
> > . . .
> >
> > I moved all files to other OSTs. "lfs find" cannot find any files on these 4 \
> > OSTs.
> > # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster
> >
> > real 936m8.528s
> > user 13m48.298s
> > sys 210m1.245s
> >
> > But still: 2624 inods are in use and 14.5G total size.
> >
> > # lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
> > cluster-OST0000_UUID 4293438576 644 4293437932 1% /cluster[OST:0]
> > cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1]
> > cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2]
> > cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3]
> >
> > # lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003
> > cluster-OST0000_UUID 29.2T 3.8G 27.6T 1% /cluster[OST:0]
> > cluster-OST0001_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:1]
> > cluster-OST0002_UUID 29.2T 3.3G 27.6T 1% /cluster[OST:2]
> > cluster-OST0003_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:3]
> >
> > I tried to check the file-system for errors:
> >
> > # umount /lustre/ost01
> > # e2fsck -fy /dev/mapper/ost01
> >
> > and
> >
> > # lctl lfsck_start --device cluster-OST0001
> > # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub
> > . . .
> > status: completed
> >
> > I tried to mount OST as ldiskfs and there are several files in /O/0/d*/
> >
> > # umount /lustre/ost01
> > # mount -t ldiskfs /dev/mapper/ost01 /mnt/
> > # ls -Rhl /mnt/O/0/d*/
> > . . .
> > /mnt/O/0/d11/:
> > -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003
> > /mnt/O/0/d12/:
> > -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028
> > -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484
> > . . .
> >
> > Is it expected behavior? Is it save to delere OST even with those files?
>
>
> You can run the debugfs "stat" command to print the "fid" xattr and it will print \
> the MDT parent FID for use with "lfs fid2path" on the client to see if there are \
> any files related to these objects. You could also run "ll_decode_filter_fid" to \
> do the same thing on the mounted ldiskfs filesystem.
>
> It is likely that there are a few stray objects from deleted files, but hard to say \
> for sure.
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>
[Attachment #5 (unknown)]
<html><head><meta http-equiv="content-type" content="text/html; \
charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: \
space; line-break: after-white-space;"><div>Thank you. I checked all those files with \
"ll_decode_filter_fid" and "lfs fid2path":</div><div><br></div><div>On OSS \
servers:</div><div><br></div><div># umount /lustre/ost01</div><div># mount -t ldiskfs \
/dev/mapper/ost01 /mnt/</div><div># find /mnt/O/0/d*/ -type \
f</div><div>/mnt/O/0/d11/450605003</div><div>/mnt/O/0/d12/450605484</div><div>. . \
.</div><div><br></div><div># ll_decode_filter_fid \
/mnt/O/0/d11/450605003</div><div>/mnt/O/0/d11/450605003: \
parent=[0x200019425:0x733f:0x0] stripe=0 stripe_size=1048576 stripe_count=1 \
layout_version=0 range=0</div><div><br></div><div># umount /mnt</div><div># mount -t \
lustre /dev/mapper/ost01 /lustre/ost01</div><div><br></div><div>On \
client:</div><div><br></div><div># lfs fid2path /cluster \
[0x200019425:0x733f:0x0]</div><div>lfs fid2path: cannot find /cluster \
[0x200019425:0x733f:0x0]: No such file or directory</div><div><br></div><div>I got a \
reply "No such file or directory" for all those unknown files.</div><div>So I can \
assume those are "stray objects from deleted files".</div><div><br></div><div>Best \
regards,</div><div>Pavlo Khmel</div><div><br><blockquote type="cite"><div>On 20 Jan \
2024, at 04:45, Andreas Dilger <adilger@whamcloud.com> wrote:</div><br \
class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;"> <br>
<div><br>
<blockquote type="cite">
<div>On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss <<a \
href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> \
wrote:</div> <br class="Apple-interchange-newline">
<div>
<div>Hi,<br>
<br>
I'm trying to remove 4 OSTs. <br>
<br>
# lfs osts<br>
OBDS:<br>
0: cluster-OST0000_UUID ACTIVE<br>
1: cluster-OST0001_UUID ACTIVE<br>
2: cluster-OST0002_UUID ACTIVE<br>
3: cluster-OST0003_UUID ACTIVE<br>
. . .<br>
<br>
I moved all files to other OSTs. "lfs find" cannot find any files on these 4 \
OSTs.<br> <br>
# time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster<br>
<br>
real<span class="Apple-tab-span" style="white-space:pre"> </span>936m8.528s<br>
user<span class="Apple-tab-span" style="white-space:pre"> </span>13m48.298s<br>
sys<span class="Apple-tab-span" style="white-space:pre"> </span>210m1.245s<br>
<br>
But still: 2624 inods are in use and 14.5G total size.<br>
<br>
# lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br>
cluster-OST0000_UUID 4293438576 \
644 4293437932 1% \
/cluster[OST:0]<br> cluster-OST0001_UUID 4293438576 \
640 4293437936 1% \
/cluster[OST:1]<br> cluster-OST0002_UUID 4293438576 \
671 4293437905 1% \
/cluster[OST:2]<br> cluster-OST0003_UUID 4293438576 \
669 4293437907 1% \
/cluster[OST:3]<br> <br>
# lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003<br>
cluster-OST0000_UUID 29.2T \
3.8G \
27.6T 1% /cluster[OST:0]<br> \
cluster-OST0001_UUID 29.2T \
3.7G \
27.6T 1% /cluster[OST:1]<br> \
cluster-OST0002_UUID 29.2T \
3.3G \
27.6T 1% /cluster[OST:2]<br> \
cluster-OST0003_UUID 29.2T \
3.7G \
27.6T 1% /cluster[OST:3]<br> <br>
I tried to check the file-system for errors:<br>
<br>
# umount /lustre/ost01<br>
# e2fsck -fy /dev/mapper/ost01<br>
<br>
and<br>
<br>
# lctl lfsck_start --device cluster-OST0001<br>
# lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub<br>
. . .<br>
status: completed<br>
<br>
I tried to mount OST as ldiskfs and there are several files in /O/0/d*/<br>
<br>
# umount /lustre/ost01<br>
# mount -t ldiskfs /dev/mapper/ost01 /mnt/<br>
# ls -Rhl /mnt/O/0/d*/<br>
. . .<br>
/mnt/O/0/d11/:<br>
-rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003<br>
/mnt/O/0/d12/:<br>
-rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028<br>
-rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484<br>
. . .<br>
<br>
Is it expected behavior? Is it save to delere OST even with those files?<br>
</div>
</div>
</blockquote>
</div>
<div><br>
</div>
You can run the debugfs "stat" command to print the "fid" xattr and it will print the \
MDT <div>parent FID for use with "lfs fid2path" on the client to see if there are any \
files related</div> <div>to these objects. You could also run \
"ll_decode_filter_fid" to do the same thing on the</div> <div>mounted ldiskfs \
filesystem.</div> <div><br>
</div>
<div>It is likely that there are a few stray objects from deleted files, but hard to \
say for sure.</div> <div><br>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"> <div>Cheers, \
Andreas</div> <div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br>
</div>
</div>
</div></blockquote></div><br></body></html>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic