[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] Migrating files doesn't free space on the OST
From: Jason Williams <jasonw () jhu ! edu>
Date: 2019-01-17 20:18:49
Message-ID: DM6PR01MB3929C2E530C7E63B504D7462D9830 () DM6PR01MB3929 ! prod ! exchangelabs ! com
[Download RAW message or body]
Chad hit the nail on the head. I thought about the fact that it was still =
deactivated yesterday but was afraid to reactivate it until I verified the =
space was free.
FWIW, the URL about handling full OSTs does not include the fact that the s=
pace will not be free until you reactivate the OST. It actually implies th=
e opposite.
http://wiki.lustre.org/Handling_Full_OSTs
--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jasonw@jhu.edu<mailto:jasonw@jhu.edu>
________________________________
From: Chad DeWitt <ccdewitt@uncc.edu>
Sent: Thursday, January 17, 2019 3:07 PM
To: Jason Williams
Cc: Alexander I Kulyavtsev; lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Migrating files doesn't free space on the OST
Hi Jason,
I do not know if this will help you or not, but I had a situation in 2.8.0 =
where an OST filled up and I marked it as disabled on the MDS:
lctl dl | grep osc
...Grab the device_id of the full OST and then deactivate it...
lctl --device device_id deactivate
IIRC, this allowed the data to be read, but deletes were not processed. Wh=
en I re-activated the OST, then the deletes were processed and space starte=
d clearing. I think you stated you had the OST deactivated. If you still =
do, try to reactive it.
lctl --device device_id activate
Once you reactivate the OST, the deletes will start processing within 10 - =
30 seconds... Just use lfs df -h to watch...
-cd
------------------------------------------------------------
Chad DeWitt, CISSP
UNC Charlotte | ITS =96 University Research Computing
9201 University City Blvd. | Charlotte, NC 28223
ccdewitt@uncc.edu<mailto:ccdewitt@uncc.edu> | www.uncc.edu
------------------------------------------------------------
If you are not the intended recipient of this transmission or a person resp=
onsible for delivering it to the intended recipient, any disclosure, copyin=
g, distribution, or other use of any of the information in this transmissio=
n is strictly prohibited. If you have received this transmission in error, =
please notify me immediately by reply email or by telephone at 704-687-7802=
. Thank you.
On Thu, Jan 17, 2019 at 2:38 PM Jason Williams <jasonw@jhu.edu<mailto:jason=
w@jhu.edu>> wrote:
Hello Alexander,
Thank you for your reply.
- We are not using zfs, it's an LDISKFS backing store, so no snapshots.
- I have re-run lfs getstripe to make sure the file is indeed moving
- I just looked for lfsck but I don't seem to have it. We are running 2.10=
.4 so I don't know what version that appeared in.
- I will try to have a look into the jobstats and see what I can find, but =
I made sure the files I moved were not in use when I moved them.
--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jasonw@jhu.edu<mailto:jasonw@jhu.edu>
________________________________
From: Alexander I Kulyavtsev <aik@fnal.gov<mailto:aik@fnal.gov>>
Sent: Thursday, January 17, 2019 12:56 PM
To: Jason Williams; lustre-discuss@lists.lustre.org<mailto:lustre-discuss@l=
ists.lustre.org>
Subject: Re: Migrating files doesn't free space on the OST
- you can re-run command to find files residing on ost to see if files are =
new or old.
- zfs may have snapshots if you ever did snapshots; it takes space.
- removing data or snapshots has some lag to release the blocks (tens of mi=
nutes) but I guess that is completed by now.
- there are can be orphan objects on OST if you had crashes. On older lustr=
e versions if the ost was emptied out you can mount underlying fs as ext4 o=
r zfs; set mount to readonly and browse ost objects - you may see if there =
are some orphan objects left. On newer lustre releases you probably can run=
lfsck (lustre scanner).
- to find what hosts / jobs currently writing to lustre you may enable lust=
re jobstats; clear counters and parse stats files in /proc . There was xlto=
p tool on github for older versions of lustre not having implemented jobsta=
ts but it was not updated for a while.
- depending on lustre version you have the implementation of lfs migrate is=
different. The older version copied file with other name to other ost, ren=
amed files and removed old file. If migration done on file open for write b=
y application the data will not be released until file closed (and data in =
new file are wrong). Recent implementation of migrate does swap of the file=
objects with file layout lock taken. I can not tell if it is safe for acti=
ve write.
- not releasing space can be a bug - did you check jira on whamcloud? What =
version of lustre do you have? Is it ldiskfs or zfs based? zfs version?
Alex.
________________________________
From: lustre-discuss <lustre-discuss-bounces@lists.lustre.org<mailto:lustre=
-discuss-bounces@lists.lustre.org>> on behalf of Jason Williams <jasonw@jhu=
.edu<mailto:jasonw@jhu.edu>>
Sent: Wednesday, January 16, 2019 10:25 AM
To: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
Subject: [lustre-discuss] Migrating files doesn't free space on the OST
I am trying to migrate files I know are not in use off of the full OST that=
I have using lfs migrate. I have verified up and down that the files I am=
moving are on that OST and that after the migrate lfs getstripe indeed sho=
ws they are no longer on that OST since it's disabled in the MDS.
The problem is, the used space on the OST is not going down.
I see one of at least two issues:
- the OST is just not freeing the space for some reason or another ( I don'=
t know)
- Or someone is writing to existing files just as fast as I am clearing the=
data (possible, but kind of hard to find)
Is there possibly something else I am missing? Also, does anyone know a goo=
d way to see if some client is writing to that OST and determine who it is =
if it's more probable that that is what is going on?
--
Jason Williams
Assistant Director
Systems and Data Center Operations.
Maryland Advanced Research Computing Center (MARCC)
Johns Hopkins University
jasonw@jhu.edu<mailto:jasonw@jhu.edu>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} \
--></style> </head>
<body dir="ltr">
<div id="divtagdefaultwrapper" \
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" \
dir="ltr"> <p style="margin-top:0;margin-bottom:0">Chad hit the nail on the \
head. I thought about the fact that it was still deactivated yesterday but was \
afraid to reactivate it until I verified the space was free.</p> <p \
style="margin-top:0;margin-bottom:0"><br> </p>
<p style="margin-top:0;margin-bottom:0">FWIW, the URL about handling full OSTs does \
not include the fact that the space will not be free until you reactivate the \
OST. It actually implies the opposite.</p> <p \
style="margin-top:0;margin-bottom:0"><br> </p>
<p style="margin-top:0;margin-bottom:0"><a \
href="http://wiki.lustre.org/Handling_Full_OSTs" class="OWAAutoLink" id="LPlnk293218" \
previewremoved="true">http://wiki.lustre.org/Handling_Full_OSTs</a><br> <br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); \
font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color \
Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", \
"Android Emoji", EmojiSymbols;"> <p style="margin-top:0; \
margin-bottom:0"><span id="ms-rterangepaste-start"></span></p> <div \
style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">--</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason \
Williams</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant \
Director</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data \
Center Operations.<br> Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" target="_blank" rel="noopener \
noreferrer" id="LPNoLP"><font \
color="#0563C1">jasonw@jhu.edu</font></a></span></font></div> <span \
id="ms-rterangepaste-end"></span><br> <p></p>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" \
style="font-size:11pt" color="#000000"><b>From:</b> Chad DeWitt \
<ccdewitt@uncc.edu><br> <b>Sent:</b> Thursday, January 17, 2019 3:07 PM<br>
<b>To:</b> Jason Williams<br>
<b>Cc:</b> Alexander I Kulyavtsev; lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Migrating files doesn't free space on the \
OST</font> <div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Hi Jason,
<div><br>
</div>
<div>I do not know if this will help you or not, but I had a situation in 2.8.0 where \
an OST filled up and I marked it as disabled on the MDS:</div> <div><br>
</div>
<div><span style="font-family:"Courier New"">lctl dl | grep \
osc</span> </div> <div>...Grab the <i>device_id</i> of the full OST and \
then deactivate it...</div> <div><span style="font-family:"Courier \
New"">lctl --device </span><i style="font-family:"Courier \
New"">device_id</i><span style="font-family:"Courier New""> \
deactivate</span></div> <div><br>
</div>
<div>IIRC, this allowed the data to be read, but deletes were not processed. \
When I re-activated the OST, then the deletes were processed and space started \
clearing. I think you stated you had the OST deactivated. If you still \
do, try to reactive it. </div> <div><br>
</div>
<div><font face="monospace, monospace">lctl --device <i>device_id</i> \
activate</font><br> </div>
<div><br>
</div>
<div>Once you reactivate the OST, the deletes will start processing within 10 - 30 \
seconds... Just use <font face="monospace, monospace">lfs df -h</font> to \
watch...</div> <div><br>
</div>
<div>-cd</div>
<div> <br clear="all">
<div>
<div dir="ltr" class="x_m_-5195253070295309020gmail-m_4138295817730096550m_3313866458073280806gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p style="margin:0in 0in 0.0001pt"><span \
style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, \
monospace">------------------------------------------------------------<u></u><u></u></font></span></p>
<p style="margin:0in 0in 0.0001pt"><span \
style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, \
monospace">Chad DeWitt, CISSP</font></span></p> <p style="margin:0in 0in \
0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" \
face="monospace, monospace">UNC Charlotte <b>| </b>ITS – University \
Research Computing</font></span></p> <p style="margin:0in 0in 0.0001pt"><font \
color="#000000" size="2" face="monospace, monospace"><span \
style="background-color:rgba(255,255,255,0)"><a style="color:rgb(34,34,34)" \
id="LPlnk498623" class="OWAAutoLink" previewremoved="true">9201 University City \
Blvd. </a><b><a style="color:rgb(34,34,34)" id="LPlnk442622" class="OWAAutoLink" \
previewremoved="true">| </a></b><a style="color:rgb(34,34,34)" id="LPlnk295528" \
class="OWAAutoLink" previewremoved="true">Charlotte, NC 28223</a></span></font></p>
<p style="margin:0in 0in 0.0001pt"><font color="#000000" size="2" face="monospace, \
monospace"><span style="background-color:rgba(255,255,255,0)"><a \
href="mailto:ccdewitt@uncc.edu" target="_blank" style="color:rgb(17,85,204)" \
id="LPlnk295655" class="OWAAutoLink" \
previewremoved="true">ccdewitt@uncc.edu</a> <b>| </b><a \
style="color:rgb(34,34,34)" id="LPlnk329507" class="OWAAutoLink" \
previewremoved="true">www.uncc.edu</a><u></u><u></u></span></font></p> <p \
style="margin:0in 0in 0.0001pt"><span \
style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, \
monospace">------------------------------------------------------------</font></span></p>
<p style="margin:0in 0in 0.0001pt"><span \
style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, \
monospace"><br> </font></span></p>
<p style="margin:0in 0in 0.0001pt">If you are not \
the <span>intended</span> recipient of this transmission or a person \
responsible for delivering it to the <span>intended</span> recipient, any \
disclosure, copying, distribution, or other use of any of the information in this \
transmission is strictly prohibited. If you have received this transmission in error, \
please notify me immediately by reply email or by telephone at 704-687-7802. Thank \
you.<span style="background-color:rgba(255,255,255,0)"><font size="2" \
face="monospace, monospace"><br> </font></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail_attr">
On Thu, Jan 17, 2019 at 2:38 PM Jason Williams <<a href="mailto:jasonw@jhu.edu" \
target="_blank" id="LPlnk603160" class="OWAAutoLink" \
previewremoved="true">jasonw@jhu.edu</a>> wrote:<br> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px \
solid rgb(204,204,204); padding-left:1ex"> <div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divtagdefaultwrapper" \
dir="ltr" style="font-size:12pt; color:rgb(0,0,0); \
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color \
Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI \
Symbol","Android Emoji",EmojiSymbols"> <p style="margin-top:0px; \
margin-bottom:0px">Hello Alexander,</p> <p style="margin-top:0px; \
margin-bottom:0px"><br> </p>
<p style="margin-top:0px; margin-bottom:0px">Thank you for your \
reply. <br> <br>
- We are not using zfs, it's an LDISKFS backing store, so no snapshots.</p>
<p style="margin-top:0px; margin-bottom:0px">- I have re-run lfs getstripe to \
make sure the file is indeed moving</p> <p style="margin-top:0px; \
margin-bottom:0px">- I just looked for lfsck but I don't seem to have it. We \
are running 2.10.4 so I don't know what version that appeared in.</p> <p \
style="margin-top:0px; margin-bottom:0px">- I will try to have a look into the \
jobstats and see what I can find, but I made sure the files I moved were not in use \
when I moved them. </p> <p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830Signature">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divtagdefaultwrapper" \
dir="ltr" style="font-size:12pt; color:rgb(0,0,0); \
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color \
Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI \
Symbol","Android Emoji",EmojiSymbols"> <p style="margin-top:0px; \
margin-bottom:0px"><span \
id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">--</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason \
Williams</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant \
Director</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data \
Center Operations.<br> Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" rel="noopener noreferrer" \
id="LPlnk754022" target="_blank" class="OWAAutoLink" previewremoved="true"><font \
color="#0563C1">jasonw@jhu.edu</font></a></span></font></div> <span \
id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830divRplyFwdMsg" \
dir="ltr"> <font face="Calibri, sans-serif" color="#000000" \
style="font-size:11pt"><b>From:</b> Alexander I Kulyavtsev <<a \
href="mailto:aik@fnal.gov" target="_blank" id="LPlnk36318" class="OWAAutoLink" \
previewremoved="true">aik@fnal.gov</a>><br> <b>Sent:</b> Thursday, January 17, \
2019 12:56 PM<br> <b>To:</b> Jason Williams; <a \
href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk661491" \
class="OWAAutoLink" previewremoved="true"> lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: Migrating files doesn't free space on the OST</font>
<div> </div>
</div>
<div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_divtagdefaultwrapper" \
dir="ltr" style="font-size:12pt; color:rgb(0,0,0); \
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color \
Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI \
Symbol","Android Emoji",EmojiSymbols"> <p style="margin-top:0px; \
margin-bottom:0px">- you can re-run command to find files residing on ost to see if \
files are new or old.</p> <p style="margin-top:0px; margin-bottom:0px">- zfs may \
have snapshots if you ever did snapshots; it takes space.</p> <p \
style="margin-top:0px; margin-bottom:0px">- removing data or snapshots has some lag \
to release the blocks (tens of minutes) but I guess that is completed by now.</p> <p \
style="margin-top:0px; margin-bottom:0px">- there are can be orphan objects on OST if \
you had crashes. On older lustre versions if the ost was emptied out you can \
mount underlying fs as ext4 or zfs; set mount to readonly and browse ost objects - \
you may see if there are some orphan objects left. On newer lustre releases you \
probably can run lfsck (lustre scanner).</p> <p style="margin-top:0px; \
margin-bottom:0px">- to find what hosts / jobs currently writing to lustre you \
may enable lustre jobstats; clear counters and parse stats files in /proc . There was \
xltop tool on github for older versions of lustre not having implemented jobstats \
but it was not updated for a while.</p> <p style="margin-top:0px; \
margin-bottom:0px">- depending on lustre version you have the implementation of lfs \
migrate is different. The older version copied file with other name to other ost, \
renamed files and removed old file. If migration done on file open for write by \
application the data will not be released until file closed (and data in new file are \
wrong). Recent implementation of migrate does swap of the file objects with file \
layout lock taken. I can not tell if it is safe for active write.</p> <p \
style="margin-top:0px; margin-bottom:0px">- not releasing space can be a bug - did \
you check jira on whamcloud? <span style="font-size:12pt">What version of lustre \
do you have? Is it ldiskfs or zfs based? </span><span style="font-size:12pt">zfs \
version?</span></p> <p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Alex.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_divRplyFwdMsg" \
dir="ltr"> <font face="Calibri, sans-serif" color="#000000" \
style="font-size:11pt"><b>From:</b> lustre-discuss <<a \
href="mailto:lustre-discuss-bounces@lists.lustre.org" target="_blank" \
id="LPlnk681351" class="OWAAutoLink" \
previewremoved="true">lustre-discuss-bounces@lists.lustre.org</a>> on behalf of \
Jason Williams <<a href="mailto:jasonw@jhu.edu" target="_blank" id="LPlnk351518" \
class="OWAAutoLink" previewremoved="true">jasonw@jhu.edu</a>><br> <b>Sent:</b> \
Wednesday, January 16, 2019 10:25 AM<br> <b>To:</b> <a \
href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk460775" \
class="OWAAutoLink" previewremoved="true"> lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> [lustre-discuss] Migrating files doesn't free space on the OST</font>
<div> </div>
</div>
<div dir="ltr">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_divtagdefaultwrapper" \
dir="ltr" style="font-size:12pt; color:rgb(0,0,0); \
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color \
Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI \
Symbol","Android Emoji",EmojiSymbols"> <p style="margin-top:0px; \
margin-bottom:0px">I am trying to migrate files I know are not in use off of the full \
OST that I have using lfs migrate. I have verified up and down that the files I \
am moving are on that OST and that after the migrate lfs getstripe indeed shows they \
are no longer on that OST since it's disabled in the MDS.</p> <p \
style="margin-top:0px; margin-bottom:0px"><br> </p>
<p style="margin-top:0px; margin-bottom:0px">The problem is, the used space on the \
OST is not going down.</p> <p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">I see one of at least two issues:</p>
<p style="margin-top:0px; margin-bottom:0px">- the OST is just not freeing the space \
for some reason or another ( I don't know)</p> <p style="margin-top:0px; \
margin-bottom:0px">- Or someone is writing to existing files just as fast as I am \
clearing the data (possible, but kind of hard to find)</p> <p style="margin-top:0px; \
margin-bottom:0px"><br> </p>
<p style="margin-top:0px; margin-bottom:0px">Is there possibly something else I am \
missing? Also, does anyone know a good way to see if some client is writing to \
that OST and determine who it is if it's more probable that that is what is going \
on?</p> <p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_Signature">
<div id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_divtagdefaultwrapper" \
dir="ltr" style="font-size:12pt; color:rgb(0,0,0); \
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color \
Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI \
Symbol","Android Emoji",EmojiSymbols"> <p style="margin-top:0px; \
margin-bottom:0px"><span \
id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_ms-rterangepaste-start"></span></p>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt">--</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Jason \
Williams</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Assistant \
Director</span></font></div> <div style="color:rgb(33,33,33); \
font-family:wf_segoe-ui_normal,"Segoe UI","Segoe \
WP",Tahoma,Arial,sans-serif,serif,EmojiFont; font-size:15px; margin:0px"> <font \
face="Calibri,sans-serif" size="2"><span style="font-size:11pt">Systems and Data \
Center Operations.<br> Maryland Advanced Research Computing Center (MARCC)<br>
Johns Hopkins University</span></font></div>
<div style="color:rgb(33,33,33); font-family:wf_segoe-ui_normal,"Segoe \
UI","Segoe WP",Tahoma,Arial,sans-serif,serif,EmojiFont; \
font-size:15px; margin:0px"> <font face="Calibri,sans-serif" size="2"><span \
style="font-size:11pt"><a href="mailto:jasonw@jhu.edu" rel="noopener noreferrer" \
id="LPlnk422296" target="_blank" class="OWAAutoLink" previewremoved="true"><font \
color="#0563C1">jasonw@jhu.edu</font></a></span></font></div> <span \
id="x_gmail-m_-5195253070295309020gmail-m_4138295817730096550gmail-m_3313866458073280806gmail-m_-8385819341340419830x_x_ms-rterangepaste-end"></span><br>
<p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk905013" \
class="OWAAutoLink" previewremoved="true">lustre-discuss@lists.lustre.org</a><br> <a \
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" \
rel="noreferrer" target="_blank" id="LPlnk726178" class="OWAAutoLink" \
previewremoved="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--===============8070332819820339342==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic