[prev in list] [next in list] [prev in thread] [next in thread]
List: toasters
Subject: AW: Dealing with a really large number of files in a single directory
From: Alexander Griesser <AGriesser () anexia-it ! com>
Date: 2016-03-17 10:35:23
Message-ID: a0a7fceaa6f9483c939300d96f793f0c () anx-i-dag02 ! anx ! local
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hi Edward,
BTDT many times unfortunately.
The volume option „maxdirsize" is what's usually limiting you in creating \
directories of that size, so what's maxdirsize set on your volume? You can check it \
with `vol options` if that's a 7-mode filer.
In order to delete this directory, you will only have a few sane choices.
One that usually works good is to rsync an empty directory over it, so f.ex.:
/nfs/reallybigdir <- this is the victim
mkdir /nfs/emptydir
rsync –a –delete /nfs/emptydir/ /nfs/reallybigdir/
Let this run for a while and make sure that you don't omit the trailing slashes of \
the directories. That of course is only valid if you want to get rid of all the files \
in this directory, if you want to keep some of them and need to find out the names of \
the files, you could try to use `find /nfs/reallybigdir –type f –print` and \
redirect that output to a file, but it will also take very long I suppose. You can \
check if this command would actually work by limiting it's output to let's say 10 \
files, by running:
find /nfs/reallybigdir –type f | head
If that completes in time, you can increase the pagination by passing arguments to \
head, f.ex.
find /nfs/reallybigdir –type f | head -2000
`rm –r /nfs/reallybigdir` usually also works pretty good, but takes notibly more \
resources on the host OS than the rsync variant.
If all of that doesn't help, I've a C program code which I used to delete 1023MB big \
directories in the past which I could share then.
Best,
Alexander Griesser
Head of Systems Operations
ANEXIA Internetdienstleistungs GmbH
E-Mail: AGriesser@anexia-it.com<mailto:AGriesser@anexia-it.com>
Web: http://www.anexia-it.com<http://www.anexia-it.com/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Von: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] Im Auftrag \
von Edward Rolison
Gesendet: Donnerstag, 17. März 2016 11:27
An: toasters@teaparty.net
Betreff: Dealing with a really large number of files in a single directory
I've got a directory on an NFS server. I think it's been filling up with log files, \
and may be in excess of a million. (ls -d shows me a 350MB rather than the usual 4k).
strace -fTt ls -1 -f is stalling on 'getdents' system call. I assume it'll complete \
eventually, but ... well, the 'leave it and wait' hasn't yet.
Does anyone have any good tricks on a filer for helping me tell that a) this \
directory is in fact, absurdly large and b) starting to retrieve filenames so I can \
delete/move them in sensibly sized baches?
(I'm wondering if a filer-side ls is going to help or hinder, for example).
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi \
Edward,<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">BTDT \
many times unfortunately.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The \
volume option „maxdirsize" is what's usually limiting you in creating directories \
of that size, so what's maxdirsize set on your volume? You can check it with `vol \
options` if that's a 7-mode filer.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">In \
order to delete this directory, you will only have a few sane \
choices.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">One \
that usually works good is to rsync an empty directory over it, so \
f.ex.:<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-f \
areast-language:EN-US">/nfs/reallybigdir \
<- this is the victim<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">mkdir \
/nfs/emptydir<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">rsync \
–a –delete /nfs/emptydir/ /nfs/reallybigdir/<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Let \
this run for a while and make sure that you don't omit the trailing slashes of the \
directories.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">That \
of course is only valid if you want to get rid of all the files in this directory, if \
you want to keep some of them and need to find out the names of the files, you could \
try to use `find /nfs/reallybigdir –type f –print` and redirect that output to a \
file, but it will also take very long I suppose.<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">You \
can check if this command would actually work by limiting it's output to let's say 10 \
files, by running:<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">find \
/nfs/reallybigdir –type f | head<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">If \
that completes in time, you can increase the pagination by passing arguments to head, \
f.ex.<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">find \
/nfs/reallybigdir –type f | head -2000<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">`rm \
–r /nfs/reallybigdir` usually also works pretty good, but takes notibly more \
resources on the host OS than the rsync variant.<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">If \
all of that doesn't help, I've a C program code which I used to delete 1023MB big \
directories in the past which I could share then.<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="DE-AT" \
style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">Alexander \
Griesser<o:p></o:p></span></b></p> <p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">Head of \
Systems Operations<o:p></o:p></span></p> <p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">ANEXIA \
Internetdienstleistungs GmbH<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="DE-AT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">E-Mail: \
<a href="mailto:AGriesser@anexia-it.com"><span \
style="color:blue">AGriesser@anexia-it.com</span></a> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">Web: <a \
href="http://www.anexia-it.com/"><span \
style="color:blue">http://www.anexia-it.com</span></a> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#595959">Anschrift \
Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt<o:p></o:p></span></p> \
<p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#595959">Geschäftsführer: \
Alexander Windbichler<o:p></o:p></span></p> <p class="MsoNormal"><span lang="DE-AT" \
style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#595959">Firmenbuch: \
FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT \
U63216601<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif">Von:</span></b><span \
style="font-size:11.0pt;font-family:"Calibri",sans-serif"> \
toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] <b>Im Auftrag \
von </b>Edward Rolison<br> <b>Gesendet:</b> Donnerstag, 17. März 2016 11:27<br>
<b>An:</b> toasters@teaparty.net<br>
<b>Betreff:</b> Dealing with a really large number of files in a single \
directory<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I've got a directory on an NFS server. I think it's been filling \
up with log files, and may be in excess of a million.<o:p></o:p></p> <div>
<p class="MsoNormal">(ls -d shows me a 350MB rather than the usual \
4k). <o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">strace -fTt ls -1 -f is stalling on 'getdents' system \
call. I assume it'll complete eventually, but ... well, the 'leave it and wait' \
hasn't yet. <o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Does anyone have any good tricks on a filer for helping me tell \
that a) this directory is in fact, absurdly large and b) starting to retrieve \
filenames so I can delete/move them in sensibly sized baches? <o:p></o:p></p> \
</div> <div>
<p class="MsoNormal"><br>
(I'm wondering if a filer-side ls is going to help or hinder, for \
example). <o:p></o:p></p> </div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>
_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
--===============2986268204219537806==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic