[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] About Lustre small files performace(8k) improve
From: Andreas Dilger via lustre-discuss <lustre-discuss () lists ! lustre ! org>
Date: 2023-03-28 4:56:50
Message-ID: 5CE79196-BAA6-4047-9498-32C9A53C1E00 () ddn ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Are your performance tests on NFS or on native Lustre clients? Native Lustre clients \
will likely be faster, and with many clients they can create files in parallel, even \
in the same directory. With a single NFS server they will be limited by the VFS \
locking for a single directory.
Are you using IB or TCP networking? IB will be faster for low-latency requests.
Are you using the Data-on-MDT feature? This can reduce overhead for very small \
files.
Are you using NVMe storage or e.g. SATA SSDs? Based on the OST size it looks like \
flash of some kind, unless you are using single-HDD OSTs?
Cheers, Andreas
On Mar 18, 2023, at 01:44, 王烁斌 via lustre-discuss \
<lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>> wrote:
Hi all,
This is my Lustre FS.
UUID 1K-blocks Used Available Use% Mounted on
ltfs-MDT0000_UUID 307826072 36904 281574768 1% /mnt/lfs[MDT:0]
ltfs-MDT0001_UUID 307826072 36452 281575220 1% /mnt/lfs[MDT:1]
ltfs-MDT0002_UUID 307826072 36600 281575072 1% /mnt/lfs[MDT:2]
ltfs-MDT0003_UUID 307826072 36300 281575372 1% /mnt/lfs[MDT:3]
ltfs-OST0000_UUID 15962575136 1027740 15156068868 1% /mnt/lfs[OST:0]
ltfs-OST0001_UUID 15962575136 1027780 15156067516 1% /mnt/lfs[OST:1]
ltfs-OST0002_UUID 15962575136 1027772 15156074212 1% /mnt/lfs[OST:2]
ltfs-OST0003_UUID 15962575136 1027756 15156067860 1% /mnt/lfs[OST:3]
ltfs-OST0004_UUID 15962575136 1027728 15156058224 1% /mnt/lfs[OST:4]
ltfs-OST0005_UUID 15962575136 1027772 15156057668 1% /mnt/lfs[OST:5]
ltfs-OST0006_UUID 15962575136 1027768 15156058568 1% /mnt/lfs[OST:6]
ltfs-OST0007_UUID 15962575136 1027792 15156056752 1% /mnt/lfs[OST:7]
filesystem_summary: 127700601088 8222108 121248509668 1% /mnt/lfs
Structure ias flow:
<image.png>
After testing, under the current structure, the write performance of 500,000 "8k" \
small files is: NFSclient1——IOPS：28,000; bandwidth——230MB
NFSclient1——IOPS：27,500; bandwidth——220MB
Now I want to improve the performance of small files to a better level，May I ask if \
there is a better way。
I have noticed a feature called "MIP-IO" that can improve small file performance, but \
I don't know how to deploy this feature. Is there any way to improve small file \
performance？
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> Are your performance tests on NFS or on native Lustre \
clients? Native Lustre clients will likely be faster, and with many clients \
they can create files in parallel, even in the same directory. With a single \
NFS server they will be limited by the VFS locking for a single directory.
<div class=""><br class="">
</div>
<div class="">Are you using IB or TCP networking? IB will be faster for \
low-latency requests.</div> <div class=""><br class="">
</div>
<div class="">Are you using the Data-on-MDT feature? This can reduce overhead \
for very small files.</div> <div class=""><br class="">
</div>
<div class="">Are you using NVMe storage or e.g. SATA SSDs? Based on the OST \
size it looks like flash of some kind, unless you are using single-HDD OSTs?</div> \
<div class=""><br class=""> </div>
<div class="">Cheers, Andreas<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 18, 2023, at 01:44, 王烁斌 via lustre-discuss <<a \
href="mailto:lustre-discuss@lists.lustre.org" \
class="">lustre-discuss@lists.lustre.org</a>> wrote:</div> <br \
class="Apple-interchange-newline"> <div class="">
<div style="line-height: 1.7; font-size: 14px; font-family: Arial;" class="">
<div class="">Hi all,<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">This is my Lustre FS. </div>
<div class="">
<div class="">UUID \
1K-blocks Used Available Use% Mounted \
on</div> <div class="">ltfs-MDT0000_UUID 307826072 \
36904 281574768 1% /mnt/lfs[MDT:0]</div> <div \
class="">ltfs-MDT0001_UUID 307826072 \
36452 281575220 1% /mnt/lfs[MDT:1]</div> <div \
class="">ltfs-MDT0002_UUID 307826072 \
36600 281575072 1% /mnt/lfs[MDT:2]</div> <div \
class="">ltfs-MDT0003_UUID 307826072 \
36300 281575372 1% /mnt/lfs[MDT:3]</div> <div \
class="">ltfs-OST0000_UUID 15962575136 1027740 \
15156068868 1% /mnt/lfs[OST:0]</div> <div \
class="">ltfs-OST0001_UUID 15962575136 1027780 \
15156067516 1% /mnt/lfs[OST:1]</div> <div \
class="">ltfs-OST0002_UUID 15962575136 1027772 \
15156074212 1% /mnt/lfs[OST:2]</div> <div \
class="">ltfs-OST0003_UUID 15962575136 1027756 \
15156067860 1% /mnt/lfs[OST:3]</div> <div \
class="">ltfs-OST0004_UUID 15962575136 1027728 \
15156058224 1% /mnt/lfs[OST:4]</div> <div \
class="">ltfs-OST0005_UUID 15962575136 1027772 \
15156057668 1% /mnt/lfs[OST:5]</div> <div \
class="">ltfs-OST0006_UUID 15962575136 1027768 \
15156058568 1% /mnt/lfs[OST:6]</div> <div \
class="">ltfs-OST0007_UUID 15962575136 1027792 \
15156056752 1% /mnt/lfs[OST:7]</div> <div class=""><br class="">
</div>
<div class="">filesystem_summary: 127700601088 8222108 \
121248509668 1% /mnt/lfs</div> </div>
<div class=""><br class="">
</div>
<div class="">Structure ias flow:</div>
<div class=""><span id="cid:ad69d04$1$186f3a691ff$Coremail$w14767780617$163.com"><image.png></span> </div>
<div class=""><br class="">
</div>
<div class="">After testing, under the current structure, the write performance of \
500,000 "8k" small files is:</div> <div \
class="">NFSclient1——IOPS：28,000; bandwidth——230MB</div> <div \
class="">NFSclient1——IOPS：27,500; bandwidth——220MB</div> <div \
class=""><br class=""> </div>
<div class="">Now I want to improve the performance of small files to a better \
level，May I ask if there is a better way。</div> <div class=""><br class="">
</div>
<div class="">I have noticed a feature called "MIP-IO" that can improve \
small file performance, but I don't know how to deploy this feature. Is there any way \
to improve small file performance？</div> <div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
_______________________________________________<br class="">
lustre-discuss mailing list<br class="">
<a href="mailto:lustre-discuss@lists.lustre.org" \
class="">lustre-discuss@lists.lustre.org</a><br class=""> \
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org<br class=""> </div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); \
color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; \
-webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div dir="auto" \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; \
text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" \
class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); \
color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; \
-webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div>Cheers, \
Andreas</div> <div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--===============2633278755165369935==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic