[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] Lustre project quotas and project IDs
From: "Passerini Marco" <marco.passerini () cscs ! ch>
Date: 2023-03-23 9:21:18
Message-ID: c79444023ac449ca9c51bc38a2f2eb15 () cscs ! ch
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hi,
I think in our situation we would need to have multiple project IDs assigned the same \
UID, and in some case they would need to be assigned to GID instead, so it might get \
a little complex.
The possibility of using a custom string instead of a project ID number would be best \
for us, I believe. I don't think we would rely on the fallback option you suggest.
Regards,
Marco Passerini
________________________________
From: Andreas Dilger <adilger@whamcloud.com>
Sent: Wednesday, March 22, 2023 5:02:36 PM
To: Passerini Marco
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre project quotas and project IDs
Of course my preference would be a contribution to improving the name-projid mapping \
in the "lfs project" command under LU-13335 so this would also help other Lustre \
users manage their project IDs.
One proposal I had in LU-13335 that I would welcome feedback on was if a name or \
projid did not exist in /etc/projid that the lfs tool would fall back to doing a \
name/uid lookup in /etc/passwd (or other database as configured in \
/etc/nsswitch.conf).
This would avoid the need to duplicate the full UID database in /etc/projid for the \
common case of projid = uid, and allows using LDAP, NIS, AD, sssd, etc. for projid \
lookup without them having explicit support for a projid database.
This behavior could optionally be configured with a "commented-out" directive at the \
start of /etc/projid, like:
#lfs fallback: passwd
or "group" or "none". If all the projects are defined in the passwd database, then \
potentially just this one line is needed in /etc/projid, or not at all if "passwd" is \
the default fallback.
Would this meet your need for using an external database, while still allowing your \
development efforts to produce a solution that helps the Lustre community?
Of course at some point it wouod be desirable to have a dedicated projid database \
supported by glibc, but that is would take much more time and effort to implement and \
deploy, while the passwd/group fallback can be handled internally by the lfs command.
Cheers, Andreas
On Mar 17, 2023, at 04:10, Passerini Marco <marco.passerini@cscs.ch> wrote:
Hi Andreas,
I'm talking the order of ~10,000s of project IDs.
I've been thinking the same as you, that is, doing PROJID=1M + UID etc. However, in \
our case, it might be better to rely on some scripting and an external DB, to keep \
track of the latest added ID, so that we could increment the highest value by 1 on \
new ID creation. The highest value could as well be looked up in:
/proc/fs/lustre/osd-ldiskfs/myfs-MDT0000/quota_slave_dt/acct_project
Regards,
Marco Passerini
________________________________
From: Andreas Dilger <adilger@whamcloud.com>
Sent: Thursday, March 16, 2023 11:35:16 PM
To: Passerini Marco
Cc: lustre-discuss@lists.lustre.org
Subject: Re: [lustre-discuss] Lustre project quotas and project IDs
On Mar 16, 2023, at 04:50, Passerini Marco \
<marco.passerini@cscs.ch<mailto:marco.passerini@cscs.ch>> wrote:
By trial and error, I found that, when using project quotas, the maximum ID available \
is 4294967294. Is this correct?
Yes, the "-1" ID is reserved for error conditions.
If I assign quota to a lot of project IDs, is the performance expected to go down \
more than having just a few or is it fixed?
Probably if you have millions or billions of different IDs there would be some \
performance loss, at a minimum just because the quota files will consume a lot of \
disk space and memory to manage. I don't think we've done specific scaling testing \
for the number of project IDs, but it has worked well for the "expected" number of \
different IDs at production sites (in the 10,000s).
I've recommended to a few sites that want to have a "unified" quota to use e.g. \
PROJID=UID for user directories, PROJID=1M + UID for scratch, and PROJID=2M+N for \
independent projects, just to make the PROJIDs easily identified (at least until \
someone implements LU-13335 to do projid<->name mapping).
How many IDs were you thinking of using?
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Whamcloud
[Attachment #3 (text/html)]
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} \
--></style> <div id="divtagdefaultwrapper" \
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" \
dir="ltr"> <p>Hi,</p>
<p><br>
</p>
<p>I think in our situation we would need to have multiple project IDs assigned the \
same UID, and in some case they would need to be assigned to GID instead, so it might \
get a little complex. <br>
</p>
<p>The possibility of using a custom string instead of a project ID number would be \
best for us, I believe. I don't think we would rely on the fallback option you \
suggest.</p> <p><br>
</p>
<p>Regards,<br>
</p>
<p>Marco Passerini<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" \
style="font-size:11pt" color="#000000"><b>From:</b> Andreas Dilger \
<adilger@whamcloud.com><br> <b>Sent:</b> Wednesday, March 22, 2023 5:02:36 \
PM<br> <b>To:</b> Passerini Marco<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre project quotas and project IDs</font>
<div> </div>
</div>
<div>Of course my preference would be a contribution to improving the name-projid \
mapping in the "lfs project" command under LU-13335 so this would also help \
other Lustre users manage their project IDs. <br> <br>
<div dir="ltr">One proposal I had in LU-13335 that I would welcome feedback on was if \
a name or projid did not exist in /etc/projid that the lfs tool would fall back to \
doing a name/uid lookup in /etc/passwd (or other database as configured in \
/etc/nsswitch.conf).</div> <div dir="ltr"><br>
</div>
<div dir="ltr">This would avoid the need to duplicate the full UID database in \
/etc/projid for the common case of projid = uid, and allows using LDAP, NIS, AD, \
sssd, etc. for projid lookup without them having explicit support for a projid \
database. </div> <div dir="ltr"><br>
</div>
<div dir="ltr">This behavior could optionally be configured with a \
"commented-out" directive at the start of /etc/projid, like:</div> <div \
dir="ltr"><br> </div>
<div dir="ltr"> #lfs fallback: passwd</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">or "group" or "none". If all the projects \
are defined in the passwd database, then potentially just this one line is needed in \
/etc/projid, or not at all if "passwd" is the default fallback. </div> \
<div dir="ltr"><br> </div>
<div dir="ltr">Would this meet your need for using an external database, while still \
allowing your development efforts to produce a solution that helps the Lustre \
community?</div> <div dir="ltr"><br>
</div>
<div dir="ltr">Of course at some point it wouod be desirable to have a dedicated \
projid database supported by glibc, but that is would take much more time and effort \
to implement and deploy, while the passwd/group fallback can be handled internally by \
the lfs command. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">
<div dir="ltr">Cheers, Andreas</div>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Mar 17, 2023, at 04:10, Passerini Marco \
<marco.passerini@cscs.ch> wrote:<br> <br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div id="divtagdefaultwrapper" \
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" \
dir="ltr"> <p>Hi Andreas,</p>
<p><br>
</p>
<p>I'm talking the order of ~10,000s of project IDs.</p>
<p>I've been thinking the same as you, that is, doing <span>PROJID=1M + UID \
</span> etc. However, in our case, it might be better to rely on some scripting and \
an external DB, to keep track of the latest added ID, so that we could increment the \
highest value by 1 on new ID creation. The highest value could as well be looked up \
in:<span><br> </span></p>
<p><span><br>
</span></p>
<p><span></span></p>
<div>/proc/fs/lustre/osd-ldiskfs/myfs-MDT0000/quota_slave_dt/acct_project<br>
<br>
</div>
Regards,<br>
<p></p>
<p>Marco Passerini<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" \
style="font-size:11pt" color="#000000"><b>From:</b> Andreas Dilger \
<adilger@whamcloud.com><br> <b>Sent:</b> Thursday, March 16, 2023 11:35:16 \
PM<br> <b>To:</b> Passerini Marco<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] Lustre project quotas and project IDs</font>
<div> </div>
</div>
<div>On Mar 16, 2023, at 04:50, Passerini Marco <<a \
href="mailto:marco.passerini@cscs.ch" class="">marco.passerini@cscs.ch</a>> \
wrote:<br class=""> <div>
<blockquote type="cite" class=""><br class="Apple-interchange-newline">
<div class="">
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; \
font-family: Calibri, Helvetica, sans-serif;" class=""> By trial and error, I found \
that, when using project quotas, the maximum ID available is<span \
class="Apple-converted-space"> </span><span class="">4294967294</span>. Is this \
correct?</div> </div>
</blockquote>
<div><br class="">
</div>
Yes, the "-1" ID is reserved for error conditions.<br class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: \
start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: \
0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; \
font-family: Calibri, Helvetica, sans-serif;" class=""> <div style="margin-top: 0px; \
margin-bottom: 0px;" class="">If I assign quota to a lot of project IDs, is the \
performance expected to go down more than having just a few or is it fixed?</div> \
</div> </div>
</blockquote>
<br class="">
</div>
<div>Probably if you have millions or billions of different IDs there would be some \
performance loss, at a minimum just because the quota files will consume a lot of \
disk space and memory to manage. I don't think we've done specific scaling \
testing for the number of project IDs, but it has worked well for the \
"expected" number of different IDs at production sites (in the \
10,000s).</div> <div><br class="">
</div>
<div>I've recommended to a few sites that want to have a "unified" quota to \
use e.g. PROJID=UID for user directories, PROJID=1M + UID for scratch, and \
PROJID=2M+N for independent projects, just to make the PROJIDs easily identified \
(at least until someone implements LU-13335 to do projid<->name \
mapping).</div> <div><br class="">
</div>
<div>How many IDs were you thinking of using?</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); \
color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; \
-webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div dir="auto" \
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; \
text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: \
break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div \
dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: \
normal; text-align: start; text-indent: 0px; text-transform: none; white-space: \
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; \
word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" \
class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); \
letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; \
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; \
text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: \
after-white-space;" class=""> <div dir="auto" style="caret-color: rgb(0, 0, 0); \
color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; \
text-transform: none; white-space: normal; word-spacing: 0px; \
-webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; \
-webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div>Cheers, \
Andreas</div> <div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</body>
</html>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--===============5366333077841948365==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic