[prev in list] [next in list] [prev in thread] [next in thread]
List: lustre-discuss
Subject: Re: [lustre-discuss] max_pages_per_rpc=4096 fails on the client nodes
From: Pinkesh Valdria <pinkesh.valdria () oracle ! com>
Date: 2019-08-14 19:02:14
Message-ID: FE308EA9-61D7-4759-BB85-6221189F4F1D () oracle ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
For others, incase they face this issue.
Solution: I had to unmount and remount for the command to work.
From: Pinkesh Valdria <pinkesh.valdria@oracle.com>
Date: Wednesday, August 14, 2019 at 9:25 AM
To: "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org>
Subject: max_pages_per_rpc=4096 fails on the client nodes
I want to enable large RPC size. I followed the steps as per the Lustre manual \
section: 33.9.2 Usage (http://doc.lustre.org/lustre_manual.xhtml), but I get the \
below we error when I try to update the client.
Updated the OSS server:
[root@lustre-oss-server-nic0-1 test]# lctl set_param obdfilter.lfsbv-*.brw_size=16
obdfilter.lfsbv-OST0000.brw_size=16
obdfilter.lfsbv-OST0001.brw_size=16
obdfilter.lfsbv-OST0002.brw_size=16
obdfilter.lfsbv-OST0003.brw_size=16
obdfilter.lfsbv-OST0004.brw_size=16
obdfilter.lfsbv-OST0005.brw_size=16
obdfilter.lfsbv-OST0006.brw_size=16
obdfilter.lfsbv-OST0007.brw_size=16
obdfilter.lfsbv-OST0008.brw_size=16
obdfilter.lfsbv-OST0009.brw_size=16
[root@lustre-oss-server-nic0-1 test]#
Add the above change permanently using MGS node:
[root@lustre-mds-server-nic0-1 ~]# lctl set_param -P obdfilter.lfsbv-*.brw_size=16
[root@lustre-mds-server-nic0-1 ~]#
Client side update – failed
[root@lustre-client-1 ~]# lctl set_param osc.lfsbv-OST*.max_pages_per_rpc=4096
error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0000-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range
error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0001-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range
error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0002-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range
error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0003-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range
…..
…..
33.9.2. Usage
In order to enable a larger RPC size, brw_size must be changed to an IO size value up \
to 16MB. To temporarily change brw_size, the following command should be run on the \
OSS:
oss# lctl set_param obdfilter.fsname-OST*.brw_size=16
To persistently change brw_size, the following command should be run:
oss# lctl set_param -P obdfilter.fsname-OST*.brw_size=16
When a client connects to an OST target, it will fetch brw_size from the target and \
pick the maximum value of brw_size and its local setting for max_pages_per_rpc as the \
actual RPC size. Therefore, the max_pages_per_rpc on the client side would have to be \
set to 16M, or 4096 if the PAGESIZE is 4KB, to enable a 16MB RPC. To temporarily make \
the change, the following command should be run on the client to \
setmax_pages_per_rpc:
client$ lctl set_param osc.fsname-OST*.max_pages_per_rpc=16M
To persistently make this change, the following command should be run:
client$ lctl set_param -P obdfilter.fsname-OST*.osc.max_pages_per_rpc=16M
Caution
The brw_size of an OST can be changed on the fly. However, clients have to be \
remounted to renegotiate the new maximum RPC size.
[Attachment #5 (text/html)]
<html xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type \
content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 \
(filtered medium)"><style><!-- /* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Times;
panose-1:0 0 5 0 0 0 0 2 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:13.5pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Calibri",sans-serif;
font-weight:bold;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div \
class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>For others, \
incase they face this issue. <o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt'><br>Solution: I had to unmount and remount for the command \
to work. <o:p></o:p></span></p><p class=MsoNormal><span \
style='font-size:11.0pt'><o:p> </o:p></span></p><div \
style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p \
class=MsoNormal><b><span style='color:black'>From: </span></b><span \
style='color:black'>Pinkesh Valdria <pinkesh.valdria@oracle.com><br><b>Date: \
</b>Wednesday, August 14, 2019 at 9:25 AM<br><b>To: \
</b>"lustre-discuss@lists.lustre.org" \
<lustre-discuss@lists.lustre.org><br><b>Subject: </b>max_pages_per_rpc=4096 \
fails on the client nodes<o:p></o:p></span></p></div><div><p class=MsoNormal><span \
style='font-size:11.0pt'><o:p> </o:p></span></p></div><p class=MsoNormal><span \
style='font-size:11.0pt'>I want to enable large RPC size. I followed the \
steps as per the Lustre manual section: 33.9.2 Usage (</span><a \
href="http://doc.lustre.org/lustre_manual.xhtml">http://doc.lustre.org/lustre_manual.xhtml</a><span \
style='font-size:11.0pt'>), but I get the below we error when I try to update \
the client. </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>Updated the OSS server: </span><o:p></o:p></p><p \
class=MsoNormal><span style='font-size:11.0pt'>[root@lustre-oss-server-nic0-1 test]# \
lctl set_param obdfilter.lfsbv-*.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0000.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0001.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0002.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0003.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0004.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0005.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0006.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0007.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0008.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span \
style='font-size:11.0pt'>obdfilter.lfsbv-OST0009.brw_size=16</span><o:p></o:p></p><p \
class=MsoNormal><span style='font-size:11.0pt'>[root@lustre-oss-server-nic0-1 \
test]#</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>Add the above change permanently using MGS node: \
</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>[root@lustre-mds-server-nic0-1 ~]# lctl set_param -P \
obdfilter.lfsbv-*.brw_size=16</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>[root@lustre-mds-server-nic0-1 ~]#</span><o:p></o:p></p><p \
class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p \
class=MsoNormal><span style='font-size:11.0pt'> </span><o:p></o:p></p><p \
class=MsoNormal><span style='font-size:11.0pt'>Client side update – failed \
</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>[root@lustre-client-1 ~]# lctl set_param \
osc.lfsbv-OST*.max_pages_per_rpc=4096</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0000-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0001-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0002-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>error: set_param: setting \
/proc/fs/lustre/osc/lfsbv-OST0003-osc-ffff8e66b4b08000/max_pages_per_rpc=4096: \
Numerical result out of range</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>…..</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'>…..</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span \
style='font-size:13.5pt;font-family:Times;color:black'>33.9.2. Usage</span></b><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:13.5pt;font-family:Times;color:black'>In order to enable a larger \
RPC size, </span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> must be changed to an IO \
size value up to 16MB. To temporarily change </span><span \
style='font-size:10.0pt;font-family:"Courier New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'>, the following command should \
be run on the OSS:</span><o:p></o:p></p><p class=MsoNormal \
style='background:#E0E0E0'><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>oss# lctl set_param \
obdfilter.<i>fsname</i>-OST*.brw_size=16</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:13.5pt;font-family:Times;color:black'>To persistently \
change </span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'>, the following command should \
be run:</span><o:p></o:p></p><p class=MsoNormal style='background:#E0E0E0'><span \
style='font-size:10.0pt;font-family:"Courier New";color:black'>oss# lctl set_param -P \
obdfilter.<i>fsname</i>-OST*.brw_size=16</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:13.5pt;font-family:Times;color:black'>When a client connects to an \
OST target, it will fetch </span><span \
style='font-size:10.0pt;font-family:"Courier New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> from the target and pick \
the maximum value of </span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> and its local setting \
for </span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>max_pages_per_rpc</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> as the actual RPC size. \
Therefore, the </span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>max_pages_per_rpc</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> on the client side would \
have to be set to 16M, or 4096 if the PAGESIZE is 4KB, to enable a 16MB RPC. To \
temporarily make the change, the following command should be run on the client to \
set</span><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>max_pages_per_rpc</span><span \
style='font-size:13.5pt;font-family:Times;color:black'>:</span><o:p></o:p></p><p \
class=MsoNormal style='background:#E0E0E0'><span \
style='font-size:10.0pt;font-family:"Courier New";color:black'>client$ lctl set_param \
osc.<i>fsname</i>-OST*.max_pages_per_rpc=16M</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:13.5pt;font-family:Times;color:black'>To persistently make this \
change, the following command should be run:</span><o:p></o:p></p><p class=MsoNormal \
style='background:#E0E0E0'><span style='font-size:10.0pt;font-family:"Courier \
New";color:black'>client$ lctl set_param -P \
obdfilter.<i>fsname</i>-OST*.osc.max_pages_per_rpc=16M</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><b><span \
style='font-size:13.5pt;font-family:Times;color:black'>Caution</span></b><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:13.5pt;font-family:Times;color:black'>The </span><span \
style='font-size:10.0pt;font-family:"Courier New";color:black'>brw_size</span><span \
style='font-size:13.5pt;font-family:Times;color:black'> of an OST can be changed \
on the fly. However, clients have to be remounted to renegotiate the new maximum RPC \
size.</span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p><p class=MsoNormal><span \
style='font-size:11.0pt'> </span><o:p></o:p></p></div></body></html>
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic