[prev in list] [next in list] [prev in thread] [next in thread] 

List:       gpfsug-discuss
Subject:    Re: [gpfsug-discuss] very low read performance in simple spectrum scale/gpfs cluster with a storage-
From:       Jan-Frode Myklebust <janfrode () tanso ! net>
Date:       2020-06-16 17:54:41
Message-ID: CAHwPatiMVTUorhqtYagOCMmQuyWj6WiYTcFmwOs6PFL+XgN3aw () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


tir. 16. jun. 2020 kl. 15:32 skrev Giovanni Bracco <giovanni.bracco@enea.it
>:

>
> > I would correct MaxMBpS -- put it at something reasonable, enable
> > verbsRdmaSend=yes and
> > ignorePrefetchLUNCount=yes.
>
> Now we have set:
> verbsRdmaSend yes
> ignorePrefetchLUNCount yes
> maxMBpS 8000
>
> but the only parameter which has a strong effect by itself is
>
> ignorePrefetchLUNCount yes
>
> and the readout performance increased of a factor at least 4, from
> 50MB/s to 210 MB/s



That's interesting.. ignoreprefetchluncount=yes should mean it more
aggresively schedules IO. Did you also try lowering maxMBpS? I'm thinking
maybe something is getting flooded somewhere..

Another knob would be to increase workerThreads, and/or prefetchPct (don't
quite renember how these influence each other).

And it would be useful to run nsdperf between client and nsd-servers, to
verify/rule out any network issue.


> fio --name=seqwrite --rw=write --buffered=1 --ioengine=posixaio --bs=1m
> --numjobs=1 --size=100G --runtime=60
>
> fio --name=seqread --rw=wread --buffered=1 --ioengine=posixaio --bs=1m
> --numjobs=1 --size=100G --runtime=60
>
>
Not too familiar with fio, but ... does it help to increase numjobs?

And.. do you tell both sides which fabric number they're on ( «verbsPorts
qib0/1/1 ») so the GPFS knows not to try to connect verbsPorts that can't
communicate?


  -jf

[Attachment #5 (text/html)]

<div><div><div><br></div><div><br><div \
class="gmail_quote"></div></div></div><div><div dir="ltr" class="gmail_attr">tir. 16. \
jun. 2020 kl. 15:32 skrev Giovanni Bracco &lt;<a \
href="mailto:giovanni.bracco@enea.it" \
target="_blank">giovanni.bracco@enea.it</a>&gt;:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><br>
 &gt; I would correct MaxMBpS -- put it at something reasonable, enable <br>
&gt; verbsRdmaSend=yes and<br>
&gt; ignorePrefetchLUNCount=yes.<br>
<br>
Now we have set:<br>
verbsRdmaSend yes<br>
ignorePrefetchLUNCount yes<br>
maxMBpS 8000<br>
<br>
but the only parameter which has a strong effect by itself is<br>
<br>
ignorePrefetchLUNCount yes<br>
<br>
and the readout performance increased of a factor at least 4, from <br>
50MB/s to 210 MB/s</blockquote><div dir="auto"><br></div><div \
dir="auto"><br></div></div></div><div><div><div dir="auto">That's interesting.. \
ignoreprefetchluncount=yes should mean it more aggresively schedules IO. Did you also \
try lowering maxMBpS? I'm thinking maybe something is getting flooded \
somewhere..</div><div dir="auto"><br></div><div dir="auto">Another knob would be to \
increase workerThreads, and/or prefetchPct (don't quite renember how these influence \
each other).</div><div dir="auto"><br></div><div dir="auto">And it would be useful to \
run nsdperf between client and nsd-servers, to verify/rule out any network \
issue.</div></div><div><div><div \
class="gmail_quote"></div></div></div></div><div><div \
dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><br>fio \
                --name=seqwrite --rw=write --buffered=1 --ioengine=posixaio --bs=1m \
                <br>
--numjobs=1 --size=100G --runtime=60<br>
<br>
fio --name=seqread --rw=wread --buffered=1 --ioengine=posixaio --bs=1m <br>
--numjobs=1 --size=100G --runtime=60<br>
<br>
</blockquote><div dir="auto"><br></div></div><div><div dir="auto">Not too familiar \
with fio, but ... does it help to increase numjobs?</div></div><div><div><div><div \
class="gmail_quote"><div dir="auto"><br></div><div dir="auto">And.. do you tell both \
sides which fabric number they're on ( «<span \
style="word-spacing:1px;color:rgb(49,49,49)">verbsPorts qib0/1/1 ») so the GPFS knows \
not to try to connect verbsPorts that can't communicate?</span></div><div \
dir="auto"><span style="word-spacing:1px;color:rgb(49,49,49)"><br></span></div><div \
dir="auto"><br></div><div dir="auto">   -jf</div></div></div> </div>
</div>



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


[prev in list] [next in list] [prev in thread] [next in thread]