[prev in list] [next in list] [prev in thread] [next in thread]
List: gpfsug-discuss
Subject: Re: [gpfsug-discuss] Tuning: single client, single thread, small files - native Scale vs NFS
From: "Michael Diederich" <diederich () de ! ibm ! com>
Date: 2018-10-16 12:31:20
Message-ID: OFAB56E449.3E6054F8-ONC1258328.0044A12D-C1258328.0044C95E () notes ! na ! collabserv ! com
[Download RAW message or body]
--=_alternative 0044C8FEC1258328_Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="ISO-8859-1"
All NFS IO requires syncing.
The client does send explicit fsync (commit). If the NFS server does not
sync, a server fail will cause data loss!
(for small files <1M it really does not matter if it is sync on write or
sync on close/explicit commit)
while that may be ok for a "git pull" or similar, in general it violates
the NFS spec.
The client can decide to cache, and usually NFSv4 does less caching (for
better consistency)
So the observed factor 100 is realistic.
Latencies will make matters worse, so the FS should be tuned for very
small random IO (small blocksize - small subblock-size will not help)
If you were to put the Linux kernel NFS server into the picture, it will
behave very much the same - although Ganesha could be a bit more efficient
(by some percent - certainly less then 200%).
But hey - this is a GPFS cluster not some NAS box.
Run "git pull" on tthe GPFS client. Enjoy the 1800 files/sec (or more).
Modify the files on your XY client mounting over NFS. Use a wrapper script
to automatically have your AD or LDAP user id SSH into the cluster to
perform it.
Michael
Mit freundlichen Grüßen / with best regards
Michael Diederich
IBM Systems Group
Spectrum Scale
Software Development
Contact Information
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
mail:
fon:
address:
michael.diederich@de.ibm.com
+49-7034-274-4062
Am Weiher 24
D-65451 Kelsterbach
From: valdis.kletnieks@vt.edu
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Cc: Silvana De Gyves <silvanad@mx1.ibm.com>, Jay Vaddi
<jayvaddi@us.ibm.com>, Michael Diederich <diederich@de.ibm.com>
Date: 10/16/2018 02:42 AM
Subject: Re: [gpfsug-discuss] Tuning: single client, single thread,
small files - native Scale vs NFS
Sent by: Valdis Kletnieks <valdis@vt.edu>
On Mon, 15 Oct 2018 18:34:50 -0400, "Kumaran Rajaram" said:
> 1. >>When writing to GPFS directly I'm able to write ~1800 files /
second in a test setup.
> > > This is roughly the same on the protocol nodes (NSD client), as well
as
> on the ESS IO nodes (NSD server).
>
> 2. >> When writing to the NFS export on the protocol node itself (to
avoid
> any network effects) I'm only able to write ~230 files / second.
> IMHO #2, writing to the NFS export on the protocol node should be same
as #1.
> Protocol node is also a NSD client and when you write from a protocol
node, it
> will use the NSD protocol to write to the ESS IO nodes. In #1, you cite
seeing
> ~1800 files from protocol node and in #2 you cite seeing ~230 file/sec
which
> seem to contradict each other.
I think he means this:
1) ssh nsd_server
2) cd /gpfs/filesystem/testarea
3) (whomp out 1800 files/sec)
4) mount -t nfs localhost:/gpfs/filesystem/testarea /mnt/test
5) cd /mnt/test
6) Watch the same test struggle to hit 230.
Indicating the issue is going from NFS to GPFS
(For what it's worth, we've had issues with Ganesha as well...)
[attachment "att4z9wh.dat" deleted by Michael Diederich/Germany/IBM]
--=_alternative 0044C8FEC1258328_Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="ISO-8859-1"
<font size=2 face="sans-serif">All NFS IO requires syncing.</font><br><font size=2 \
face="sans-serif">The client does send explicit fsync (commit). If the NFS server \
does not sync, a server fail will cause data loss!</font><br><font size=2 \
face="sans-serif">(for small files <1M it really does not matter if it is sync on \
write or sync on close/explicit commit)</font><br><br><br><font size=2 \
face="sans-serif">while that may be ok for a "git pull" or similar, in \
general it violates the NFS spec.</font><br><br><font size=2 face="sans-serif">The \
client can decide to cache, and usually NFSv4 does less caching (for better \
consistency)</font><br><br><font size=2 face="sans-serif">So the observed factor 100 \
is realistic. </font><br><font size=2 face="sans-serif">Latencies will make matters \
worse, so the FS should be tuned for very small random IO (small blocksize - small
subblock-size will not help)</font><br><br><font size=2 face="sans-serif">If you were \
to put the Linux kernel NFS server into the picture, it will behave very much the \
same - although Ganesha could be a bit more efficient (by some percent - certainly \
less then 200%). </font><br><br><br><font size=2 face="sans-serif">But hey - this is \
a GPFS cluster not some NAS box.</font><br><font size=2 face="sans-serif">Run \
"git pull" on tthe GPFS client. Enjoy the 1800 files/sec (or more). Modify \
the files on your XY client mounting over NFS. Use a wrapper script to automatically \
have your AD or LDAP user id SSH into the cluster to perform \
it.</font><br><br><font size=2 face="sans-serif">Michael</font><br><table width=1024 \
style="border-collapse:collapse;"><tr valign=top height=8><td width=1024 colspan=4 \
style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px \
0px;padding:0px 0px;"><font size=2 face="Tahoma">Mit freundlichen Grüßen / with best \
regards <br> </font><tr valign=top height=8><td width=289 rowspan=2 \
style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px \
0px;padding:0px 0px;"><a \
href="http://w3.ibm.com/bluepages/simpleSearch.wss?searchBy=Internet+address&location=All+locations&searchFor=DIEDERIC@de.ibm.com"><font \
size=2 color=#004080 face="Tahoma"><b>Michael Diederich</b></font></a><font size=2 \
color=#5f5f5f face="Tahoma"><br>IBM Systems Group <br>Spectrum Scale</font><br><font \
size=2 color=#5f5f5f face="Tahoma">Software Development</font><td width=376 colspan=2 \
style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px \
0px;padding:0px 0px;"><font size=2 color=#4f4f4f face="Tahoma"><b>Contact \
Information</b></font><td width=358 rowspan=2 style="border-style:none none none \
none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 \
color=#5f5f5f face="Tahoma">IBM Deutschland Research & Development \
GmbH<br>Vorsitzender des Aufsichtsrats: Martina Koederitz<br>Geschäftsführung: Dirk \
Wittkopp <br>Sitz der Gesellschaft: Böblingen <br>Registergericht: Amtsgericht \
Stuttgart, HRB 243294 <br></font><tr valign=top height=8><td width=79 \
style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px \
0px;padding:0px 0px;"><font size=2 color=#5f5f5f \
face="Tahoma">mail:<br>fon:<br>address:</font><td width=297 style="border-style:none \
none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a \
href=mailto:DIEDERIC@de.ibm.com><font size=2 color=#004080 \
face="Tahoma">michael.diederich@de.ibm.com</font></a><font size=2 color=#5f5f5f \
face="Tahoma"><br>+49-7034-274-4062<br>Am Weiher 24<br>D-65451 Kelsterbach</font><tr \
valign=top height=8><td width=1024 colspan=4 bgcolor=#efefef style="border-style:none \
none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px \
0px;"><div align=center></div></table><br><br><br><br><br><font size=1 color=#5f5f5f \
face="sans-serif">From: </font><font size=1 \
face="sans-serif">valdis.kletnieks@vt.edu</font><br><font size=1 color=#5f5f5f \
face="sans-serif">To: </font><font size=1 \
face="sans-serif">gpfsug main discussion list \
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f \
face="sans-serif">Cc: </font><font size=1 \
face="sans-serif">Silvana De Gyves <silvanad@mx1.ibm.com>, Jay Vaddi \
<jayvaddi@us.ibm.com>, Michael Diederich \
<diederich@de.ibm.com></font><br><font size=1 color=#5f5f5f \
face="sans-serif">Date: </font><font size=1 \
face="sans-serif">10/16/2018 02:42 AM</font><br><font size=1 color=#5f5f5f \
face="sans-serif">Subject: </font><font size=1 \
face="sans-serif">Re: [gpfsug-discuss]
Tuning: single client, single thread, small files - native Scale vs \
NFS</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by: \
</font><font size=1 face="sans-serif">Valdis Kletnieks \
<valdis@vt.edu></font><br><hr noshade><br><br><br><tt><font size=2>On Mon, 15 \
Oct 2018 18:34:50 -0400, "Kumaran Rajaram" said:<br><br>> 1. \
>>When writing to GPFS directly I'm able to write ~1800 files / second in \
a test setup. <br>> >>This is roughly the same on the protocol nodes (NSD \
client), as well as <br>> on the ESS IO nodes (NSD server). <br>><br>> 2. \
>> When writing to the NFS export on the protocol node itself (to avoid \
<br>> any network effects) I'm only able to write ~230 files / second.<br><br>> \
IMHO #2, writing to the NFS export on the protocol node should be same as #1.<br>> \
Protocol node is also a NSD client and when you write from a protocol node, \
it<br>> will use the NSD protocol to write to the ESS IO nodes. In #1, you cite \
seeing<br>> ~1800 files from protocol node and in #2 you cite seeing ~230 file/sec \
which<br>> seem to contradict each other. <br><br>I think he means this:<br><br>1) \
ssh nsd_server<br>2) cd /gpfs/filesystem/testarea<br>3) (whomp out 1800 \
files/sec)<br>4) mount -t nfs localhost:/gpfs/filesystem/testarea /mnt/test<br>5) cd \
/mnt/test<br>6) Watch the same test struggle to hit 230.<br><br>Indicating the issue \
is going from NFS to GPFS<br><br>(For what it's worth, we've had issues with Ganesha \
as well...)<br>[attachment "att4z9wh.dat" deleted by Michael \
Diederich/Germany/IBM] </font></tt><br><br><BR>
--=_alternative 0044C8FEC1258328_=--
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic