[prev in list] [next in list] [prev in thread] [next in thread]
List: gpfsug-discuss
Subject: Re: [gpfsug-discuss] Odd behavior with cat followed by grep. (John Hanks)
From: "Steve Xiao" <sxiao () us ! ibm ! com>
Date: 2018-02-14 21:53:17
Message-ID: OF419A376B.CFE27863-ON85258234.0077B700-85258234.00783E13 () notes ! na ! collabserv ! com
[Download RAW message or body]
--=_alternative 00783DF885258234_Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="US-ASCII"
This could be related to the following flash:
http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012054
You should contact IBM service to obtain the fix for your release.
Steve Y. Xiao
gpfsug-discuss-bounces@spectrumscale.org wrote on 02/14/2018 02:18:02 PM:
> From: gpfsug-discuss-request@spectrumscale.org
> To: gpfsug-discuss@spectrumscale.org
> Date: 02/14/2018 02:18 PM
> Subject: gpfsug-discuss Digest, Vol 73, Issue 36
> Sent by: gpfsug-discuss-bounces@spectrumscale.org
>
> Send gpfsug-discuss mailing list submissions to
> gpfsug-discuss@spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> or, via email, send a message with subject or body 'help' to
> gpfsug-discuss-request@spectrumscale.org
>
> You can reach the person managing the list at
> gpfsug-discuss-owner@spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: Odd behavior with cat followed by grep. (John Hanks)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 14 Feb 2018 11:17:19 -0800
> From: John Hanks <griznog@gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Odd behavior with cat followed by grep.
> Message-ID:
> <CAGrHuK7okijzHLysDNCxR8tn1fKeMDwW_iT5898tBvaCR_QJUg@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Thanks Bryan, mystery solved :)
>
> We also stumbled across these related items, in case anyone else wanders
> into this thread.
>
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__bug-2Dgrep.gnu.narkive.com_Y8cfvWDt_bug-2D27666-2Dgrep-2Don-2Dgpfs-2Dfilesystem-2Dseek-2Dhole-2Dproblem&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=FgxYBxqHZ0bHdWirEs1U_B3oDpeHJe8iRd-
> TYrXh6FI&e=
>
> https://www.ibm.com/developerworks/community/forums/html/topic?
> id=c2a94433-9ec0-4a4b-abfe-d0a1e721d630
>
> GPFS, the gift that keeps on giving ... me more things to do instead of
> doing the things I want to be doing.
>
> Thanks all,
>
> jbh
>
> On Wed, Feb 14, 2018 at 10:48 AM, Bryan Banister
<bbanister@jumptrading.com>
> wrote:
>
> > Hi all,
> >
> >
> >
> > We found this a while back and IBM fixed it. Here?s your answer:
> > http://www-01.ibm.com/support/docview.wss?uid=isg1IV87385
> >
> >
> >
> > Cheers,
> >
> > -Bryan
> >
> >
> >
> > *From:* gpfsug-discuss-bounces@spectrumscale.org [
mailto:gpfsug-discuss-
> > bounces@spectrumscale.org] *On Behalf Of *John Hanks
> > *Sent:* Wednesday, February 14, 2018 12:31 PM
> > *To:* gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> > *Subject:* Re: [gpfsug-discuss] Odd behavior with cat followed by
grep.
> >
> >
> >
> > *Note: External Email*
> > ------------------------------
> >
> > Straces are interesting, but don't immediately open my eyes:
> >
> >
> >
> > strace of grep on NFS (works as expected)
> >
> >
> >
> > openat(AT_FDCWD, "/home/griznog/pipetest.tmp.txt", O_RDONLY) = 3
> >
> > fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0
> >
> > ioctl(3, TCGETS, 0x7ffe2c26b0b0) = -1 ENOTTY (Inappropriate
ioctl
> > for device)
> >
> > read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768
> >
> > lseek(3, 32768, SEEK_HOLE) = 530721
> >
> > lseek(3, 32768, SEEK_SET) = 32768
> >
> > fstat(1, {st_mode=S_IFREG|0644, st_size=5977, ...}) = 0
> >
> > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) =
> > 0x7f3bf6c43000
> >
> > write(1, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 8192chr1
> >
> >
> >
> > strace on GPFS (thinks file is binary)
> >
> >
> >
> > openat(AT_FDCWD, "/srv/gsfs0/projects/pipetest.tmp.txt", O_RDONLY) = 3
> >
> > fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0
> >
> > ioctl(3, TCGETS, 0x7ffc9b52caa0) = -1 ENOTTY (Inappropriate
ioctl
> > for device)
> >
> > read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768
> >
> > lseek(3, 32768, SEEK_HOLE) = 262144
> >
> > lseek(3, 32768, SEEK_SET) = 32768
> >
> > fstat(1, {st_mode=S_IFREG|0644, st_size=6011, ...}) = 0
> >
> > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) =
> > 0x7fd45ee88000
> >
> > close(3) = 0
> >
> > write(1, "Binary file /srv/gsfs0/projects/"..., 72Binary file
> > /srv/gsfs0/projects/levinson/xwzhu/pipetest.tmp.txt matches
> >
> > ) = 72
> >
> >
> >
> > Do the lseek() results indicate that the grep on the GPFS mounted
version
> > thinks the file is a sparse file? For comparison I strace'd md5sum in
place
> > of the grep and it does not lseek() with SEEK_HOLE, it's access in
both
> > cases look identical, like:
> >
> >
> >
> > open("/home/griznog/pipetest.tmp.txt", O_RDONLY) = 3
> >
> > fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
> >
> > fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0
> >
> > mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) =
> > 0x7fb7d2c2b000
> >
> > read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768
> >
> > ...[reads clipped]...
> >
> > read(3, "", 24576) = 0
> >
> > lseek(3, 0, SEEK_CUR) = 530721
> >
> > close(3) = 0
> >
> >
> >
> >
> >
> > jbh
> >
> >
> >
> >
> >
> > On Wed, Feb 14, 2018 at 9:51 AM, Aaron Knister
<aaron.s.knister@nasa.gov>
> > wrote:
> >
> > Just speculating here (also known as making things up) but I wonder if
> > grep is somehow using the file's size in its determination of binary
> > status. I also see mmap in the strace so maybe there's some issue with
> > mmap where some internal GPFS buffer is getting truncated
> > inappropriately but leaving a bunch of null values which gets returned
> > to grep.
> >
> > -Aaron
> >
> > On 2/14/18 10:21 AM, John Hanks wrote:
> > > Hi Valdis,
> > >
> > > I tired with the grep replaced with 'ls -ls' and 'md5sum', I don't
think
> > > this is a data integrity issue, thankfully:
> > >
> > > $ ./pipetestls.sh
> > > 256 -rw-r--r-- 1 39073 3001 530721 Feb 14 07:16
> > > /srv/gsfs0/projects/pipetest.tmp.txt
> > > 0 -rw-r--r-- 1 39073 3953 530721 Feb 14 07:16
> > /home/griznog/pipetest.tmp.txt
> > >
> > > $ ./pipetestmd5.sh
> > > 15cb81a85c9e450bdac8230309453a0a
/srv/gsfs0/projects/pipetest.tmp.txt
> > > 15cb81a85c9e450bdac8230309453a0a /home/griznog/pipetest.tmp.txt
> > >
> > > And replacing grep with 'file' even properly sees the files as
ASCII:
> > > $ ./pipetestfile.sh
> > > /srv/gsfs0/projects/pipetest.tmp.txt: ASCII text, with very long
lines
> > > /home/griznog/pipetest.tmp.txt: ASCII text, with very long lines
> > >
> > > I'll poke a little harder at grep next and see what the difference
in
> > > strace of each reveals.
> > >
> > > Thanks,
> > >
> > > jbh
> > >
> > >
> > >
> > >
> > > On Wed, Feb 14, 2018 at 7:08 AM, <valdis.kletnieks@vt.edu
> >
> > > <mailto:valdis.kletnieks@vt.edu>> wrote:
> > >
> > > On Wed, 14 Feb 2018 06:20:32 -0800, John Hanks said:
> > >
> > > > # ls -aln /srv/gsfs0/projects/pipetest.tmp.txt
> > $HOME/pipetest.tmp.txt
> > > > -rw-r--r-- 1 39073 3953 530721 Feb 14 06:10
> > /home/griznog/pipetest.tmp.txt
> > > > -rw-r--r-- 1 39073 3001 530721 Feb 14 06:10
> > > > /srv/gsfs0/projects/pipetest.tmp.txt
> > > >
> > > > We can "fix" the user case that exposed this by not using a
temp
> > file or
> > > > inserting a sleep, but I'd still like to know why GPFS is
behaving
> > this way
> > > > and make it stop.
> > >
> > > May be related to replication, or other behind-the-scenes
behavior.
> > >
> > > Consider this example - 4.2.3.6, data and metadata replication
both
> > > set to 2, 2 sites 95 cable miles apart, each is 3 Dell servers
with
> > > a full
> > > fiberchannel mesh to 3 Dell MD34something arrays.
> > >
> > > % dd if=/dev/zero bs=1k count=4096 of=sync.test; ls -ls
sync.test;
> > > sleep 5; ls -ls sync.test; sleep 5; ls -ls sync.test
> > > 4096+0 records in
> > > 4096+0 records out
> > > 4194304 bytes (4.2 MB) copied, 0.0342852 s, 122 MB/s
> > > 2048 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test
> > > 8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test
> > > 8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test
> > >
> > > Notice that the first /bin/ls shouldn't be starting until after
the
> > > dd has
> > > completed - at which point it's only allocated half the blocks
> > > needed to hold
> > > the 4M of data at one site. 5 seconds later, it's allocated the
> > > blocks at both
> > > sites and thus shows the full 8M needed for 2 copies.
> > >
> > > I've also seen (but haven't replicated it as I write this) a
small
> > > file (4-8K
> > > or so) showing first one full-sized block, then a second
full-sized
> > > block, and
> > > then dropping back to what's needed for 2 1/32nd fragments. That
> > had me
> > > scratching my head
> > >
> > > Having said that, that's all metadata fun and games, while your
case
> > > appears to have some problems with data integrity (which is a
whole
> > lot
> > > scarier). It would be *really* nice if we understood the
problem
> > here.
> > >
> > > The scariest part is:
> > >
> > > > The first grep | wc -l returns 1, because grep outputs "Binary
> > file /path/to/
> > > > gpfs/mount/test matches"
> > >
> > > which seems to be implying that we're failing on semantic
> > consistency.
> > > Basically, your 'cat' command is completing and closing the
file,
> > > but then a
> > > temporally later open of the same find is reading something
other
> > > that only the
> > > just-written data. My first guess is that it's a race condition
> > > similar to the
> > > following: The cat command is causing a write on one NSD server,
and
> > > the first
> > > grep results in a read from a *different* NSD server, returning
the
> > > data that
> > > *used* to be in the block because the read actually happens
before
> > > the first
> > > NSD server actually completes the write.
> > >
> > > It may be interesting to replace the grep's with pairs of 'ls
-ls /
> > > dd' commands to grab the
> > > raw data and its size, and check the following:
> > >
> > > 1) does the size (both blocks allocated and logical length)
reported
> > by
> > > ls match the amount of data actually read by the dd?
> > >
> > > 2) Is the file length as actually read equal to the written
length,
> > > or does it
> > > overshoot and read all the way to the next block boundary?
> > >
> > > 3) If the length is correct, what's wrong with the data that's
> > > telling grep that
> > > it's a binary file? ( od -cx is your friend here).
> > >
> > > 4) If it overshoots, is the remainder all-zeros (good) or does
it
> > > return semi-random
> > > "what used to be there" data (bad, due to data exposure issues)?
> > >
> > > (It's certainly not the most perplexing data consistency issue
I've
> > > hit in 4 decades - the
> > > winner *has* to be a intermittent data read corruption on a GPFS
3.5
> > > cluster that
> > > had us, IBM, SGI, DDN, and at least one vendor of networking
gear
> > > all chasing our
> > > tails for 18 months before we finally tracked it down. :)
> > >
> > > _______________________________________________
> > > gpfsug-discuss mailing list
> >
> > > gpfsug-discuss at spectrumscale.org <https://
> urldefense.proofpoint.com/v2/url?
> u=http-3A__spectrumscale.org&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=jUBFb8C9yai1TUTu1BVnNTNcOnJXGxupWiEKkEjT4pM&e=
> >
> > > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> > > <https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > gpfsug-discuss mailing list
> > > gpfsug-discuss at spectrumscale.org
> > > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> > >
> >
> > --
> > Aaron Knister
> > NASA Center for Climate Simulation (Code 606.2)
> > Goddard Space Flight Center
> > (301) 286-2776
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> >
> >
> >
> > ------------------------------
> >
> > Note: This email is for the confidential use of the named addressee(s)
> > only and may contain proprietary, confidential or privileged
information.
> > If you are not the intended recipient, you are hereby notified that
any
> > review, dissemination or copying of this email is strictly prohibited,
and
> > to please notify the sender immediately and destroy this email and any
> > attachments. Email transmission cannot be guaranteed to be secure or
> > error-free. The Company, therefore, does not make any guarantees as to
the
> > completeness or accuracy of this email or any attachments. This email
is
> > for informational purposes only and does not constitute a
recommendation,
> > offer, request or solicitation of any kind to buy, sell, subscribe,
redeem
> > or perform any type of transaction of a financial product.
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180214_d62fc203_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=nUcKIKr84CRhS0EbxV5vwjSlEr4p3Wf6Is3EDKvOjJg&e=
> >
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?
>
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-
>
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=
>
>
> End of gpfsug-discuss Digest, Vol 73, Issue 36
> **********************************************
>
--=_alternative 00783DF885258234_Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="US-ASCII"
<span style=" font-size:10pt;font-family:sans-serif">This could be related
to the following flash:</span><br><br><a \
href="http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012054"><span style=" \
font-size:10pt;color:blue;font-family:sans-serif">http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012054</span></a><br><br><span \
style=" font-size:10pt;font-family:sans-serif">You should contact IBM service to \
obtain the fix for your release.<br><br>Steve Y. Xiao<br></span><br><tt><span style=" \
font-size:10pt">gpfsug-discuss-bounces@spectrumscale.org wrote on 02/14/2018 02:18:02 \
PM:<br><br>> From: \
gpfsug-discuss-request@spectrumscale.org</span></tt><br><tt><span style=" \
font-size:10pt">> To: gpfsug-discuss@spectrumscale.org</span></tt><br><tt><span \
style=" font-size:10pt">> Date: 02/14/2018 02:18 PM</span></tt><br><tt><span \
style=" font-size:10pt">> Subject: gpfsug-discuss Digest, Vol 73, Issue \
36</span></tt><br><tt><span style=" font-size:10pt">> Sent by: \
gpfsug-discuss-bounces@spectrumscale.org</span></tt><br><tt><span style=" \
font-size:10pt">> <br>> Send gpfsug-discuss mailing list submissions to<br>> \
gpfsug-discuss@spectrumscale.org<br>> <br>> To subscribe or \
unsubscribe via the World Wide Web, visit<br>> </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
or, via email, send a message with subject or body 'help' to<br>> \
gpfsug-discuss-request@spectrumscale.org<br>> <br>> You can reach the \
person managing the list at<br>> \
gpfsug-discuss-owner@spectrumscale.org<br>> <br>> When replying, please \
edit your Subject line so it is more specific<br>> than "Re: Contents of \
gpfsug-discuss digest..."<br>> <br>> <br>> Today's Topics:<br>> \
<br>> 1. Re: Odd behavior with cat followed by grep. (John \
Hanks)<br>> <br>> <br>> \
----------------------------------------------------------------------<br>> \
<br>> Message: 1<br>> Date: Wed, 14 Feb 2018 11:17:19 -0800<br>> From: John \
Hanks <griznog@gmail.com><br>> To: gpfsug main discussion list \
<gpfsug-discuss@spectrumscale.org><br>> Subject: Re: [gpfsug-discuss] Odd \
behavior with cat followed by grep.<br>> Message-ID:<br>> \
<CAGrHuK7okijzHLysDNCxR8tn1fKeMDwW_iT5898tBvaCR_QJUg@mail.gmail.com><br>> \
Content-Type: text/plain; charset="utf-8"<br>> <br>> Thanks Bryan, \
mystery solved :)<br>> <br>> We also stumbled across these related items, in \
case anyone else wanders<br>> into this thread.<br>> <br>> </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__bug-2Dgrep.gnu.narkive.com_Y8cfvWDt_bug-2D27666-2Dgrep-2Don-2Dgpfs-2Dfilesystem-2Dseek-2Dhole-2Dproblem&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=FgxYBxqHZ0bHdWirEs1U_B3oDpeHJe8iRd-<br>> \
TYrXh6FI&e=<br>> <br>> </span></tt><a \
href=https://www.ibm.com/developerworks/community/forums/html/topic?><tt><span \
style=" font-size:10pt">https://www.ibm.com/developerworks/community/forums/html/topic?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> id=c2a94433-9ec0-4a4b-abfe-d0a1e721d630<br>> \
<br>> GPFS, the gift that keeps on giving ... me more things to do instead \
of<br>> doing the things I want to be doing.<br>> <br>> Thanks all,<br>> \
<br>> jbh<br>> <br>> On Wed, Feb 14, 2018 at 10:48 AM, Bryan Banister \
<bbanister@jumptrading.com><br>> wrote:<br>> <br>> > Hi \
all,<br>> ><br>> ><br>> ><br>> > We found this a while back \
and IBM fixed it. Here?s your answer:<br>> > </span></tt><a \
href="http://www-01.ibm.com/support/docview.wss?uid=isg1IV87385"><tt><span style=" \
font-size:10pt">http://www-01.ibm.com/support/docview.wss?uid=isg1IV87385</span></tt></a><tt><span \
style=" font-size:10pt"><br>> ><br>> ><br>> ><br>> > \
Cheers,<br>> ><br>> > -Bryan<br>> ><br>> ><br>> \
><br>> > *From:* gpfsug-discuss-bounces@spectrumscale.org [</span></tt><a \
href="mailto:gpfsug-discuss-"><tt><span style=" \
font-size:10pt">mailto:gpfsug-discuss-</span></tt></a><tt><span style=" \
font-size:10pt"><br>> > bounces@spectrumscale.org] *On Behalf Of *John \
Hanks<br>> > *Sent:* Wednesday, February 14, 2018 12:31 PM<br>> > *To:* \
gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> > \
*Subject:* Re: [gpfsug-discuss] Odd behavior with cat followed by grep.<br>> \
><br>> ><br>> ><br>> > *Note: External Email*<br>> > \
------------------------------<br>> ><br>> > Straces are interesting, but \
don't immediately open my eyes:<br>> ><br>> ><br>> ><br>> > \
strace of grep on NFS (works as expected)<br>> ><br>> ><br>> \
><br>> > openat(AT_FDCWD, "/home/griznog/pipetest.tmp.txt", \
O_RDONLY) = 3<br>> ><br>> > fstat(3, {st_mode=S_IFREG|0644, \
st_size=530721, ...}) = 0<br>> ><br>> > ioctl(3, TCGETS, 0x7ffe2c26b0b0) \
=
-1 ENOTTY (Inappropriate ioctl<br>> > for device)<br>> ><br>> > \
read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768<br>> \
><br>> > lseek(3, 32768, SEEK_HOLE) \
= 530721<br>> ><br>> > lseek(3, 32768, SEEK_SET) \
= 32768<br>> ><br>> > fstat(1, \
{st_mode=S_IFREG|0644, st_size=5977, ...}) = 0<br>> ><br>> > mmap(NULL, \
8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) =<br>> > 0x7f3bf6c43000<br>> ><br>> > write(1, \
"chr1\t43452652\t43452652\tL1HS\tlib1"..., 8192chr1<br>> ><br>> \
><br>> ><br>> > strace on GPFS (thinks file is binary)<br>> \
><br>> ><br>> ><br>> > openat(AT_FDCWD, \
"/srv/gsfs0/projects/pipetest.tmp.txt", O_RDONLY) = 3<br>> ><br>> \
> fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0<br>> ><br>> \
> ioctl(3, TCGETS, 0x7ffc9b52caa0) =
-1 ENOTTY (Inappropriate ioctl<br>> > for device)<br>> ><br>> > \
read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768<br>> \
><br>> > lseek(3, 32768, SEEK_HOLE) \
= 262144<br>> ><br>> > lseek(3, 32768, SEEK_SET) \
= 32768<br>> ><br>> > fstat(1, \
{st_mode=S_IFREG|0644, st_size=6011, ...}) = 0<br>> ><br>> > mmap(NULL, \
8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) =<br>> > 0x7fd45ee88000<br>> ><br>> > close(3) \
\
= 0<br>> ><br>> > write(1, "Binary file \
/srv/gsfs0/projects/"..., 72Binary file<br>> > \
/srv/gsfs0/projects/levinson/xwzhu/pipetest.tmp.txt matches<br>> ><br>> > \
) = 72<br>> ><br>> ><br>> ><br>> > Do the lseek() results \
indicate that the grep on the GPFS mounted version<br>> > thinks the file is a \
sparse file? For comparison I strace'd md5sum in place<br>> > of the grep and \
it does not lseek() with SEEK_HOLE, it's access in both<br>> > cases look \
identical, like:<br>> ><br>> ><br>> ><br>> > \
open("/home/griznog/pipetest.tmp.txt", O_RDONLY) = 3<br>> ><br>> \
> fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0<br>> ><br>> > fstat(3, \
{st_mode=S_IFREG|0644, st_size=530721, ...}) = 0<br>> ><br>> > mmap(NULL, \
8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) =<br>> > 0x7fb7d2c2b000<br>> ><br>> > read(3, \
"chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768<br>> \
><br>> > ...[reads clipped]...<br>> ><br>> > read(3, \
"", 24576) \
= 0<br>> ><br>> > lseek(3, 0, SEEK_CUR) \
= 530721<br>> ><br>> > close(3) \
\
= 0<br>> ><br>> ><br>> ><br>> \
><br>> ><br>> > jbh<br>> ><br>> ><br>> ><br>> \
><br>> ><br>> > On Wed, Feb 14, 2018 at 9:51 AM, Aaron Knister \
<aaron.s.knister@nasa.gov><br>> > wrote:<br>> ><br>> > Just \
speculating here (also known as making things up) but I wonder if<br>> > grep \
is somehow using the file's size in its determination of binary<br>> > status. \
I also see mmap in the strace so maybe there's some issue with<br>> > mmap \
where some internal GPFS buffer is getting truncated<br>> > inappropriately but \
leaving a bunch of null values which gets returned<br>> > to grep.<br>> \
><br>> > -Aaron<br>> ><br>> > On 2/14/18 10:21 AM, John Hanks \
wrote:<br>> > > Hi Valdis,<br>> > ><br>> > > I tired with \
the grep replaced with 'ls -ls' and 'md5sum', I don't think<br>> > > this is \
a data integrity issue, thankfully:<br>> > ><br>> > > $ \
./pipetestls.sh<br>> > > 256 -rw-r--r-- 1 39073 3001 530721 Feb 14 \
07:16<br>> > > /srv/gsfs0/projects/pipetest.tmp.txt<br>> > > 0 \
-rw-r--r-- 1 39073 3953 530721 Feb 14 07:16<br>> > \
/home/griznog/pipetest.tmp.txt<br>> > ><br>> > > $ \
./pipetestmd5.sh<br>> > > 15cb81a85c9e450bdac8230309453a0a \
/srv/gsfs0/projects/pipetest.tmp.txt<br>> > > \
15cb81a85c9e450bdac8230309453a0a /home/griznog/pipetest.tmp.txt<br>> > \
><br>> > > And replacing grep with 'file' even properly sees the files as \
ASCII:<br>> > > $ ./pipetestfile.sh<br>> > > \
/srv/gsfs0/projects/pipetest.tmp.txt: ASCII text, with very long lines<br>> > \
> /home/griznog/pipetest.tmp.txt: ASCII text, with very long lines<br>> > \
><br>> > > I'll poke a little harder at grep next and see what the \
difference in<br>> > > strace of each reveals.<br>> > ><br>> \
> > Thanks,<br>> > ><br>> > > jbh<br>> > ><br>> \
> ><br>> > ><br>> > ><br>> > > On Wed, Feb 14, 2018 \
at 7:08 AM, <valdis.kletnieks@vt.edu<br>> ><br>> > > \
<</span></tt><a href=mailto:valdis.kletnieks@vt.edu><tt><span style=" \
font-size:10pt">mailto:valdis.kletnieks@vt.edu</span></tt></a><tt><span style=" \
font-size:10pt">>> wrote:<br>> > ><br>> > > On \
Wed, 14 Feb 2018 06:20:32 -0800, John Hanks said:<br>> > ><br>> > > \
> # ls -aln /srv/gsfs0/projects/pipetest.tmp.txt<br>> > \
$HOME/pipetest.tmp.txt<br>> > > > -rw-r--r-- 1 39073 3953 \
530721 Feb 14 06:10<br>> > /home/griznog/pipetest.tmp.txt<br>> > > \
> -rw-r--r-- 1 39073 3001 530721 Feb 14 06:10<br>> > > \
> /srv/gsfs0/projects/pipetest.tmp.txt<br>> > > \
><br>> > > > We can "fix" the user case \
that exposed this by not using a temp<br>> > file or<br>> > > \
> inserting a sleep, but I'd still like to know why GPFS is \
behaving<br>> > this way<br>> > > > and make it \
stop.<br>> > ><br>> > > May be related to \
replication, or other behind-the-scenes behavior.<br>> > ><br>> > > \
Consider this example - 4.2.3.6, data and metadata replication \
both<br>> > > set to 2, 2 sites 95 cable miles apart, each is \
3 Dell servers with<br>> > > a full<br>> > > \
fiberchannel mesh to 3 Dell MD34something arrays.<br>> > ><br>> \
> > % dd if=/dev/zero bs=1k count=4096 of=sync.test; ls -ls \
sync.test;<br>> > > sleep 5; ls -ls sync.test; sleep 5; ls -ls \
sync.test<br>> > > 4096+0 records in<br>> > > \
4096+0 records out<br>> > > 4194304 bytes (4.2 MB) \
copied, 0.0342852 s, 122 MB/s<br>> > > 2048 -rw-r--r-- 1 root \
root 4194304 Feb 14 09:35 sync.test<br>> > > 8192 -rw-r--r-- 1 \
root root 4194304 Feb 14 09:35 sync.test<br>> > > 8192 \
-rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>> > ><br>> > \
> Notice that the first /bin/ls shouldn't be starting until after \
the<br>> > > dd has<br>> > > completed \
- at which point it's only allocated half the blocks<br>> > > \
needed to hold<br>> > > the 4M of data at one site. 5 \
seconds later, it's allocated the<br>> > > blocks at \
both<br>> > > sites and thus shows the full 8M needed for 2 \
copies.<br>> > ><br>> > > I've also seen (but haven't \
replicated it as I write this) a small<br>> > > file \
(4-8K<br>> > > or so) showing first one full-sized block, then \
a second full-sized<br>> > > block, and<br>> > > \
then dropping back to what's needed for 2 1/32nd fragments. \
That<br>> > had me<br>> > > scratching my \
head<br>> > ><br>> > > Having said that, that's all \
metadata fun and games, while your case<br>> > > appears to \
have some problems with data integrity (which is a whole<br>> > lot<br>> \
> > scarier). It would be *really* nice if we understood \
the problem<br>> > here.<br>> > ><br>> > > The \
scariest part is:<br>> > ><br>> > > > The first \
grep | wc -l returns 1, because grep outputs "Binary<br>> > file \
/path/to/<br>> > > > gpfs/mount/test matches"<br>> \
> ><br>> > > which seems to be implying that we're \
failing on semantic<br>> > consistency.<br>> > > \
Basically, your 'cat' command is completing and closing the file,<br>> > > \
but then a<br>> > > temporally later open of the \
same find is reading something other<br>> > > that only \
the<br>> > > just-written data. My first guess is that \
it's a race condition<br>> > > similar to the<br>> > \
> following: The cat command is causing a write on one NSD server, \
and<br>> > > the first<br>> > > grep \
results in a read from a *different* NSD server, returning the<br>> > > \
data that<br>> > > *used* to be in the block \
because the read actually happens before<br>> > > the \
first<br>> > > NSD server actually completes the \
write.<br>> > ><br>> > > It may be interesting to \
replace the grep's with pairs of 'ls -ls /<br>> > > dd' \
commands to grab the<br>> > > raw data and its size, and check \
the following:<br>> > ><br>> > > 1) does the size \
(both blocks allocated and logical length) reported<br>> > by<br>> > > \
ls match the amount of data actually read by the dd?<br>> > \
><br>> > > 2) Is the file length as actually read equal to \
the written length,<br>> > > or does it<br>> > > \
overshoot and read all the way to the next block boundary?<br>> > \
><br>> > > 3) If the length is correct, what's wrong with \
the data that's<br>> > > telling grep that<br>> > > \
it's a binary file? ( od -cx is your friend here).<br>> > \
><br>> > > 4) If it overshoots, is the remainder all-zeros \
(good) or does it<br>> > > return semi-random<br>> > \
> "what used to be there" data (bad, due to data exposure \
issues)?<br>> > ><br>> > > (It's certainly not the \
most perplexing data consistency issue I've<br>> > > hit in 4 \
decades - the<br>> > > winner *has* to be a intermittent data \
read corruption on a GPFS 3.5<br>> > > cluster that<br>> \
> > had us, IBM, SGI, DDN, and at least one vendor of networking \
gear<br>> > > all chasing our<br>> > > \
tails for 18 months before we finally tracked it down. :)<br>> > ><br>> \
> > _______________________________________________<br>> > \
> gpfsug-discuss mailing list<br>> ><br>> > > \
gpfsug-discuss at spectrumscale.org <https://<br>> \
urldefense.proofpoint.com/v2/url?<br>> \
u=http-3A__spectrumscale.org&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=jUBFb8C9yai1TUTu1BVnNTNcOnJXGxupWiEKkEjT4pM&e=<br>> \
><br>> > > </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
> > <</span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
><br>> > ><br>> > ><br>> > ><br>> > ><br>> \
> > _______________________________________________<br>> > > \
gpfsug-discuss mailing list<br>> > > gpfsug-discuss at \
spectrumscale.org<br>> > > </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
> ><br>> ><br>> > --<br>> > Aaron Knister<br>> > NASA \
Center for Climate Simulation (Code 606.2)<br>> > Goddard Space Flight \
Center<br>> > (301) 286-2776<br>> ><br>> > \
_______________________________________________<br>> > gpfsug-discuss mailing \
list<br>> > gpfsug-discuss at spectrumscale.org<br>> > </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
><br>> ><br>> ><br>> > ------------------------------<br>> \
><br>> > Note: This email is for the confidential use of the named \
addressee(s)<br>> > only and may contain proprietary, confidential or \
privileged information.<br>> > If you are not the intended recipient, you are \
hereby notified that any<br>> > review, dissemination or copying of this email \
is strictly prohibited, and<br>> > to please notify the sender immediately and \
destroy this email and any<br>> > attachments. Email transmission cannot be \
guaranteed to be secure or<br>> > error-free. The Company, therefore, does not \
make any guarantees as to the<br>> > completeness or accuracy of this email or \
any attachments. This email is<br>> > for informational purposes only and does \
not constitute a recommendation,<br>> > offer, request or solicitation of any \
kind to buy, sell, subscribe, redeem<br>> > or perform any type of transaction \
of a financial product.<br>> ><br>> > \
_______________________________________________<br>> > gpfsug-discuss mailing \
list<br>> > gpfsug-discuss at spectrumscale.org<br>> > </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
><br>> ><br>> -------------- next part --------------<br>> An HTML \
attachment was scrubbed...<br>> URL: <</span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20180214_d62fc203_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=nUcKIKr84CRhS0EbxV5vwjSlEr4p3Wf6Is3EDKvOjJg&e=<br>> \
><br>> <br>> ------------------------------<br>> <br>> \
_______________________________________________<br>> gpfsug-discuss mailing \
list<br>> gpfsug-discuss at spectrumscale.org<br>> </span></tt><a \
href=https://urldefense.proofpoint.com/v2/url?><tt><span style=" \
font-size:10pt">https://urldefense.proofpoint.com/v2/url?</span></tt></a><tt><span \
style=" font-size:10pt"><br>> \
u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br>> \
siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=Jv87Pffe4kSlhiO2NmMbL4HQo_zJ-8s8CkIRy7p92r4&s=aVWMptxcCR3po3ijmRweTyjbs1Pp5D7WEiJTYvSYLUk&e=<br>> \
<br>> <br>> End of gpfsug-discuss Digest, Vol 73, Issue 36<br>> \
**********************************************<br>> \
<br></span></tt><BR>
--=_alternative 00783DF885258234_=--
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic