[prev in list] [next in list] [prev in thread] [next in thread]
List: busybox
Subject: Re: [bug] dd piped input truncates ~1% of values to %4096 on multi-core systems
From: Nicholas Clark <nicholas.clark () gmail ! com>
Date: 2020-11-30 18:15:03
Message-ID: CAKNeuBrEN256zYu9OjfaRUQfJOuvuz4JwMTuUR=peBmHAUzG9g () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
iflag=fullblock is the right solution. I added that feature a couple of
years ago. I thought it should be busybox's default behavior, but that goes
against the POSIX spec.
POSIX dd specifies that one block == one read() call, unless modified by an
iflag. Read() calls can return partial data pretty often, especially when
reading from pipes. A filesystem read() will _usually_ give you as much
data as you request, but it's not guaranteed at all.
So if you want to make _sure_ you're getting all of the data you want, the
only guaranteed safe approach is to use iflag=fullblock. It's good practice
to always use it with dd.
On Mon, Nov 30, 2020 at 2:11 AM John Thomson <
lists@johnthomson.fastmail.com.au> wrote:
> On Mon, 30 Nov 2020, at 08:39, Walter Harms wrote:
> > did you play with the buffering (man stdbuf) ?
> > I suspect the kernel has a more aggressive buffering these days.
> > In one project i had to add fflush() to make sure that i really get all
> > data. I never found a good explanation.
>
> I will look into it. Thank you.
>
> I have now found the fullblock iflag [1], which consistently gives me the
> expected result.
> Guessing the issue I was seeing relates to my bs being greater than
> PIPE_BUF?
>
> [1]:
> https://git.busybox.net/busybox/commit/coreutils/dd.c?id=77a6678c427db5ea15d6d0005a579f441277df06
>
> Cheers,
> --
> John Thomson
> _______________________________________________
> busybox mailing list
> busybox@busybox.net
> http://lists.busybox.net/mailman/listinfo/busybox
>
[Attachment #5 (text/html)]
<div dir="ltr"><div>iflag=fullblock is the right solution. I added that feature a
couple of years ago. I thought it should be busybox's default behavior,
but that goes against the POSIX spec.<br><br></div><div>POSIX dd
specifies that one block == one read() call, unless modified by an
iflag. Read() calls can return partial data pretty often, especially
when reading from pipes. A filesystem read() will _usually_ give you as
much data as you request, but it's not guaranteed at \
all.<br></div><div><br></div><div>So if you want to make _sure_ you're getting \
all of the data you want, the only guaranteed safe approach is to use \
iflag=fullblock. It's good practice to always use it with \
dd.</div><div><br><br></div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Nov 30, 2020 at 2:11 AM John Thomson <<a \
href="mailto:lists@johnthomson.fastmail.com.au">lists@johnthomson.fastmail.com.au</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 30 Nov 2020, \
at 08:39, Walter Harms wrote:<br> > did you play with the buffering (man stdbuf) \
?<br> > I suspect the kernel has a more aggressive buffering these days.<br>
> In one project i had to add fflush() to make sure that i really get all<br>
> data. I never found a good explanation.<br>
<br>
I will look into it. Thank you.<br>
<br>
I have now found the fullblock iflag [1], which consistently gives me the expected \
result.<br> Guessing the issue I was seeing relates to my bs being greater than \
PIPE_BUF?<br> <br>
[1]: <a href="https://git.busybox.net/busybox/commit/coreutils/dd.c?id=77a6678c427db5ea15d6d0005a579f441277df06" \
rel="noreferrer" target="_blank">https://git.busybox.net/busybox/commit/coreutils/dd.c?id=77a6678c427db5ea15d6d0005a579f441277df06</a><br>
<br>
Cheers,<br>
-- <br>
John Thomson<br>
_______________________________________________<br>
busybox mailing list<br>
<a href="mailto:busybox@busybox.net" target="_blank">busybox@busybox.net</a><br>
<a href="http://lists.busybox.net/mailman/listinfo/busybox" rel="noreferrer" \
target="_blank">http://lists.busybox.net/mailman/listinfo/busybox</a><br> \
</blockquote></div>
_______________________________________________
busybox mailing list
busybox@busybox.net
http://lists.busybox.net/mailman/listinfo/busybox
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic