[prev in list] [next in list] [prev in thread] [next in thread]
List: opensolaris-storage-discuss
Subject: Re: [zfs-discuss] Re: [storage-discuss] NCQ performance
From: "Robert B. Wood" <Robert.B.Wood () Sun ! COM>
Date: 2007-05-30 14:25:12
Message-ID: 86566E73-B439-4A14-B3C7-611E62E02BD0 () Sun ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On May 29, 2007, at 2:59 PM, johansen-osdev@sun.com wrote:
>> When sequential I/O is done to the disk directly there is no
>> performance
>> degradation at all.
>
> All filesystems impose some overhead compared to the rate of raw disk
> I/O. It's going to be hard to store data on a disk unless some
> kind of
> filesystem is used. All the tests that Eric and I have performed show
> regressions for multiple sequential I/O streams. If you have data
> that
> shows otherwise, please feel free to share.
>
>> [I]t does not take any additional time in ldi_strategy(),
>> bdev_strategy(), mv_rw_dma_start(). In some instance it actually
>> takes less time. The only thing that sometimes takes additional
>> time
>> is waiting for the disk I/O.
>
> Let's be precise about what was actually observed. Eric and I saw
> increased service times for the I/O on devices with NCQ enabled when
> running multiple sequential I/O streams. Everything that we observed
> indicated that it actually took the disk longer to service requests
> when
> many sequential I/Os were queued.
>
> -j
>
It could very well be that on-disc cache is being partitioned
differently when NCQ is enabled in certain implementations. For
example, with NCQ disabled, on disc look ahead may be enabled,
netting sequential I/O improvements. Just guessing, as this level of
disc implementation detail is vendor specific and generally
proprietary. I would not expect the elevator sort algorithm to
impose any performance penalty unless it were fundamentally flawed.
There's a bit of related discussion here
I'm actually struck by the minimal gains being seen in random I/O. A
few years ago, when NCQ was in prototype, I saw better than 50%
improvement in average random I/O response time with large queue
depths. My gut feeling is that the issue is farther up the stack .. Bob
>
> _______________________________________________
> storage-discuss mailing list
> storage-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/storage-discuss
[Attachment #5 (text/html)]
<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: \
after-white-space; "><BR><DIV><DIV>On May 29, 2007, at 2:59 PM, <A \
href="mailto:johansen-osdev@sun.com">johansen-osdev@sun.com</A> wrote:</DIV><BR \
class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><BLOCKQUOTE \
type="cite"><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">When sequential I/O is done to the disk directly there is no \
performance</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">degradation at all. <SPAN class="Apple-converted-space"> \
</SPAN></DIV> </BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">All filesystems impose some overhead compared to the rate of raw disk</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">I/O.<SPAN class="Apple-converted-space"> </SPAN>It's going to be hard to store \
data on a disk unless some kind of</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; ">filesystem is used.<SPAN \
class="Apple-converted-space"> </SPAN>All the tests that Eric and I have performed \
show</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">regressions for multiple sequential I/O streams.<SPAN \
class="Apple-converted-space"> </SPAN>If you have data that</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">shows otherwise, please feel free to share.</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><BR></DIV> <BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">[I]t does not take any additional time in \
ldi_strategy(),</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">bdev_strategy(), mv_rw_dma_start().<SPAN \
class="Apple-converted-space"> </SPAN>In some instance it actually</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">takes less time. <SPAN class="Apple-converted-space"> </SPAN>The only thing that \
sometimes takes additional time</DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">is waiting for the disk I/O.</DIV> \
</BLOCKQUOTE><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Let's be precise about \
what was actually observed.<SPAN class="Apple-converted-space"> </SPAN>Eric and I \
saw</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">increased service times for the I/O on devices with NCQ enabled \
when</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">running multiple sequential I/O streams.<SPAN \
class="Apple-converted-space"> </SPAN>Everything that we observed</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">indicated that it actually took the disk longer to service requests when</DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">many sequential I/Os were queued.</DIV><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><BR></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">-j</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><BR></DIV></BLOCKQUOTE>It could very well be \
that on-disc cache is being partitioned differently when NCQ is enabled in certain \
implementations. For example, with NCQ disabled, on disc look ahead may be enabled, \
netting sequential I/O improvements. Just guessing, as this level of disc \
implementation detail is vendor specific and generally proprietary. I would not \
expect the elevator sort algorithm to impose any performance penalty unless it were \
fundamentally flawed.</DIV><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>There's a bit of related discussion <A \
href="http://www.futurehardware.in/555851.htm">here</A></DIV><DIV><BR \
class="khtml-block-placeholder"></DIV><DIV>I'm actually struck by the minimal gains \
being seen in random I/O. A few years ago, when NCQ was in prototype, I saw better \
than 50% improvement in average random I/O response time with large queue depths. My \
gut feeling is that the issue is farther up the stack .. \
Bob</DIV><DIV><BR><BLOCKQUOTE type="cite"><DIV style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><BR></DIV><DIV style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; \
">_______________________________________________</DIV><DIV style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">storage-discuss mailing \
list</DIV><DIV style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; "><A \
href="mailto:storage-discuss@opensolaris.org">storage-discuss@opensolaris.org</A></DIV><DIV \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><A \
href="http://mail.opensolaris.org/mailman/listinfo/storage-discuss">http://mail.opensolaris.org/mailman/listinfo/storage-discuss</A></DIV> \
</BLOCKQUOTE></DIV><BR></BODY></HTML>
_______________________________________________
storage-discuss mailing list
storage-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic