[prev in list] [next in list] [prev in thread] [next in thread]
List: zfs-discuss
Subject: Re: [zfs-discuss] Resilver restarting several times
From: Richard Elling <richard.elling () gmail ! com>
Date: 2012-05-13 14:34:48
Message-ID: 6601DDC6-857F-4D93-909D-9E6DAE46F864 () gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
comments below...
On May 12, 2012, at 8:10 AM, Jim Klimov wrote:
> 2012-05-12 7:01, Jim Klimov wrote:
>> Overall the applied question is whether the disk will
>> make it back into the live pool (ultimately with no
>> continuous resilvering), and how fast that can be done -
>> I don't want to risk the big pool with nonredundant
>> arrays for too long.
>
> Here lies another "grumpy gripe", although maybe pertaining
> to the oldish snv_117 on that box: the system is not making
> its best possible effort to complete the resilver ASAP :)
>
> According to "iostat 60", disk utilizations of this raidz
> set vary 15-50%busy, queue lengths vary within 5 outstanding
> tasks, the CPU kernel time is 2-7% with over 90% idling,
> over 2GB RAM remains free... Why won't it go to complete
> the quest faster? Can some tire be kicked? ;)
>
> Sat May 12 19:06:09 MSK 2012
> extended device statistics
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 309.6 3.8 14863.0 5.0 0.0 4.7 0.0 15.0 0 65 c0t1d0
> 312.5 3.9 14879.7 5.1 0.0 4.6 0.0 14.7 0 64 c4t3d0
> 308.5 4.0 14855.0 5.2 0.0 4.7 0.0 15.1 0 66 c6t5d0
> 310.7 3.9 14855.7 5.1 0.0 4.6 0.0 14.8 0 65 c7t6d0
> 0.0 225.3 0.0 14484.2 0.0 8.1 0.0 36.0 0 83 c5t6d0
> Sat May 12 19:07:09 MSK 2012
> extended device statistics
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 228.0 3.0 6859.7 4.0 0.0 6.9 0.0 29.9 0 81 c0t1d0
> 227.7 3.3 6850.0 4.3 0.0 6.9 0.0 30.0 0 81 c4t3d0
> 228.1 3.4 6857.9 4.4 0.0 7.0 0.0 30.0 0 81 c6t5d0
> 227.6 3.1 6860.4 4.1 0.0 7.1 0.0 30.7 0 82 c7t6d0
> 0.0 225.8 0.0 6379.1 0.0 8.1 0.0 35.8 0 85 c5t6d0
In general asvc_t of this magnitude along with actv of this size
means you might be better off lowering zfs_vdev_max_pending.
> ...
>
> On some minutes the disks sit there doing almost nothing at all:
>
> Sat May 12 19:01:09 MSK 2012
> extended device statistics
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 10.7 0.8 665.4 0.7 0.0 0.1 0.0 11.4 0 13 c0t1d0
> 10.7 0.9 667.5 0.7 0.0 0.1 0.0 11.6 0 13 c4t3d0
> 10.7 0.8 666.4 0.7 0.0 0.1 0.0 11.9 0 13 c6t5d0
> 10.7 0.9 668.5 0.7 0.0 0.1 0.0 11.6 0 13 c7t6d0
> 0.1 15.5 0.6 20.3 0.0 0.0 0.0 0.2 0 0 c5t6d0
This behaviour cannot be debugged with iostat or any of the various
CPU-monitoring stat utilities. There is blocking somewhere, and it is
likely to be in the data path. You might try iosnoop and look for I/O
completion times that are large (> 1 second).
-- richard
> last pid: 18121; load avg: 0.16, 0.15, 0.12; up 0+16:03:44 19:06:51
> 96 processes: 95 sleeping, 1 on cpu
> CPU states: 96.6% idle, 0.2% user, 3.2% kernel, 0.0% iowait, 0.0% swap
> Memory: 16G phys mem, 2476M free mem, 16G total swap, 16G free swap
> ...
>
>> It has already taken 2 days to try and resilver a 250Gb
>> disk into the pool, but never made it past 100Gb progress. :(
>> Reports no errors that I'd see either... :)
>
> Well, that part seems to have been explained in my other
> mails, and hopefully worked-around by the hotspare.
>
> //Jim
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
ZFS Performance and Training
Richard.Elling@RichardElling.com
+1-760-896-4422
[Attachment #5 (unknown)]
<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space; ">comments below...<div><br><div><div>On May \
12, 2012, at 8:10 AM, Jim Klimov wrote:</div><br \
class="Apple-interchange-newline"><blockquote type="cite"><div>2012-05-12 7:01, Jim \
Klimov wrote:<br><blockquote type="cite">Overall the applied question is whether the \
disk will<br></blockquote><blockquote type="cite">make it back into the live pool \
(ultimately with no<br></blockquote><blockquote type="cite">continuous resilvering), \
and how fast that can be done -<br></blockquote><blockquote type="cite">I don't want \
to risk the big pool with nonredundant<br></blockquote><blockquote type="cite">arrays \
for too long.<br></blockquote><br>Here lies another "grumpy gripe", although maybe \
pertaining<br>to the oldish snv_117 on that box: the system is not making<br>its best \
possible effort to complete the resilver ASAP :)<br><br>According to "iostat 60", \
disk utilizations of this raidz<br>set vary 15-50%busy, queue lengths vary within 5 \
outstanding<br>tasks, the CPU kernel time is 2-7% with over 90% idling,<br>over 2GB \
RAM remains free... Why won't it go to complete<br>the quest faster? Can some tire be \
kicked? ;)<br><br>Sat May 12 19:06:09 MSK 2012<br> \
extended \
device statistics<br> r/s w/s kr/s \
kw/s wait actv wsvc_t asvc_t %w %b device<br> 309.6 \
3.8 14863.0 5.0 0.0 4.7 \
0.0 15.0 0 65 c0t1d0<br> 312.5 \
3.9 14879.7 5.1 0.0 4.6 \
0.0 14.7 0 64 c4t3d0<br> 308.5 \
4.0 14855.0 5.2 0.0 4.7 \
0.0 15.1 0 66 c6t5d0<br> 310.7 \
3.9 14855.7 5.1 0.0 4.6 \
0.0 14.8 0 65 c7t6d0<br> \
0.0 225.3 0.0 14484.2 0.0 8.1 \
0.0 36.0 0 83 c5t6d0<br>Sat May 12 \
19:07:09 MSK 2012<br> \
extended \
device statistics<br> r/s w/s kr/s \
kw/s wait actv wsvc_t asvc_t %w %b device<br> 228.0 \
3.0 6859.7 4.0 0.0 6.9 \
0.0 29.9 0 81 c0t1d0<br> 227.7 \
3.3 6850.0 4.3 0.0 6.9 \
0.0 30.0 0 81 c4t3d0<br> 228.1 \
3.4 6857.9 4.4 0.0 7.0 \
0.0 30.0 0 81 c6t5d0<br> 227.6 \
3.1 6860.4 4.1 0.0 7.1 \
0.0 30.7 0 82 c7t6d0<br> \
0.0 225.8 0.0 6379.1 0.0 8.1 \
0.0 35.8 0 85 \
c5t6d0<br></div></blockquote><div><br></div><div>In general asvc_t of this magnitude \
along with actv of this size </div><div>means you might be better off lowering \
zfs_vdev_max_pending.</div><br><blockquote type="cite"><div>...<br><br>On some \
minutes the disks sit there doing almost nothing at all:<br><br>Sat May 12 19:01:09 \
MSK 2012<br> extended \
device statistics<br> r/s w/s kr/s \
kw/s wait actv wsvc_t asvc_t %w %b device<br> \
10.7 0.8 665.4 0.7 0.0 \
0.1 0.0 11.4 0 13 c0t1d0<br> \
10.7 0.9 667.5 0.7 0.0 \
0.1 0.0 11.6 0 13 c4t3d0<br> \
10.7 0.8 666.4 0.7 0.0 \
0.1 0.0 11.9 0 13 c6t5d0<br> \
10.7 0.9 668.5 0.7 0.0 \
0.1 0.0 11.6 0 13 c7t6d0<br> \
0.1 15.5 0.6 20.3 \
0.0 0.0 0.0 0.2 0 \
0 c5t6d0<br></div></blockquote><div><br></div><div>This behaviour cannot \
be debugged with iostat or any of the various</div><div>CPU-monitoring stat \
utilities. There is blocking somewhere, and it is</div><div>likely to be in the data \
path. You might try iosnoop and look for I/O</div><div>completion times that are \
large (> 1 second).</div><div> -- richard</div><br><blockquote \
type="cite"><div>last pid: 18121; load avg: 0.16, 0.15, 0.12; \
up 0+16:03:44 19:06:51<br>96 processes: 95 sleeping, 1 on cpu<br>CPU states: 96.6% \
idle, 0.2% user, 3.2% kernel, 0.0% iowait, 0.0% swap<br>Memory: 16G phys mem, \
2476M free mem, 16G total swap, 16G free swap<br>...<br><br><blockquote \
type="cite">It has already taken 2 days to try and resilver a \
250Gb<br></blockquote><blockquote type="cite">disk into the pool, but never made it \
past 100Gb progress. :(<br></blockquote><blockquote type="cite">Reports no errors \
that I'd see either... :)<br></blockquote><br>Well, that part seems to have been \
explained in my other<br>mails, and hopefully worked-around by the \
hotspare.<br><br>//Jim<br>_______________________________________________<br>zfs-discuss \
mailing list<br><a href="mailto:zfs-discuss@opensolaris.org">zfs-discuss@opensolaris.o \
rg</a><br>http://mail.opensolaris.org/mailman/listinfo/zfs-discuss<br></div></blockquote></div><br><div \
apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: \
separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; \
font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: \
normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; \
white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: \
0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; \
"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, \
0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: \
normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: \
-webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; \
word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; \
-webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; \
"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; \
color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: \
normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; \
text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: \
normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; \
-webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; \
"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; \
color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: \
normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; \
text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: \
normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; \
-webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; \
"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; \
color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: \
normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; \
text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: \
normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; \
-webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; \
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; \
"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: \
after-white-space; "><div><div>--</div><div>ZFS Performance and Training</div><div><a \
href="mailto:Richard.Elling@RichardElling.com">Richard.Elling@RichardElling.com</a></div><div>+1-760-896-4422</div></div><div><br></div></div></span><br \
class="Apple-interchange-newline"></div></span><br \
class="Apple-interchange-newline"></div></span><br \
class="Apple-interchange-newline"></div></span><br \
class="Apple-interchange-newline"></span><br class="Apple-interchange-newline"> \
</div> <br></div></body></html>
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic