[prev in list] [next in list] [prev in thread] [next in thread]
List: illumos-developer
Subject: Re: [developer] zfs panic
From: "Igor Kozhukhov" <ikozhukhov () gmail ! com>
Date: 2016-06-28 19:40:43
Message-ID: D699D260-18B4-456F-B40E-73F1415B49ED () gmail ! com
[Download RAW message or body]
hi John,
i can see it by commit:
https://github.com/openzfs/openzfs/commit/373cb2f \
<https://github.com/openzfs/openzfs/commit/373cb2f>
and i vae it on my tree - just checked.
i have all commits from illumos on my dilos tree, but probably we have regression?
i have some others updates from Joynet tree with lx zones ports.
i'll try setup zfs tests env based on smartos and try to reproduce it.
could you try run zfs tests on your test openzfs env in loop for 2-3 days and take a \
look result? i have this panic after 4 tries without test VM reboot, but i have KVM \
VM with virtio drives. will be interested in results on aonther env.
Bets regards,
-Igor
> On Jun 28, 2016, at 10:28 PM, John Kennedy <john.kennedy@delphix.com> wrote:
>
> Hi Igor,
>
> On Tue, Jun 28, 2016 at 11:19 AM, Igor Kozhukhov <ikozhukhov@gmail.com \
> <mailto:ikozhukhov@gmail.com>> wrote: Hi All,
>
> I have found zfs panic:
>
> panic[cpu3]/thread=ffffff040b204880:
> assertion failed: rt->rt_space == 0 (0x200 == 0x0), file: ../../common/fs/zfs/ra
> nge_tree.c, line: 153
>
>
> ffffff001177aa00 genunix:process_type+18e829 ()
> ffffff001177aa30 zfs:range_tree_destroy+72 ()
> ffffff001177aa70 zfs:metaslab_fini+bf ()
> ffffff001177aab0 zfs:vdev_metaslab_fini+59 ()
> ffffff001177aaf0 zfs:vdev_free+d2 ()
> ffffff001177ab30 zfs:vdev_free+6e ()
> ffffff001177ab70 zfs:spa_unload+ca ()
> ffffff001177abe0 zfs:spa_export_common+118 ()
> ffffff001177ac00 zfs:spa_destroy+1a ()
> ffffff001177ac40 zfs:zfs_ioc_pool_destroy+1e ()
> ffffff001177ace0 zfs:zfsdev_ioctl+51a ()
> ffffff001177ad20 genunix:cdev_ioctl+25 ()
> ffffff001177ad70 specfs:spec_ioctl+50 ()
> ffffff001177adf0 genunix:fop_ioctl+55 ()
> ffffff001177aef0 genunix:ioctl+a4 ()
> ffffff001177af00 unix:brand_sys_syscall+32c ()
>
> syncing file systems...
>
> reproduce: i have configured Jenkins job to do zfs tests.
> on DilOS i have the same tests results as on OpenZFS builds.
>
> I have:
> i host with 2 CPU, 48GB RAM, 1 SATA drive, 1 SSD,
>
> I have configured kvm VM for zfs tests:
> 4 CPU, 4 GB RAM, 20GB primary virtio disk, 3 x 8GB virtio disks for zfs tests.
>
> virtio drives have been provided to kvm vm as zfs row disks.
>
> after first try i have the same known fialed 13 tests.
>
> tests circle about 3.5h
>
> no vm reboot - run zfs tests again.
> i can see 14 failed tests (13 known + 1 other) .
>
> start tests again - i can see 15 failed tests (13 known + 2 others)
>
> no VM reboots - just start tests again - i have zfs panic.
>
> This is a bug we've seen at Delphix. It's fixed, and you should be able to get \
> the fix by applying openzfs commit 373cb2f.
> --
> John Wren Kennedy
>
> 275 Middlefield Road, Suite 50
> Menlo Park, CA 94025
> http://www.delphix.com/ <http://www.delphix.com/>
-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/25758058-4e9228dc
Modify Your Subscription: \
https://www.listbox.com/member/?member_id=25758058&id_secret=25758058-c19b436a \
Powered by Listbox: http://www.listbox.com
[Attachment #3 (unknown)]
<html><html><html><head><meta content="text/html charset=utf-8" \
http-equiv="Content-Type" /></head><body class="" style="word-wrap: break-word; \
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">hi \
John,</div><div class=""><br class="" /></div><div class="">i can see it by \
commit:</div><div class=""><a class="" \
href="https://github.com/openzfs/openzfs/commit/373cb2f">https://github.com/openzfs/openzfs/commit/373cb2f</a></div><div \
class=""><br class="" /></div><div class="">and i vae it on my tree - just \
checked.</div><div class="">i have all commits from illumos on my dilos tree, but \
probably we have regression?</div><div class=""><br class="" /></div><div class="">i \
have some others updates from Joynet tree with lx zones ports.</div><div \
class="">i’ll try setup zfs tests env based on smartos and try to reproduce \
it.</div><div class=""><br class="" /></div><div class="">could you try run zfs tests \
on your test openzfs env in loop for 2-3 days and take a look result?</div><div \
class="">i have this panic after 4 tries without test VM reboot, but i have KVM VM \
with virtio drives.</div><div class="">will be interested in results on aonther \
env.</div><div class=""><br class="" /></div><div class="">Bets regards,</div><div \
class="">-Igor</div><div class=""><br class="" /></div><div><blockquote class="" \
type="cite"><div class="">On Jun 28, 2016, at 10:28 PM, John Kennedy <<a class="" \
href="mailto:john.kennedy@delphix.com">john.kennedy@delphix.com</a>> \
wrote:</div><br class="Apple-interchange-newline" /><div class=""><div class="" \
dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; \
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: \
auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; \
widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div \
class="gmail_default" style="font-family: arial, helvetica, sans-serif;">Hi \
Igor,</div><div class="gmail_extra"><br class="" /><div class="gmail_quote">On Tue, \
Jun 28, 2016 at 11:19 AM, Igor Kozhukhov<span \
class="Apple-converted-space"> </span><span class="" dir="ltr"><<a class="" \
href="mailto:ikozhukhov@gmail.com" \
target="_blank">ikozhukhov@gmail.com</a>></span><span \
class="Apple-converted-space"> </span>wrote:<br class="" /><blockquote \
class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; \
border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: \
1ex;"><div class="" style="word-wrap: break-word;">Hi All,<div class=""><br class="" \
/></div><div class="">I have found zfs panic:</div><div class=""><br class="" \
/></div><div class=""><div class="" style="margin: 0px; line-height: normal; \
font-family: 'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, \
0, 0);"><span class="">panic[cpu3]/thread=ffffff040b204880: </span></div><div \
class="" style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; \
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span class="">assertion \
failed: rt->rt_space == 0 (0x200 == 0x0), file: \
../../common/fs/zfs/ra</span></div><div class="" style="margin: 0px; line-height: \
normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">nge_tree.c, line: \
153</span></div><div class="" style="margin: 0px; line-height: normal; font-family: \
'Andale Mono'; color: rgb(41, 249, 20); min-height: 14px; background-color: \
rgb(0, 0, 0);"><span class=""></span><br class="" /></div><div class="" \
style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: \
rgb(41, 249, 20); min-height: 14px; background-color: rgb(0, 0, 0);"><span \
class=""></span><br class="" /></div><div class="" style="margin: 0px; line-height: \
normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177aa00 \
genunix:process_type+18e829 ()</span></div><div class="" style="margin: 0px; \
line-height: normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177aa30 \
zfs:range_tree_destroy+72 ()</span></div><div class="" style="margin: 0px; \
line-height: normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177aa70 zfs:metaslab_fini+bf \
()</span></div><div class="" style="margin: 0px; line-height: normal; font-family: \
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, \
0);"><span class="">ffffff001177aab0 zfs:vdev_metaslab_fini+59 ()</span></div><div \
class="" style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; \
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span \
class="">ffffff001177aaf0 zfs:vdev_free+d2 ()</span></div><div class="" \
style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: \
rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span class="">ffffff001177ab30 \
zfs:vdev_free+6e ()</span></div><div class="" style="margin: 0px; line-height: \
normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177ab70 zfs:spa_unload+ca \
()</span></div><div class="" style="margin: 0px; line-height: normal; font-family: \
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, \
0);"><span class="">ffffff001177abe0 zfs:spa_export_common+118 ()</span></div><div \
class="" style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; \
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span \
class="">ffffff001177ac00 zfs:spa_destroy+1a ()</span></div><div class="" \
style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: \
rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span class="">ffffff001177ac40 \
zfs:zfs_ioc_pool_destroy+1e ()</span></div><div class="" style="margin: 0px; \
line-height: normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177ace0 zfs:zfsdev_ioctl+51a \
()</span></div><div class="" style="margin: 0px; line-height: normal; font-family: \
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, \
0);"><span class="">ffffff001177ad20 genunix:cdev_ioctl+25 ()</span></div><div \
class="" style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; \
color: rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span \
class="">ffffff001177ad70 specfs:spec_ioctl+50 ()</span></div><div class="" \
style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; color: \
rgb(41, 249, 20); background-color: rgb(0, 0, 0);"><span class="">ffffff001177adf0 \
genunix:fop_ioctl+55 ()</span></div><div class="" style="margin: 0px; line-height: \
normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">ffffff001177aef0 genunix:ioctl+a4 \
()</span></div><div class="" style="margin: 0px; line-height: normal; font-family: \
'Andale Mono'; color: rgb(41, 249, 20); background-color: rgb(0, 0, \
0);"><span class="">ffffff001177af00 unix:brand_sys_syscall+32c ()</span></div><div \
class="" style="margin: 0px; line-height: normal; font-family: 'Andale Mono'; \
color: rgb(41, 249, 20); min-height: 14px; background-color: rgb(0, 0, 0);"><span \
class=""></span><br class="" /></div><div class="" style="margin: 0px; line-height: \
normal; font-family: 'Andale Mono'; color: rgb(41, 249, 20); \
background-color: rgb(0, 0, 0);"><span class="">syncing file \
systems...</span></div></div><div class=""><span class=""><br class="" \
/></span></div><div class="">reproduce: i have configured Jenkins job to do zfs \
tests.</div><div class="">on DilOS i have the same tests results as on OpenZFS \
builds.</div><div class=""><br class="" /></div><div class="">I have:</div><div \
class="">i host with 2 CPU, 48GB RAM, 1 SATA drive, 1 SSD,</div><div class=""><br \
class="" /></div><div class="">I have configured kvm VM for zfs tests:</div><div \
class="">4 CPU, 4 GB RAM, 20GB primary virtio disk, 3 x 8GB virtio disks for zfs \
tests.</div><div class=""><br class="" /></div><div class="">virtio drives have been \
provided to kvm vm as zfs row disks.</div><div class=""><br class="" /></div><div \
class="">after first try i have the same known fialed 13 tests.</div><div \
class=""><br class="" /></div><div class="">tests circle about 3.5h</div><div \
class=""><br class="" /></div><div class="">no vm reboot - run zfs tests \
again.</div><div class="">i can see 14 failed tests (13 known + 1 other) \
.</div><div class=""><br class="" /></div><div class="">start tests again - i can see \
15 failed tests (13 known + 2 others)</div><div class=""><br class="" /></div><div \
class="">no VM reboots - just start tests again - i have zfs \
panic.</div></div></blockquote><div class=""><br class="" /></div><div \
class="gmail_default" style="font-family: arial, helvetica, sans-serif;">​This \
is a bug we've seen at Delphix. It's fixed, and you should be able to get the \
fix by applying openzfs commit <span class="" style="font-family: arial, \
sans-serif;">373cb2f.</span>​</div></div><div class=""><br class="" \
/></div>--<span class="Apple-converted-space"> </span><br class="" /><div \
class="gmail_signature" data-smartmail="gmail_signature">John Wren Kennedy<br \
class="" /><br class="" />275 Middlefield Road, Suite 50<br class="" />Menlo Park, CA \
94025<br class="" /><a class="" href="http://www.delphix.com/" \
target="_blank">http://www.delphix.com/</a></div></div></div></div></blockquote></div><br \
class="" /><div bgcolor="#ffffff" id="listbox-footer" \
style="width:auto;margin:0;padding:5px;background-color:#fff;clear:both;border-top: \
1px solid #ccc;"><table bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" \
style="background-color:#fff" width="100%"><tr><td padding="4px"><font \
color="#333333" size="1" style="font-family:helvetica, sans-serif;"> \
<strong>illumos-developer</strong> | <a \
href="https://www.listbox.com/member/archive/182179/=now" \
style="text-decoration:none;color:#669933;border-bottom: 1px solid #444444" title="Go \
to archives for illumos-developer">Archives</a> <a border="0" \
href="https://www.listbox.com/member/archive/rss/182179/25758058-4e9228dc" \
style="text-decoration:none;color:#669933" title="RSS feed for \
illumos-developer"><img border="0" \
src="http://postlink.www.listbox.com/2151651/833487e62783d55fe81f119fb93ef644/25758058 \
/bb3fe179.jpg?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2ZlZWQtaWNvbi0xMHgxMC5qcGc" \
/></a> | <a href="https://www.listbox.com/member/?member_id=25758058&id_secret=25758058-c19b436a" \
style="text-decoration:none;color:#669933;border-bottom: 1px solid #444444" \
title="">Modify</a> Your Subscription<td align="right" valign="top"><a \
href="http://www.listbox.com" style="border-bottom:none;"> <img border="0" \
src="http://postlink.www.listbox.com/2151652/3379085af0f1cf7fc3708f04b4471ae2/25758058 \
/bb3fe179.png?uri=aHR0cHM6Ly93d3cubGlzdGJveC5jb20vaW1hZ2VzL2xpc3Rib3gtbG9nby1zbWFsbC5wbmc" \
title="Powered by Listbox" \
/></a></td></font></td></tr></table></div></body></html></html></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic