[prev in list] [next in list] [prev in thread] [next in thread]
List: xen-users
Subject: Re: [Xen-users] Dom0 crashes without logging lately on Debian Stretch with Xen 4.8
From: "Roalt Zijlstra | webpower" <roalt.zijlstra () webpower ! nl>
Date: 2018-10-30 15:39:05
Message-ID: CAC7+sxWVvhAajccATndSFUJ9pjNRWOFbQmNhL5rW3sGJJXiuNg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Jean-Louis,
Thanks for sharing this info.
If I look at my set of servers then we also have CentOS 6 servers which
used to crash after 3 or 4 weeks.
However with the 4.9.112-32.el6.x86_64 kernel and Xen 4.8.4-1.el6 it looks
to be more stable with 65+ days of uptime on six servers running that
particular setup.
I still think that the 4.8.4-pre Xen package for Debian is the stable
version, so if you think of upgrading to 4.8.5-pre I would not recommend
it yet.
Best regards,
Roalt Zijlstra
Teamleader Infra & Deliverability
roalt.zijlstra@webpower.nl
+31 342 423 262
roalt.zijlstra
https://www.webpower-group.com
<https://www.webpower-group.com/>
[image: Facebook]
<https://www.facebook.com/webpower.marketingautomation/> [image:
Twitter] <https://twitter.com/webpower> [image: Linkedin]
<https://www.linkedin.com/company/36782/>
Barcelona | Barneveld | Beijing | Chengdu | Guangzhou
Hamburg | Shanghai | Shenzhen | Stockholm
<https://webpower.nl/event/kennissessies/?utm_source=GML&utm_medium=EMAIL&utm_campaign=EVENT&utm_term=KNOWLDGS&utm_content=NL>
Op di 30 okt. 2018 om 12:12 schreef Jean-Louis Dupond <jean-louis@dupond.be
> >
> Hi Roalt,
>
> We are running Xen 4.6 on 4.9.x kernel and CentOS 6, and are having the
> same issues.
> But not that frequent as you state. Only like once every month.
>
> The systems (Dell R630) also crashes/resets without any message. So
> nothing is logged unfortunately :(
>
> The crashes were not observed on Xen 4.4.
>
> We configured the servers to print kernel logs to SOL (Serial Over LAN via
> iDRAC), and we log those.
> But since then no crashed servers anymore, so we don't know yet if this
> will give us some more details.
>
>
> Thanks
> Jean-Louis
> On 29/10/18 12:57, Roalt Zijlstra | webpower wrote:
>
> Hi there,
>
> Ever since all the Meltdown and Spectre kernel updates and possibly also
> Xen 4.8 updates, we experience crashes of the Dom0 just out of the blue.
> Sometimes after 1 day, sometimes after a few days or even 14 days,
> completely random.
>
> We have two Dell P730 servers and two Dell P720 servers with this
> behaviour. One thing is that we updated these machine to the latest
> available firmware, because that is the most secure way. Then we installed
> Debian Stretch with Xen 4.8 support
>
> We have done serveral installs and 4 servers seem to crash pretty fast and
> other don't. In the end we think that we can lead it back to the
> xen-4.8.4-pre version being stable and the xen-4.8.5-pre being unstable.
> This was kinda independent of the kernel that we were using 4.14 or
> 4.9.0-8-amd64. This is off course all Debian package numbering.
>
> As last resort we updated on one server all DomU kernels of our Jessie
> servers on this Dom0 to 4.9.0 from backports instead of the 3.16 kernel.
> For now that seems to work, but the crashes are random so it could happen
> any time again. The idea is that these kernels are completely spectre&
> meltdown unaware and might cause trouble in Xen kernel support. I am not
> sure if this is true at all, but we are pretty lost what the actual cause
> is.
>
> We also tested with CentOS and we also had these crashes there with
> certain combinations of kernel/Xen. The most recent updates seem to be more
> stable tough. The most frustrating part is the there is absolutely no logs
> to be found. No kernel oops or what.. the server just resets and boots
> again.
>
> Are there others experiencing problems like this? Do you see more frequent
> server/kernel crashes on production servers?
>
> Best regards,
> Roalt Zijlstra
>
>
> _______________________________________________
> Xen-users mailing listXen-users@lists.xenproject.orghttps://lists.xenproject.org/mailman/listinfo/xen-users
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Hi \
Jean-Louis,</div><div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div \
class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Thanks for sharing \
this info. </div><div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div \
class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">If I look at my set of \
servers then we also have CentOS 6 servers which used to crash after 3 or 4 \
weeks.</div>However with the 4.9.112-32.el6.x86_64 kernel and Xen 4.8.4-1.el6 it \
looks to be more stable<span class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"> with 65+ days of \
uptime on six servers running that particular setup.</span></div><div dir="ltr"><font \
face="arial, helvetica, sans-serif"><br></font></div><div dir="ltr"><font \
face="arial, helvetica, sans-serif"><span class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">I still think that the \
4.8.4-pre Xen package for Debian is the stable version, so if you think of upgrading \
to 4.8.5-pre I would not recommend it yet.</span></font><div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br></div><div><div \
dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div \
style="color:rgb(102,102,102);font-family:arial,helvetica,sans-serif;font-size:12.8px"><div \
dir="ltr"><div dir="ltr" \
style="font-size:12.8px;color:rgb(34,34,34);font-family:arial,sans-serif"><table \
width="100%" cellpadding="0" cellspacing="0" \
style="color:rgb(0,0,0);font-family:"Times New \
Roman";font-size:medium"><tbody><tr><td valign="top" align="left" \
style="color:rgb(88,89,91);font-size:13px;line-height:16px"><table width="100%" \
cellpadding="0" cellspacing="0" style="font-family:"Times New \
Roman";color:rgb(0,0,0);font-size:medium"><tbody><tr><td valign="top" \
align="left" style="color:rgb(88,89,91);font-size:13px;line-height:16px"><font \
face="verdana, sans-serif">Best regards,</font></td></tr><tr><td valign="top" \
align="left" width="20"><font face="verdana, sans-serif"> \
</font></td></tr></tbody></table><table width="320" cellpadding="0" cellspacing="0" \
style="color:rgb(0,0,0);font-size:medium"><tbody><tr><td valign="top" align="left" \
width="20"><img src="https://www.webpower.nl/signature/images/icon_user.jpg" \
title="Naam" width="20" style="display: block;"></td><td valign="top" align="left" \
width="20"><font face="verdana, sans-serif"> </font></td><td valign="middle" \
align="left" style="color:rgb(102,44,138);font-size:18px;font-weight:bold;line-height:16px"><font \
face="verdana, sans-serif">Roalt Zijlstra</font></td></tr><tr><td valign="top" \
align="left" width="20"><font face="verdana, sans-serif"> </font></td><td \
valign="top" align="left" width="20"><font face="verdana, sans-serif"> \
</font></td><td valign="middle" align="left" \
style="font-style:italic;color:rgb(88,89,91);font-size:13px;line-height:16px"><font \
face="verdana, sans-serif">Teamleader Infra & \
Deliverability</font></td></tr><tr><td valign="top" align="left" width="20"><font \
face="verdana, sans-serif"> </font></td><td valign="top" align="left" \
width="20"><font face="verdana, sans-serif"> </font></td><td valign="top" \
align="left" style="color:rgb(88,89,91);font-size:13px;line-height:16px"><font \
face="verdana, sans-serif"> </font></td></tr><tr><td valign="top" align="left" \
width="20"><img src="https://www.webpower.nl/signature/images/icon_email.jpg" \
width="20" title="Email" style="display: block; width: 20px;"></td><td valign="top" \
align="left" width="20"><font face="verdana, sans-serif"> </font></td><td \
valign="middle" align="left" \
style="color:rgb(88,89,91);font-size:13px;line-height:16px"><a \
href="mailto:roalt.zijlstra@webpower.nl" title="webpower" \
style="color:rgb(245,126,55)" target="_blank"><font face="verdana, \
sans-serif">roalt.zijlstra@webpower.nl</font></a></td></tr><tr><td valign="top" \
align="left" width="20"><img \
src="https://www.webpower.nl/signature/images/icon_phone.jpg" width="20" \
title="Phone" style="display: block; width: 20px;"></td><td valign="top" align="left" \
width="20"><font face="verdana, sans-serif"> </font></td><td valign="middle" \
align="left" style="color:rgb(88,89,91);font-size:13px;line-height:16px"><font \
face="verdana, sans-serif">+31 342 423 262</font></td></tr><tr><td valign="top" \
align="left" width="20"><img \
src="https://www.webpower.nl/signature/images/icon_skype.jpg" title="Skype" \
width="20" style="display: block; width: 20px;"></td><td valign="top" align="left" \
width="20"><font face="verdana, sans-serif"> </font></td><td valign="middle" \
align="left" style="color:rgb(88,89,91);font-size:13px;line-height:16px"><font \
face="verdana, sans-serif">roalt.zijlstra</font></td></tr><tr><td valign="top" \
align="left" width="20"><img \
src="https://www.webpower.nl/signature/images/icon_url.jpg" width="20" title="Phone" \
style="display: block; width: 20px;"></td><td valign="top" align="left" \
width="20"><font face="verdana, sans-serif"> </font></td><td valign="middle" \
align="left" style="color:rgb(88,89,91);font-size:13px;line-height:16px"><a \
href="https://www.webpower-group.com/" title="webpower" style="color:rgb(245,126,55)" \
target="_blank"><font face="verdana, \
sans-serif">https://www.webpower-group.com</font></a></td></tr><tr><td valign="top" \
align="left" width="20"><font face="verdana, sans-serif"> </font></td><td \
valign="top" align="left" width="20"></td><td valign="top" align="left" \
style="color:rgb(88,89,91);font-size:13px;line-height:16px"></td></tr></tbody></table><table \
width="320" cellpadding="0" cellspacing="0" \
style="color:rgb(0,0,0);font-size:medium"><tbody><tr><td valign="top" align="left"><a \
href="https://www.webpower-group.com/" title="Webpower" target="_blank"><font \
face="verdana, sans-serif"><img \
src="https://www.webpower.nl/signature/images/webpower.jpg" width="202" \
title="Webpower" border="0" style="display: block; border: none; width: \
202px;"></font></a></td></tr><tr><td valign="top" align="left" height="10" \
style="line-height:10px"><font face="verdana, sans-serif"> </font></td></tr><tr><td \
valign="top" align="left"><table cellpadding="0" cellspacing="0"><tbody><tr><td \
valign="top" align="left" width="24"><a \
href="https://www.facebook.com/webpower.marketingautomation/" title="Facebook" \
target="_blank"><font face="verdana, sans-serif"><img \
src="https://www.webpower.nl/signature/images/wp_facebook.jpg" width="24" \
alt="Facebook" border="0" style="display: block; width: 24px;"></font></a></td><td \
valign="top" align="left" width="10"><font face="verdana, sans-serif"> \
</font></td><td valign="top" align="left" width="24"><a \
href="https://twitter.com/webpower" title="Twitter" target="_blank"><font \
face="verdana, sans-serif"><img \
src="https://www.webpower.nl/signature/images/wp_twitter.jpg" width="24" \
alt="Twitter" border="0" style="display: block; width: 24px;"></font></a></td><td \
valign="top" align="left" width="10"><font face="verdana, sans-serif"> \
</font></td><td valign="top" align="left" width="24"><a \
href="https://www.linkedin.com/company/36782/" title="Linkedin" target="_blank"><font \
face="verdana, sans-serif"><img \
src="https://www.webpower.nl/signature/images/wp_linkedin.jpg" width="24" \
alt="Linkedin" border="0" style="display: block; width: \
24px;"></font></a></td></tr></tbody></table></td></tr></tbody></table><table \
width="320" cellpadding="0" cellspacing="0" \
style="color:rgb(0,0,0);font-size:medium"><tbody><tr><td valign="top" align="left" \
width="5"></td><td valign="top" align="left" \
style="padding-top:5px;padding-bottom:5px;font-family:Verdana,sans-serif,"Lucida \
Grande";border-top:1px solid \
rgb(221,221,221);font-size:9px;line-height:13px;color:rgb(88,89,91)"><font \
face="verdana, sans-serif">Barcelona | Barneveld | Beijing | Chengdu | \
Guangzhou<br>Hamburg | Shanghai | Shenzhen | Stockholm</font></td><td valign="top" \
align="left" width="5"><font face="verdana, sans-serif"> \
</font></td></tr></tbody></table></td></tr></tbody></table><table width="320" \
cellpadding="0" cellspacing="0" style="color:rgb(0,0,0);font-family:"Times New \
Roman";font-size:medium"><tbody><tr><td valign="top" align="left" \
width="5"></td><td valign="top" align="left" \
style="padding-top:5px;padding-bottom:5px;font-family:Verdana,sans-serif,"Lucida \
Grande";border-top:1px solid \
rgb(221,221,221);font-size:9px;line-height:13px;color:rgb(88,89,91)"><a \
href="https://webpower.nl/event/kennissessies/?utm_source=GML&utm_medium=EMAIL&utm_campaign=EVENT&utm_term=KNOWLDGS&utm_content=NL" \
title="Webpower Food for Thought-kennissessies" style="font-family:"Times New \
Roman";font-size:medium" target="_blank"><img \
src="https://webpower.nl/signature/images/wp_footer_banner.jpg" border="0" \
style="display: block;"></a></td></tr></tbody></table></div></div></div></div></div></ \
div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr">Op di 30 okt. 2018 om 12:12 schreef Jean-Louis \
Dupond <<a href="mailto:jean-louis@dupond.be">jean-louis@dupond.be</a>>:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Roalt,</p>
<p>We are running Xen 4.6 on 4.9.x kernel and CentOS 6, and are
having the same issues.<br>
But not that frequent as you state. Only like once every month.</p>
<p>The systems (Dell R630) also crashes/resets without any message.
So nothing is logged unfortunately :(<br>
<br>
The crashes were not observed on Xen 4.4.<br>
<br>
We configured the servers to print kernel logs to SOL (Serial Over
LAN via iDRAC), and we log those.<br>
But since then no crashed servers anymore, so we don't know yet if
this will give us some more details.</p>
<p><br>
Thanks<br>
Jean-Louis<br>
</p>
<div class="m_564822250698883823moz-cite-prefix">On 29/10/18 12:57, Roalt \
Zijlstra | webpower wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Hi there,</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Ever since all the \
Meltdown and Spectre kernel updates and possibly also Xen 4.8 updates, we experience \
crashes of the Dom0 just out of the blue. Sometimes after 1 day, sometimes after a \
few days or even 14 days, completely random.</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">We have two Dell P730 \
servers and two Dell P720 servers with this behaviour. One thing is that we updated \
these machine to the latest available firmware, because that is the most secure
way. Then we installed Debian Stretch with Xen 4.8 support</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">We have done serveral \
installs and 4 servers seem to crash pretty
fast and other don't. In the end we think that we can lead it
back to the xen-4.8.4-pre version being stable and the
xen-4.8.5-pre being unstable. This was kinda independent of
the kernel that we were using 4.14 or 4.9.0-8-amd64. This is
off course all Debian package numbering.</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">As last resort we \
updated on one server all DomU kernels of our Jessie servers on this Dom0 to 4.9.0 \
from backports instead of the 3.16 kernel. For now that seems to work, but the \
crashes are random so it could happen any time again. The idea is that
these kernels are completely spectre& meltdown unaware and
might cause trouble in Xen kernel support. I am not sure if
this is true at all, but we are pretty lost what the actual
cause is.</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">We also tested with \
CentOS and we also had these crashes there with certain combinations of kernel/Xen. \
The most recent updates seem to be more stable tough. The most frustrating
part is the there is absolutely no logs to be found. No kernel
oops or what.. the server just resets and boots again.</div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Are there others \
experiencing problems like this? Do you see more frequent server/kernel crashes on \
production servers? <br> </div>
<div class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small"><br> </div>
<div>
<div dir="ltr" \
class="m_564822250698883823m_1274878354686728973gmail_signature" \
data-smartmail="gmail_signature"> <div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div \
style="color:rgb(102,102,102);font-family:arial,helvetica,sans-serif;font-size:12.8000001907349px">
<div dir="ltr">
<div dir="ltr" \
style="font-size:12.8px;color:rgb(34,34,34);font-family:arial,sans-serif">
<table width="100%" cellspacing="0" \
cellpadding="0"> <tbody>
<tr>
<td \
style="color:rgb(88,89,91);font-size:13px;line-height:16px" valign="top" \
align="left">
<table width="100%" cellspacing="0" \
cellpadding="0"> <tbody>
<tr>
<td \
style="color:rgb(88,89,91);font-size:13px;line-height:16px" valign="top" \
align="left"><font face="verdana, sans-serif">Best
regards,</font></td>
</tr>
<tr>
<td width="20" valign="top" \
align="left"><font face="verdana,
sans-serif"> </font></td>
</tr>
</tbody>
</table>
<span class="gmail_default" \
style="font-family:arial,helvetica,sans-serif;font-size:small">Roalt \
Zijlstra</span><br> <br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_564822250698883823mimeAttachmentHeader"></fieldset>
<pre class="m_564822250698883823moz-quote-pre">_______________________________________________
Xen-users mailing list
<a class="m_564822250698883823moz-txt-link-abbreviated" \
href="mailto:Xen-users@lists.xenproject.org" \
target="_blank">Xen-users@lists.xenproject.org</a> <a \
class="m_564822250698883823moz-txt-link-freetext" \
href="https://lists.xenproject.org/mailman/listinfo/xen-users" \
target="_blank">https://lists.xenproject.org/mailman/listinfo/xen-users</a></pre> \
</blockquote> </div>
</blockquote></div>
[Attachment #6 (text/plain)]
_______________________________________________
Xen-users mailing list
Xen-users@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic