[prev in list] [next in list] [prev in thread] [next in thread]
List: xen-users
Subject: Re: [Xen-users] Xen PV Guest won't boot latest kernel- OSError: [Errno 28] No space left on device
From: Michael Barkdoll <mabarkdoll () gmail ! com>
Date: 2018-01-12 17:50:11
Message-ID: CAC7iewECc3s3ZDD48AXpOxufsDWbyOCw424VWwQU8juuOmKmgQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thanks for the reply Andy. I tried as you suggested here is the output of
the command.
> I don't know what the problem is, but this suggests it lies within
> pygrub. Try running pygrub separately to see what happens, i.e.:
> # /usr/lib/xen-4.4/bin/pygrub /dev/host-disk2-vg/guest-disk
````
$ /usr/lib/xen-4.4/bin/pygrub /dev/vps03-disk2-vg/www2-disk
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg
WARNING:root:grub2's saved_entry/next_entry not supported
WARNING:root:Unknown directive else
WARNING:root:...
Traceback (most recent call last):
File "/usr/lib/xen-4.4/bin/pygrub", line 873, in <module>
output_directory, not_really)
File "/usr/lib/xen-4.4/bin/pygrub", line 718, in copy_from_image
dir=output_directory)
File "/usr/lib/python2.7/tempfile.py", line 308, in mkstemp
return _mkstemp_inner(dir, prefix, suffix, flags)
File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/var/run/xend/boot/boot_
kernel.jedvP5'
````
Directory /var/run/xend/boot/ don't exist in the host Xen Ubuntu14.04 of
the Guest Ubuntu 16.04. Guest VM still fails to reboot after a kernel
upgrade, unless I manually select the old kernel.
On Thu, Jan 11, 2018 at 11:53 AM, Michael Barkdoll <mabarkdoll@gmail.com>
wrote:
> Hello, I have an Ubuntu 14.04 running Xen and a VM running Ubuntu 16.04
> LTS that I'm having issues with it booting after upgrading the kernel for
> meltdown and spectre vulnerability.
>
> ````
> sudo apt-get update
> sudo apt-get upgrade
> sudo apt-get dist-upgrade
> ````
>
> Ubuntu 14.04 xen host reboot failure log:
> ````
> $ tailf /var/log/xen/servername.log
> Domain 24 needs to be cleaned up: destroying the domain
> Done. Rebooting now
> libxl: error: libxl_bootloader.c:628:bootloader_finished: bootloader
> failed - consult logfile /var/log/xen/bootloader.27.log
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader
> [-1] exited with error status 1
> libxl: error: libxl_create.c:1024:domcreate_rebuild_done: cannot
> (re-)build domain: -3
> ````
>
>
> ````
> cat /var/log/xen/bootloader.27.log
> Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /grub/grub.cfg
> pyGRUB version 0.6
> ââââââââââââââââââââââââââââ \
> âââââââââââââââââââââââââââââââ
> âââââââââââââââ
> â Ubuntu â
> â Ubuntu, with Linux 4.4.0-109-generic â
> â Ubuntu, with Linux 4.4.0-109-generic (recovery mode) â
> â Ubuntu, with Linux 4.4.0-108-generic â
> â Ubuntu, with Linux 4.4.0-108-generic (recovery mode) â
> â Ubuntu, with Linux 4.4.0-104-generic â
> â Ubuntu, with Linux 4.4.0-104-generic (recovery mode) â
> â Ubuntu, with Linux 4.4.0-15-generic â
> ââââââââââââââââââââââââââââ \
> âââââââââââââââââââââââââââââââ
> âââââââââââââââ
> Use the ^ and â´ keys to select which entry is highlighted.
> Press enter to boot the selected OS, 'e' to edit the
> commands before booting, 'a' to modify the kernel arguments
> before booting, or 'c' for a command line.
>
> Traceback (most recent call last):
> File "/usr/lib/xen-4.4/bin/pygrub", line 905, in <module>
> os.write(fd, ostring)
> OSError: [Errno 28] No space left on device
> ````
>
> I was able to get the vm to boot only by rapidly attempt to access the
> menu options and boot the old kernel (4.4.0-15-generic). Disk space and
> inodes seems fine on the host Ubuntu 14.04LTS and Ubuntu 16.04LTS Guest.
>
> Ubuntu 14.04 Host:
> ````
> root@host:/var/log/xen# df -hFilesystem Size Used Avail Use%
> Mounted on
> udev 197M 12K 197M 1% /dev
> tmpfs 42M 1.1M 41M 3% /run
> /dev/dm-0 104G 49G 50G 50% /
> none 4.0K 0 4.0K 0% /sys/fs/cgroup
> none 5.0M 0 5.0M 0% /run/lock
> none 208M 0 208M 0% /run/shm
> none 100M 0 100M 0% /run/user
> /dev/sda1 236M 40M 184M 18% /boot
> root@host:/var/log/xen# df -hi
> Filesystem Inodes IUsed IFree IUse% Mounted on
> udev 50K 721 49K 2% /dev
> tmpfs 52K 849 52K 2% /run
> /dev/dm-0 6.6M 86K 6.5M 2% /
> none 52K 2 52K 1% /sys/fs/cgroup
> none 52K 5 52K 1% /run/lock
> none 52K 1 52K 1% /run/shm
> none 52K 2 52K 1% /run/user
> /dev/sda1 61K 298 61K 1% /boot
> root@host:/var/log/xen#
> ````
>
> Ubuntu 16.04 PV Guest:
> ````
> root@guest:~# df -hFilesystem Size Used Avail
> Use% Mounted on
> udev 950M 0 950M 0% /dev
> tmpfs 200M 3.1M 197M 2% /run
> /dev/mapper/guest--vg-root 47G 4.3G 40G 10% /
> tmpfs 997M 0 997M 0% /dev/shm
> tmpfs 5.0M 0 5.0M 0% /run/lock
> tmpfs 997M 0 997M 0% /sys/fs/cgroup
> /dev/xvda1 472M 192M 256M 43% /boot
> tmpfs 200M 0 200M 0% /run/user/1000
> root@www2:~# df -hi
> Filesystem Inodes IUsed IFree IUse% Mounted on
> udev 238K 406 237K 1% /dev
> tmpfs 250K 491 249K 1% /run
> /dev/mapper/guest--vg-root 3.0M 177K 2.8M 6% /
> tmpfs 250K 1 250K 1% /dev/shm
> tmpfs 250K 7 250K 1% /run/lock
> tmpfs 250K 16 250K 1%
> /sys/fs/cgroup
> /dev/xvda1 122K 315 122K 1% /boot
> tmpfs 250K 4 250K 1%
> /run/user/1000
> root@guest:~#
> ````
>
> Here is the guest Ubuntu 16.04 OS kernels:
> ````
> root@guest:~# dpkg -l | grep linux-image-
> ii linux-image-4.4.0-104-generic 4.4.0-104.127
> amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
> ii linux-image-4.4.0-108-generic 4.4.0-108.131
> amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
> ii linux-image-4.4.0-109-generic 4.4.0-109.132
> amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
> ii linux-image-4.4.0-15-generic 4.4.0-15.31
> amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP
> ii linux-image-extra-4.4.0-104-generic 4.4.0-104.127
> amd64 Linux kernel extra modules for version 4.4.0 on 64
> bit x86 SMP
> ii linux-image-extra-4.4.0-108-generic 4.4.0-108.131
> amd64 Linux kernel extra modules for version 4.4.0 on 64
> bit x86 SMP
> ii linux-image-extra-4.4.0-109-generic 4.4.0-109.132
> amd64 Linux kernel extra modules for version 4.4.0 on 64
> bit x86 SMP
> ii linux-image-extra-4.4.0-15-generic 4.4.0-15.31
> amd64 Linux kernel extra modules for version 4.4.0 on 64
> bit x86 SMP
> ii linux-image-generic 4.4.0.109.114
> amd64 Generic Linux kernel image
> root@guest:~#
> ````
>
> I'm currently able to boot to 4.4.0-15.31
>
> Here is my xen config file that I use to start the vm:
> ````
> # cat /etc/xen/guest.cfg
> # /var/lib/xen/images/ubuntu-netboot/xenial16LTS
>
>
> name = "guest"
>
>
> #kernel = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/vmlinuz"
> #ramdisk = "/var/lib/xen/images/ubuntu-netboot/xenial16LTS/initrd.gz"
> bootloader = "/usr/lib/xen-4.4/bin/pygrub"
>
>
> memory = 2048
> vcpus = 1
>
>
>
>
> disk = [ '/dev/host-disk2-vg/guest-disk,raw,xvda,rw' ]
>
>
>
>
> #
> # Networking
> #
> vif = [ 'ip=xxx.xxx.xxx.xxx ,mac=AA:BB:CC:DD:EE:FF' ]
>
>
> #
> # Behaviour
> #
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
> ````
>
> I also tried launching the vm with strace output of
> ````
> strace xl create -c file.cfg
> ````
> is here: https://pastebin.com/8ctS3TQh
>
> I'm a bit lost as to the cause of this... I was hoping to just upgrade the
> kernel for meltdown and spetre, but this issue I've never faced before. I
> think it might be related to pygrub and the new kernel? I don't know for
> sure though anything I can check to figure this out better? Thanks.
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div>Thanks for the reply Andy. I tried as you suggested here is the \
output of the command. </div><div><br></div><div><pre style="border:1px solid \
rgb(221,221,221);background:rgb(250,250,250);color:rgb(51,51,51);font-size:12px">> \
I don't know what the problem is, but this suggests it lies within > pygrub. \
Try running pygrub separately to see what happens, i.e.:
> # /usr/lib/xen-4.4/bin/pygrub \
/dev/host-disk2-vg/guest-disk</pre></div><div><br></div><div>````</div><div>$ \
/usr/lib/xen-4.4/bin/pygrub /dev/vps03-disk2-vg/www2-disk</div><div>Using <class \
'grub.GrubConf.<wbr>Grub2ConfigFile'> to parse \
/grub/grub.cfg</div><div>WARNING:root:grub2's saved_entry/next_entry not \
supported</div><div>WARNING:root:Unknown directive \
else</div><div>WARNING:root:...</div><div><br></div><div>Traceback (most recent call \
last):</div><div> File "/usr/lib/xen-4.4/bin/pygrub", line 873, in \
<module></div><div> output_directory, not_really)</div><div> File \
"/usr/lib/xen-4.4/bin/pygrub", line 718, in copy_from_image</div><div> \
dir=output_directory)</div><div> File \
"/usr/lib/python2.7/tempfile.<wbr>py", line 308, in mkstemp</div><div> \
return _mkstemp_inner(dir, prefix, suffix, flags)</div><div> File \
"/usr/lib/python2.7/tempfile.<wbr>py", line 239, in \
_mkstemp_inner</div><div> fd = _os.open(file, flags, 0600)</div><div>OSError: \
[Errno 2] No such file or directory: \
'/var/run/xend/boot/boot_<wbr>kernel.jedvP5'</div><div>````<br></div><div><br></div><div>Directory \
/var/run/xend/boot/ don't exist in the host Xen Ubuntu14.04 of the Guest Ubuntu \
16.04. Guest VM still fails to reboot after a kernel upgrade, unless I manually \
select the old kernel.</div><div class="gmail_extra"><div><div \
class="m_3305324266061706932gmail_signature" data-smartmail="gmail_signature"><div \
dir="ltr"><br></div></div></div> <br><div class="gmail_quote">On Thu, Jan 11, 2018 at \
11:53 AM, Michael Barkdoll <span dir="ltr"><<a href="mailto:mabarkdoll@gmail.com" \
target="_blank">mabarkdoll@gmail.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><div>Hello, I have an Ubuntu 14.04 running Xen \
and a VM running Ubuntu 16.04 LTS that I'm having issues with it booting after \
upgrading the kernel for meltdown and spectre \
vulnerability.</div><div><br></div><div>````</div><div>sudo apt-get \
update</div><div>sudo apt-get upgrade</div><div>sudo apt-get \
dist-upgrade</div><div>````</div><div><br></div><div>Ubuntu 14.04 xen host reboot \
failure log:</div><div>````</div><div>$ tailf \
/var/log/xen/servername.log</div><div>Domain 24 needs to be cleaned up: destroying \
the domain</div><div>Done. Rebooting now</div><div>libxl: error: \
libxl_bootloader.c:628:bootloa<wbr>der_finished: bootloader failed - consult logfile \
/var/log/xen/bootloader.27.log</div><div>libxl: error: \
libxl_exec.c:118:libxl_report_<wbr>child_exitstatus: bootloader [-1] exited with \
error status 1</div><div>libxl: error: \
libxl_create.c:1024:domcreate_<wbr>rebuild_done: cannot (re-)build domain: \
-3</div><div>````</div><div><br></div><div><br></div><div>````</div><div>cat \
/var/log/xen/bootloader.27.log</div><div>Using <class \
'grub.GrubConf.Grub2ConfigFile<wbr>'> to parse /grub/grub.cfg pyGRUB \
version 0.6</div><div> \
┌────────────────────────────<wbr>──────────────────────────────<wbr>──────────────┐</div><div> \
│ Ubuntu │</div><div> \
│ Ubuntu, with Linux 4.4.0-109-generic │</div><div> \
│ Ubuntu, with Linux 4.4.0-109-generic (recovery mode) │</div><div> \
│ Ubuntu, with Linux 4.4.0-108-generic │</div><div> \
│ Ubuntu, with Linux 4.4.0-108-generic (recovery mode) │</div><div> \
│ Ubuntu, with Linux 4.4.0-104-generic │</div><div> \
│ Ubuntu, with Linux 4.4.0-104-generic (recovery mode) │</div><div> \
│ Ubuntu, with Linux 4.4.0-15-generic │</div><div> \
└────────────────────────────<wbr>──────────────────────────────<wbr>──────────────┘</div><div>Use \
the ^ and ┴ keys to select which entry is highlighted.</div><div>Press enter to boot \
the selected OS, 'e' to edit the</div><div>commands before booting, \
'a' to modify the kernel arguments</div><div>before booting, or 'c' \
for a command line.</div><div><br></div><div>Traceback (most recent call \
last):</div><div> File "/usr/lib/xen-4.4/bin/pygrub", line 905, in \
<module></div><div> os.write(fd, ostring)</div><div>OSError: [Errno 28] No \
space left on device</div><div>````</div><div><br></div><div>I was able to get the vm \
to boot only by rapidly attempt to access the menu options and boot the old kernel \
(4.4.0-15-generic). Disk space and inodes seems fine on the host Ubuntu 14.04LTS and \
Ubuntu 16.04LTS Guest.</div><div><br></div><div>Ubuntu 14.04 \
Host:</div><div>````</div><div>root@host:/var/log/xen# df -hFilesystem Size \
Used Avail Use% Mounted on</div><div>udev 197M 12K 197M 1% \
/dev</div><div>tmpfs 42M 1.1M 41M 3% /run</div><div>/dev/dm-0 \
104G 49G 50G 50% /</div><div>none 4.0K 0 4.0K 0% \
/sys/fs/cgroup</div><div>none 5.0M 0 5.0M 0% \
/run/lock</div><div>none 208M 0 208M 0% /run/shm</div><div>none \
100M 0 100M 0% /run/user</div><div>/dev/sda1 236M 40M 184M 18% \
/boot</div><div>root@host:/var/log/xen# df -hi</div><div>Filesystem Inodes IUsed \
IFree IUse% Mounted on</div><div>udev 50K 721 49K 2% \
/dev</div><div>tmpfs 52K 849 52K 2% /run</div><div>/dev/dm-0 \
6.6M 86K 6.5M 2% /</div><div>none 52K 2 52K 1% \
/sys/fs/cgroup</div><div>none 52K 5 52K 1% \
/run/lock</div><div>none 52K 1 52K 1% /run/shm</div><div>none \
52K 2 52K 1% /run/user</div><div>/dev/sda1 61K 298 61K 1% \
/boot</div><div>root@host:/var/log/xen#</div><div>````</div><div><br></div><div>Ubuntu \
16.04 PV Guest:</div><div>````</div><div>root@guest:~# df -hFilesystem \
Size Used Avail Use% Mounted on</div><div>udev 950M \
0 950M 0% /dev</div><div>tmpfs 200M 3.1M 197M \
2% /run</div><div>/dev/mapper/guest--vg-root 47G 4.3G 40G 10% \
/</div><div>tmpfs 997M 0 997M 0% \
/dev/shm</div><div>tmpfs 5.0M 0 5.0M 0% \
/run/lock</div><div>tmpfs 997M 0 997M 0% \
/sys/fs/cgroup</div><div>/dev/xvda1 472M 192M 256M 43% \
/boot</div><div>tmpfs 200M 0 200M 0% \
/run/user/1000</div><div>root@www2:~# df -hi</div><div>Filesystem \
Inodes IUsed IFree IUse% Mounted on</div><div>udev \
238K 406 237K 1% /dev</div><div>tmpfs 250K \
491 249K 1% /run</div><div>/dev/mapper/guest--vg-root 3.0M 177K \
2.8M 6% /</div><div>tmpfs 250K 1 250K 1% \
/dev/shm</div><div>tmpfs 250K 7 250K 1% \
/run/lock</div><div>tmpfs 250K 16 250K 1% \
/sys/fs/cgroup</div><div>/dev/xvda1 122K 315 122K \
1% /boot</div><div>tmpfs 250K 4 250K 1% \
/run/user/1000</div><div>root@guest:~#</div><div>````</div><div><br></div><div>Here \
is the guest Ubuntu 16.04 OS kernels:</div><div>````</div><div>root@guest:~# dpkg -l \
| grep linux-image-</div><div>ii linux-image-4.4.0-104-generic 4.4.0-104.127 \
amd64 Linux kernel image for version 4.4.0 on 64 bit x86 SMP</div><div>ii \
linux-image-4.4.0-108-generic 4.4.0-108.131 amd64 \
Linux kernel image for version 4.4.0 on 64 bit x86 SMP</div><div>ii \
linux-image-4.4.0-109-generic 4.4.0-109.132 amd64 \
Linux kernel image for version 4.4.0 on 64 bit x86 SMP</div><div>ii \
linux-image-4.4.0-15-generic 4.4.0-15.31 amd64 \
Linux kernel image for version 4.4.0 on 64 bit x86 SMP</div><div>ii \
linux-image-extra-4.4.0-104-ge<wbr>neric 4.4.0-104.127 \
amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 \
SMP</div><div>ii linux-image-extra-4.4.0-108-ge<wbr>neric 4.4.0-108.131 \
amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 \
SMP</div><div>ii linux-image-extra-4.4.0-109-ge<wbr>neric 4.4.0-109.132 \
amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 \
SMP</div><div>ii linux-image-extra-4.4.0-15-gen<wbr>eric 4.4.0-15.31 \
amd64 Linux kernel extra modules for version 4.4.0 on 64 bit x86 \
SMP</div><div>ii linux-image-generic 4.4.0.109.114 \
amd64 Generic Linux kernel \
image</div><div>root@guest:~#</div><div>````</div><div><br></div><div>I'm \
currently able to boot to 4.4.0-15.31</div><div><br></div><div>Here is my xen config \
file that I use to start the vm:</div><div>````</div><div># cat \
/etc/xen/guest.cfg</div><div># \
/var/lib/xen/images/ubuntu-net<wbr>boot/xenial16LTS</div><div><br></div><div><br></div><div>name \
= "guest"</div><div><br></div><div><br></div><div>#kernel = \
"/var/lib/xen/images/ubuntu-ne<wbr>tboot/xenial16LTS/vmlinuz"</div><div>#ramdisk \
= "/var/lib/xen/images/ubuntu-ne<wbr>tboot/xenial16LTS/initrd.gz"</div><div>bootloader \
= "/usr/lib/xen-4.4/bin/pygrub"</div><div><br></div><div><br></div><div>memory \
= 2048</div><div>vcpus = \
1</div><div><br></div><div><br></div><div><br></div><div><br></div><div>disk = [ \
'/dev/host-disk2-vg/guest-disk<wbr>,raw,xvda,rw' \
]</div><div><br></div><div><br></div><div><br></div><div><br></div><div>#</div><div># \
Networking</div><div>#</div><div>vif = [ 'ip=xxx.xxx.xxx.xxx \
,mac=AA:BB:CC:DD:EE:FF' ]</div><div><br></div><div><br></div><div>#</div><div># \
Behaviour</div><div>#</div><div>on_poweroff = 'destroy'</div><div>on_reboot \
= 'restart'</div><div>on_crash = \
'restart'</div><div>````</div><div><br></div><div>I also tried launching the \
vm with strace output of</div><div>````</div><div>strace xl create -c file.cfg \
</div><div>````</div><div>is here: <a href="https://pastebin.com/8ctS3TQh" \
target="_blank">https://pastebin.com/8ctS3TQh</a></div><div><br></div><div>I'm a \
bit lost as to the cause of this... I was hoping to just upgrade the kernel for \
meltdown and spetre, but this issue I've never faced before. I think it might be \
related to pygrub and the new kernel? I don't know for sure though anything I can \
check to figure this out better? Thanks.</div><div><br></div> </div>
</blockquote></div><br></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
Xen-users mailing list
Xen-users@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-users
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic