[prev in list] [next in list] [prev in thread] [next in thread] 

List:       qubes-devel
Subject:    Re: [qubes-devel] PCI passthrough appears to be regressing
From:       Joanna Rutkowska <joanna () invisiblethingslab ! com>
Date:       2016-02-01 8:57:59
Message-ID: 20160201085758.GA855 () work-mutt
[Download RAW message or body]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jan 31, 2016 at 02:07:01PM -0800, Eric Shelton wrote:
> On Sunday, January 31, 2016 at 11:27:40 AM UTC-5, joanna wrote:
> > 
> > -----BEGIN PGP SIGNED MESSAGE----- 
> > Hash: SHA256 
> > 
> > On Sat, Jan 30, 2016 at 08:20:45AM -0800, Eric Shelton wrote: 
> > > I'm not sure that I have anything concrete enough to open an issue yet 
> > > (aside from https://github.com/QubesOS/qubes-issues/issues/1659), but I 
> > > think it is worth initiating a discussion about PCI passthrough support 
> > > having regressed from Qubes 2.0 (where pretty much everything seemed to 
> > > work, including passthrough to HVM domains), and that it appears to only 
> > be 
> > > getting worse over time.  For example, see these recent discussions: 
> > > 
> > > https://groups.google.com/forum/#!topic/qubes-users/VlbfFyNGNTs 
> > > https://groups.google.com/d/msg/qubes-devel/YqdhlmYtMY0/vCO3QHLBBgAJ 
> > > https://groups.google.com/d/msg/qubes-users/cmPRMOkxkdA/gIV68O0-CQAJ 
> > > 
> > > The state of things seems to be something along these lines: 
> > > - everything worked pretty well under Qubes 2.0 
> > > - starting with Qubes 3.0, PCI passthrough to HVM domains broke (I think 
> > it 
> > > is a libvirt related issue, as passthrough would still work using 'xl' 
> > to 
> > > start a domain) 
> > > - it seems people are having less success doing passthrough under Xen 
> > 4.6 
> > > than under Xen 4.4.  However, there is no hard evidence on this. 
> > > - Linux kernel 4.4 appears to break PCI passthrough for network devices 
> > > entirely (not so much what is going on with Qubes today, but evidence 
> > that 
> > > Xen's PCI passthrough support is getting worse, not better).  Although 
> > the 
> > > Qubes team has opted to move forward despite the above issues, this 
> > > represents the point at which Qubes will no longer be able to do the one 
> > > passthrough thing it relies on - isolating network adapters. 
> > > 
> > > The effects of this today are that PCI passthrough questions are popping 
> > up 
> > > more frequently on qubes-users, and the fixes that used to reliably 
> > address 
> > > passthrough problems (for example, setting passthrough to permissive) 
> > seem 
> > > to be less helpful, as problems seem to be lurking deeper within Xen, or 
> > > perhaps changes to Xen mean that other fixes should be used instead. 
> > > 
> > > The effects of this in the future is that it grows less and less likely 
> > > that the vision of doing a separate GUI domain via GPU passthrough can 
> > be 
> > > successfully executed.  It is hard enough to get a GPU passed through to 
> > > begin with (thanks to weird abuses of PCI features by GPU makers to 
> > > facilitate DRM, I'm uncertain it will ever work for anything other than 
> > > Intel GPUs, which is only due to specific efforts on Intel's part to 
> > make 
> > > it work in Xen).  The above issues make it much worse. 
> > > 
> > 
> > The longer-term plan is for us to move to HVMs for everything: AppVMs, 
> > ServiceVMs, and, of course, for the GUI domain also. This should not only 
> > resolve the passthrough problems (assuming IOMMU works well), but also 
> > reduce 
> > complexity in the hypervisor required to handle such VMs (see e.g. XSA 
> > 148). We 
> > plan on starting this work once an early 4.0 is out. 
> > 
> 
> I don't see how this resolves things, but it's a little hard to at this 
> time since HVM PCI passthrough support is currently broken.  There is no 
> way for me to compare the bugginess of PCI passthrough on HVM versus PV.  I 
> have not experimented with Linux HVMs using PCI passthrough - do 
> pcifront/pciback drop out of the picture, and it just looks and behaves 
> like normal (I would expect the hypervisor to still impose some 
> restrictions on use/abuse of PCI devices)?  Is there some other significant 
> difference?
> 
> 
> > Are the DRM-related problems with passthrough for some GPUs you mentioned 
> > above 
> > also encountered on HVMs, or are they limited to PVs only? 
> > 
> 
> All of the efforts I have seen involving GPU passthrough have involved HVM, 
> maybe since most people pursuing it are seeking to run 3D Windows 
> applications (games or otherwise). 
> http://wiki.xen.org/wiki/XenVGAPassthrough#Why_can.27t_I_do_VGA_passthrough_to_Xen_PV_.28paravirtual.29_guest.3F \
>  discusses why passthrough cannot be done to a PV domain.
> 
> The bizarre things GPU makers have done with PCI registers and bars 
> generally defies how one expects well-mannered PCI devices to behave.  MS 
> Vista not only introduced, but required this nonsense 
> (http://www.cypherpunks.to/~peter/vista.pdf).  As noted on page 22 of that 
> PDF file, each GPU type stepping is/was required to have a different 
> mechanism in place.  As a result, no two cards, even by the same vendor, 
> have distorted PCI behavior in the quite same way.  The patch 
> at http://old-list-archives.xenproject.org/archives/html/xen-devel/2010-10/txtfLpL6CdMGC.txt \
>  describes one example:
> 
> * ATI VBIOS Working Mechanism 
> *
> * Generally there are three memory resources (two MMIO and one PIO) 
> * associated with modern ATI gfx. VBIOS uses special tricks to figure out 
> * BARs, instead of using regular PCI config space read.
> *
> *  (1) VBIOS relies on I/O port 0x3C3 to retrieve PIO BAR 
> *  (2) VBIOS maintains a shadow copy of PCI configure space. It retries the 
> *      MMIO BARs from this shadow copy via sending I/O requests to first 
> two 
> *      registers of PIO (MMINDEX and MMDATA). The workflow is like this: 
> *      MMINDEX (register 0) is written with an index value, specifying the 
> *      register VBIOS wanting to access. Then the shadowed data can be 
> *      read/written from MMDATA (register 1). For two MMIO BARs, the index 
> *      values are 0x4010 and 0x4014 respectively. 
> 
> Not how your typical PCI device behaves.
> 
> Often 1:1 mapping of bars has been required to get video drivers to work 
> with passthrough GPUs.  Sometimes needing to deal with booting the card's 
> BIOS through QEMU's emulated BIOS becomes an issue (with the solution being 
> copying the VBIOS using some hacked together mechanism, and rolling it into 
> the SEABIOS image).
> 
> Between NVIDIA and AMD, generally users have had a worse time getting 
> NVIDIA devices working via passthrough (although AMD has been plenty 
> difficult).  The most reliable technique, in my experience, is to use an 
> NVIDIA QUADRO that is compatible with NVIDIA GRID 
> (http://www.nvidia.com/object/dedicated-gpus.html).  It's the only thing 
> I've used for GPU passthrough that "just works" - the NVIDIA drivers are 
> specifically written to accept the GPU is running in a virtualized 
> environment.
> 
> On top of all of this, apparently NVIDIA is _actively_ doing things to 
> frustrate attempts at running their hardware within a 
> VM: https://www.reddit.com/r/linux/comments/2twq7q/nvidia_apparently_pulls_some_shady_shit_to_do/ \
>  with one developer describing the situation as "VM developers have been 
> having a little arms race with NVIDIA."
> 
> About 1-2 years ago, KVM + QEMU put in a lot of effort into getting GPU 
> passthrough working more smoothly.  However, I do not know what the state 
> of things is today, and Xen has not engaged in similar efforts to anywhere 
> near the same degree.  Looking the last 6 months or so of xen-devel, most 
> of subject lines mentioning PCI passthrough are efforts towards getting 
> Intel's IGD working via passthrough (related to this, Intel is also still 
> actively developing XenGT) .  Given the number of posts on the topic, I'm 
> guessing it was nontrivial - and this is with an actively cooperating GPU 
> vendor.
> 
> As noted above, most attempts have involved running GPUs in a virtualized 
> Windows session.  It is be possible that things will go more smoothly when 
> running Linux drivers.  On the other hand, I have not heard particularly 
> good things about running with nouveau, and using NVIDIA's own Linux 
> drivers may pull in the same headaches seen on Windows.
> 
> So, expect getting GPU passthrough working across the wide range of 
> notebook and desktop GPUs deployed out there to be a substantial challenge, 
> and make sure you have lots of different GPUs on hand for testing purposes. 
> You may discover, after not digging too deeply into it, that it is not 
> worth the headaches and risk of rendering a lot of hardware 
> Qubes-incompatible.  I understand (at least some of) the reasons for 
> wanting to move the GUI out of dom0, but it may not be practical due to 
> what the GPU vendors have done to support DRM.
> 

We would like to primarily target laptops, and this means we're primely
interested in getting the Intel integrated GPUs to work. Fortunately, Intel
seems to be putting lots of work into making GPUs not only VT-d-assignable, but
also virtualizable (as in: multiplexible) -- as evidenced by their GVT work
(formerly XenGT):

https://01.org/igvt-g

Thanks,
joanna.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBCAAGBQJWrx4VAAoJEDOT2L8N3GcYZOsP/RLTfDl2ws3HUfhfPGv3axCD
WlcaOn2taH5T5w5pVWXtK7uk84Cvc4ZA4DfZUJMVTyCzdGE5jel64VMgV/IasVx6
Lt05jYFyoi2T64UpqvFaoKDXPesVMhm22x/mGMuZIwsQe8hd5gLItWEbkKouvXJK
QQVFzO+1kxFENgOql6T/v9JNHr1G5ue/8cX3izhuoHa0IQykJaoKuTXXwWbkMgk0
aTVd6+Bp++WL4CUeyujQhNTmwIEl93QMiHMbhtnPpxcW30qHdrqj4IKn6AfL2hLf
zG0h7DLnrn7B3KJrIEfT0MMHAymPvbLpEQosxDrCdbkkgCt6x3ihlmXKOzfTB5Bz
tiieP5gHN9iUo/ToZVa4tRLNVE7VLSUdYxvBA24bN/KHm5ad43SfgcNdNTML3Lrn
gj6OJ44tHyJo7PgjIH80bVF3f0mM2jS3HDriO0at+lWaWAY+00dN8XMb69EahrKb
Q5vyG+ovjq8sd4oSLEPmIpdpTwq+2RRlENRbjb4VvQ9jqG18sagxs9wmGuXq1iO0
KIWZ/UporVbWm9nP6rbchlzPnIwCCe4gHOfNzECRnYZjtmeMkSQSeTvmg2X+qx+w
jXyFE97CpcRqpQ/qQL+IRm1CGT97siHx8p4NaJLbMKmG2uUiE9l0Cif0EdcVQx1c
ii5TV5hIDJYQRMkqLRRk
=1Llj
-----END PGP SIGNATURE-----

-- 
You received this message because you are subscribed to the Google Groups \
"qubes-devel" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to qubes-devel+unsubscribe@googlegroups.com. To post to this group, \
send email to qubes-devel@googlegroups.com. To view this discussion on the web visit \
https://groups.google.com/d/msgid/qubes-devel/20160201085758.GA855%40work-mutt. For \
more options, visit https://groups.google.com/d/optout.


[prev in list] [next in list] [prev in thread] [next in thread]