[prev in list] [next in list] [prev in thread] [next in thread]
List: qubes-devel
Subject: [qubes-devel] Re: [GSoC] Qubes-MIME-Handlers Weekly Progress Report #8
From: Andrew Morgan <andrew () amorgan ! xyz>
Date: 2017-08-09 20:14:02
Message-ID: omfqe5$inc$1 () blaine ! gmane ! org
[Download RAW message or body]
[Attachment #2 (multipart/mixed)]
On 08/09/2017 12:02 PM, Andrew Morgan wrote:
> On 08/09/2017 08:03 AM, Marek Marczykowski-Górecki wrote:
> > On Wed, Aug 09, 2017 at 07:28:21AM -0700, Andrew Morgan wrote:
> > > On 08/08/2017 03:38 AM, Andrew Morgan wrote:
> > > > Another report with some long-awaited news, enjoy!
> > > >
> > > > Blog post with screenshots:
> > > > https://blog.amorgan.xyz/gsoc-weekly-progress-report-8.html
> > > >
> > > > Otherwise text-only is below (with screenshots attached):
> > > >
> > > > ---
> > > >
> > > > Hey there, welcome to another report! Since the last one, you may
> > > > remember that I had finished setting up the build environment for
> > > > Nautilus inside of GNOME Builder and was just getting started on
> > > > properly doing the patch for supporting file trust attributes. Well, I'm
> > > > happy to report that the patch is now finished! Screenshots and details
> > > > below.
> > > >
> > > > ## Working Demo
> > > >
> > > > As screen recording in dom0 still isn't really doable yet, you'll have
> > > > to settle for some screenshots of it in action :)
> > > >
> > > > 
> > > > *Double-clicking a trusted file opens normally...*
> > > >
> > > > 
> > > > *Everyone knows gedit is the ultimate text editor.*
> > > >
> > > > 
> > > > *Whereas an untrusted file summons a DispVM!*
> > > >
> > > > 
> > > > *Virus free, yes-sir-ee!*
> > > >
> > > > This short demo makes use of a majority of all the different components
> > > > that have been worked on throughout the project. The now-patched
> > > > Nautilus makes a call to the patched NautilusPython which asks our
> > > > `qvm_trust` python extension to open a file, which it gives a response
> > > > based on what our cli tool, `qvm-file-trust`, says about it, and if it
> > > > is determined an untrusted file, `open-file-trust-based` will open it
> > > > securely in a DisposableVM. The contents of the file were never parsed
> > > > on the local machine!
> > > >
> > > > The low-level part of this process was already working, but it took the
> > > > patch to Nautilus allowing us to set off the chain from the GUI to
> > > > really bring it all together. Let's talk about how that was done and how
> > > > it works.
> > > >
> > > > ## Nautilus and NautilusPython
> > > >
> > > > By default, Nautilus does not support extensions written in Python, but
> > > > rather only those written in C. To get around this, a _Nautilus C
> > > > extension called NautilusPython_ was created. This extension has the
> > > > ability to provide a bridging interface from Nautilus' extension
> > > > infastructure to Python, handling any calls Python programs make to
> > > > Nautilus and calling them from C. Arguments and return variables are all
> > > > passed along seamlessly.
> > > >
> > > > This is a really nice feature, as writing extensions in Python are about
> > > > 90% less work than writing them in C, due to the large amount of
> > > > boiler-plate associated with setting up and tearing down an extension.
> > > > If you recall, the point of patching Nautilus was to allow extensions to
> > > > be able to be notified of when a file is being opened, and block that
> > > > request if necessary. I didn't want to subject myself or anyone else to
> > > > only being able to write an extension that made use of this new
> > > > functionality in C, thus patching NautilusPython as well as Nautilus
> > > > itself to add our new `file_open` function was necessary. So how was
> > > > each one done then?
> > > >
> > > > Patching NautilusPython to add a new method was relatively simple, you
> > > > only need look at the existing methods that an extension can subclass,
> > > > find one that is functionally similar your own (I used
> > > > `update_file_info` as my guide), then insert your own method
> > > > (`file_open` in this case) at all the relevant points as well as create
> > > > the appropriate function bodies and set argument/return variables
> > > > appropriately.
> > > >
> > > > So overall NautilusPython wasn't too bad really. The real time sink was
> > > > Nautilus itself.
> > > >
> > > > ## Patching Nautilus
> > > >
> > > > First of all, I almost immediately ditched GNOME Builder after my last
> > > > blog post. It's not that it was a bad IDE, far from it actually, I quite
> > > > liked the interface and the various tools it contained. The ability to
> > > > download a project from git, package and run it inside a Flatpak, and
> > > > then send patches upstream all from within the editor was quite
> > > > appealing! The first major showstopper however, was that the version of
> > > > Nautilus that Qubes makes use of (v3.22.3), doesn't support Flatpak.
> > > > While building and running from GNOME Builder without Flatpak seemed
> > > > possible, no matter what I tried I could not get extensions to function
> > > > properly with the GNOME Builder version.
> > > >
> > > > After a lot of fiddling about, I settled on what I probably should have
> > > > been using since the very beginning:
> > > > [qubes-builder](https://github.com/qubesos/qubes-builder)!
> > > >
> > > > Not only does it build the correct version (and supports extensions),
> > > > but I knew that whatever I did with this build system would be exactly
> > > > what users would be using when they built Qubes' templates or got them
> > > > from the build servers. So, unfortunately no Flatpak'd builds, but I do
> > > > have pre-packaged RPMs with debug-symbols! That's probably even better,
> > > > right?
> > > >
> > > > I'll post them in a separate blog post with instructions on how to set
> > > > everything up soon.
> > > >
> > > > qubes-builder worked very well and was quite easy to drill into and mess
> > > > about with, as it is mostly just a collection of various scripts. I
> > > > eventually built a script that ran in dom0 that would build Nautilus in
> > > > my development VM, transfer it to my testing VM, delete the old version
> > > > and install the new version.
> > > >
> > > > The only downside to this process was that building Nautilus took quite
> > > > a while each time (2-3m) which made quick changes quite an agonizing
> > > > process. I believe this was due to both having to rebuild _every_ file
> > > > each time (as they were packaged in a .tar.xz before being built,
> > > > meaning all files were considered to be modified) as well as a call to
> > > > `dnf` to fetch any updates was made, _every time_. I found out the
> > > > latter when working without internet and realizing my could would not
> > > > build! Each of these things probably could have been worked out or
> > > > changed if I had felt the need to, but I just stuck with the loading
> > > > times and usually had internet, so it was not too terrible.
> > > >
> > > > Was _was_ terrible and time-consuming were a couple of very strange
> > > > issues I had during the creation of the patch.
> > > >
> > > > The first was the dreaded _provider issue_. You see, every time Nautilus
> > > > calls a method that is meant to be subclassed by a Nautilus extension,
> > > > you must _provide a provider_. This can be an `INFO` provider, `MENU`
> > > > provider, and so on. Each method only accepts one of each type,
> > > > typically corresponding to the category of method you were calling. For
> > > > the `file_open` method I decided to stick it under the `INFO` provider
> > > > category, as that had to do with information about files, which I felt
> > > > 'when they were opened' best fell into.
> > > >
> > > > Well, these providers don't just pop out of nowhere. You have to create
> > > > them! You can do that by calling the intuitively-named
> > > > `nautilus_module_get_extensions_for_type` method, providing it the
> > > > `NAUTILUS_TYPE_INFO_PROVIDER` enum. It is then _supposed_ to return a
> > > > provider with a type that matches the one you gave it, but no matter
> > > > what I did it would always just return a `NULL` pointer! I spent _days_
> > > > on this problem, trying loads of different things. Every other call of
> > > > `nautilus_module_get_extensions_for_type` had no special setup or
> > > > initialization, leading me confused why my own calling of it brought
> > > > such different results.
> > > >
> > > > After picking through many lines of code, talking with GNOME people over
> > > > IRC who were currently GUADEC'ing, and trying to understand the whole
> > > > GNOME development ecosystem, I eventually traced it all back to
> > > > NautilusPython not finding an extension that made use of a subclassed
> > > > method from the `INFO` method type. But wait, we _do_ have an extension
> > > > that does that with `qvm_trust`, so why doesn't it work??
> > > >
> > > > Well I hit my head after this one folks. The version of NautilusPython
> > > > that Nautilus was loading in my test VM was the original, from-repo
> > > > version, not our modified version. **D'oh!**
> > > >
> > > > "Well that's just a simple fix, I just have to build it and import it
> > > > into Nautilus in the testing VM!", I assumed, before realizing that
> > > > while my modified NautilusPython built, Nautilus _would not accept it_.
> > > > Nautilus would just _hang_ with little to no output.
> > > >
> > > > After a few emails, Marek pointed me in the correct direction with the
> > > > `nautilus-3.0` branch of NautilusPython, which, while 6 years since the
> > > > last commit, worked with the latest version. Alright no problem. Rebase
> > > > my commits on that branch, rebuild, transfer the newly-built
> > > > shared-library over and bam, Nautilus runs again! And we have a provider
> > > > that's not NULL, hooray!
> > > >
> > > > And then, ladies and gentlemen, problem #2 reared _its_ ugly head.
> > > >
> > > > Here's the code to get a list (`GList*`) of providers to use with a
> > > > Nautilus extension:
> > > >
> > > > providers = nautilus_module_get_extensions_for_type
> > > > (NAUTILUS_TYPE_INFO_PROVIDER);
> > > >
> > > > While we were getting a _non-null_ provider value, the value we received
> > > > was _different_ from the one returned by the method we called. "Wat?"
> > > > you say, and I too said the same. For some reason we were returning a
> > > > 64-bit `GList` pointer from `nautilus_module_get_extensions_for_type`,
> > > > but storing a 32-bit `GList` pointer in `providers`.
> > > >
> > > > I tried all sorts of weird tricks to get this fixed. Nobody I talked to
> > > > seemed to have a solution, and after wrestling with it for a few days I
> > > > begrudingly settled on this very elegant workaround:
> > > >
> > > > // TODO: Providers ends up as 0x55f65ac0
> > > > // when method is returning 0x555555f65ac0, very odd...
> > > > providers = (GList*) (nautilus_module_get_extensions_for_type
> > > > (NAUTILUS_TYPE_INFO_PROVIDER) + 0x555500000000);
> > > >
> > > > Yes I know this is terrible. It did work though, and I needed to stop
> > > > hitting this strange bug and get on with patching, so that's where it
> > > > currently sits. I'm still actively looking for a proper solution. If
> > > > anyone knows why a 64bit pointer might get truncated to a 32bit pointer
> > > > after a function call in C please do let me know! You can find the full
> > > > code
> > > > [here](https://github.com/anoadragon453/nautilus/blob/master/src/nautilus-mime-actions.c#L2496)
> > > > and the `type` method
> > > > [here](https://github.com/anoadragon453/nautilus/blob/master/src/nautilus-module.c#L269).
> > > >
> > > > After this was out of the way, and a handful of more methods were
> > > > implemented, we finally had a working solution between the `qvm_trust`
> > > > extension and our patched Nautilus! It was beautiful, really.
> > > >
> > > > ## Conclusion and going forward
> > > >
> > > > My [email](https://amorgan.xyz/upstream-request.html) to the [Nautilus
> > > > mailing list](https://mail.gnome.org/archives/nautilus-list/) is still
> > > > awaiting moderation. GUADEC has now concluded, and with no update on the
> > > > situation it seems I need to do some more active poking of people to get
> > > > moderation approval. We'll see where that lands, the code still needs to
> > > > be cleaned up a bit before sending it upstream anyways, so I'm not too
> > > > worried.
> > > >
> > > > At this point in the project, we essentially have the following tasks left:
> > > >
> > > > * Patch KDE Dolphin to provide the same functionality as Nautilus now does
> > > > * Finish up some small tasks with the daemon like logging and some more
> > > > interfacing with the cli tool
> > > > * Code cleanup, linting, tests
> > > > * Integrate into main QubesOS repos
> > > >
> > > > I'm going to prioritize the latter 3 bullet points at this point to
> > > > ensure we get a really solid core implementation down of the project. I
> > > > don't want to spend another couple weeks patching Dolphin only to run
> > > > out of time for a general cleanup and other polish. I will try to pour
> > > > through the Dolphin code and note down everything that one would need to
> > > > do with Dolphin to make it work with file trust attributes, so that in
> > > > the future either myself or someone else can give it a shot.
> > > >
> > > > It may end up that Dolphin already has most of the functionality and the
> > > > patch will be very small, we'll see!
> > > >
> > > > Anyways, for now I'm going to start on finishing the remaining small
> > > > tasks, and will post any questions regarding packaging and/or cleanup to
> > > > the qubes-devel mailing list.
> > > >
> > > > As always, you can find the code
> > > > [here](https://github.com/anoadragon453/qubes-mime-types).
> > > >
> > > > And as of now, you can also find the patched code for
> > > > [Nautilus](https://github.com/anoadragon453/nautilus) and
> > > > [NautilusPython](https://github.com/anoadragon453/nautilus-python/tree/nautilus-3.0)
> > > > on Github as well. Make sure that when you're inspecting the
> > > > NautilusPython code, you're looking at the `nautilus-3.0` branch. That's
> > > > where all the important commits are!
> > > >
> > > > That's all for now, thanks for reading!
> > > >
> >
> > > Marek,
> >
> > > The git commits for nautilus/python seem like a bit of a mess at the
> > > moment. Are you currently reviewing them individually or can I squash
> > > them into just a few and force push to the repo?
> >
> > Feel free to squash them.
> >
> >
>
> Hey Marek,
>
> Nautilus has been squashed into one, easily-reviewable commit:
> https://github.com/anoadragon453/nautilus/commit/1325725451220447f53e5ab3016423e6d3372682
>
> Currently doing the same for nautilus-python.
>
> Thanks,
> Andrew Morgan
>
NautilusPython commit has been cleaned up to the best of my abilities
now too:
https://github.com/anoadragon453/nautilus-python/commit/6a9c0a1c4137eb23202e3dab2b8ceaf9bc0eb8c0
Andrew Morgan
--
You received this message because you are subscribed to the Google Groups \
"qubes-devel" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to qubes-devel+unsubscribe@googlegroups.com. To post to this group, \
send email to qubes-devel@googlegroups.com. To view this discussion on the web visit \
https://groups.google.com/d/msgid/qubes-devel/omfqe5%24inc%241%40blaine.gmane.org. \
For more options, visit https://groups.google.com/d/optout.
["signature.asc" (application/pgp-signature)]
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic