Introducing Qubes Odyssey Framework

Qubes OS is becoming more and more advanced, polished, and user friendly OS.

But Qubes OS, even as advanced as it is now, surely have its limitations. Limitations, that for some users might be difficult to accept, and might discourage them from even trying out the OS. One such limitation is lack of 3D graphics support for applications running in AppVMs. Another one is still-far-from-ideal hardware compatibility – a somehow inherent problem for most (all?) Linux-based systems.

There is also one more “limitation” of Qubes OS, particularly difficult to overcome... Namely that it is a standalone Operating System, not an application that could be installed inside the user's existing OS. While installing a new application that increases system's security is a no-brianer for most people, switching to a new, exotic OS, is quite a different story...

Before I discuss how we plan to address those limitations, let's first make a quick digression about what Qubes really is, as many people often get that wrong...

What Qubes IS, and what Qubes IS NOT?

Qubes surely is not Xen! Qubes only uses Xen to create isolated containers – security domains (or zones). Qubes also is not a Linux distribution! Sure, we currently use Fedora 18 as the default template for AppVMs, but at the same time we also support Windows VMs. And while we also use Linux as GUI and admin domain, we could really use something different – e.g. Windows as GUI domain.

So, what is Qubes then? Qubes (note how I've suddenly dropped the OS suffix) is several things:

• The way how to configure, harden, and use the VMM (e.g. Xen) to create isolated security domains, and to minimize overall system TCB.
• Secure GUI virtualization that provides strong gui isolation, while at the same time, provides also seamless integration of all applications running in different VMs onto one common desktop. Plus a customized GUI environment, including trusted Window Manager that provides unspoofable decorations for the applications' windows.
• Secure inter-domain communication and services infrastructure with centrally enforced policy engine. Plus some “core” services built on top of this, such as secure file exchange between domains.
• Various additional services, or “addons”, built on top of Qubes infrastructure, such as Disposable VMs, Split GPG, TorVM, Trusted PDF converter, etc. These are just few examples, as basically the sky is the limit here.
• Various additional customizations to all the guest OSes that run in various domains: GUI, Admin, ServiceVMs, and AppVMs.

Introducing Qubes HAL: Hypervisor Abstraction Layer

Because Qubes is a bunch of technologies and approaches that are mostly independent from the underlying hypervisor, as discussed above, it's quite natural to consider if we could easily build an abstraction layer to allow the use of different VMMs with Qubes, instead of just Xen? Turns out this is not as difficult as we originally thought, and this is exactly the direction we're taking right now with Qubes Odyssey! To make this possible we're going to use the libvirt project.

So, we might imagine Qubes that is based on Hyper-V or even Virtual Box or VMWare Workstation. In the case of the last two Qubes would no longer be a standalone OS, but rather an “application” that one installs on top of an existing OS, such as Windows. The obvious advantage we're gaining here is improved hardware compatibility, and ease of deployment.

And we can go even further and ask: why not use Windows Native Isolation, i.e. mechanisms such as user account separation, process isolation, and ACLs, to implement domain isolation? In other words why not use Windows OS as a kind of “VMM”? This would further dramatically improve then lightness of the system...

Of course the price we pay for all this is progressively degraded security, as e.g. Virtual Box cannot be a match to Xen in terms of security, both architecturally and implementation-wise, and not to mention the quality of isolation provided by the Windows kernel, which is even less.

 

But on the other hand, it's still better than using “just Windows” which offers essentially only one “zone”, so no domain isolation at all! And if we can get, with minimal effort, most of our Qubes code to work with all those various isolation providers then... why not?

Being able to seamlessly switch between various hypervisors is only part of the story, of course. The remaining part is the support for different OSes used for various Qubes domains. Currently we use Linux, specifically Fedora 18, in our GUI & Admin domain, but there is no fundamental reason why we couldn't use Windows there instead. We discuss this more in-depth in one of the paragraphs below.

The diagram below tries to illustrate the trade-offs between hardware compatibility and ease of deployment vs. security when using different isolation backends with Qubes. Some variants might also offer additional benefits, such as “super-lightness” in terms of CPU and memory resources required, as is the case with Windows Native Isolation.

 

Some example configurations

Let's now discuss two extreme variants of Qubes – one based on the baremetal Xen hypervisor and the other one based on Windows Native Isolation, so a variant from the opposite end of the spectrum (as shown on the illustration above).

The diagram below shows a configuration that uses a decent baremetal hypervisor, such as Xen, with abilities to securely assign devices to untrusted service domains (NetVM, UsbVM). So, this is very similar to the current Qubes OS.

 

Additionally we see separate GUI and Admin domains: the GUI domain might perhaps be based on Windows, to provide users with a familiar UI, while the Admin domain, tasked with domain management and policy enforcement, might be based on some minimal Linux distribution.

In the current Qubes OS there is no distinction between a GUI and an Admin domain -- both are hosted within one domain called “dom0”. But in some cases there are benefits of separating the GUI domain from the Admin domain. In a corporate scenario, for example, the Admin domain might be accessible only to the IT department and not to the end user. This way the user would have no way of modifying system-wide policies, and e.g. allowing their “work” domain to suddenly talk to the wild open Internet, or to copy work project files from “work” to “personal” domains (save for the exotic, low-bandwidth covert channels, such as through CPU cache).

The ability to deprivilege networking and USB stacks by assigning corresponding devices (NICs, and USB controllers) to untrusted, or semi-trused, domains provides great security benefits. This automatically prevents various attacks against the bugs in WiFi stacks or USB stacks.

What is not seen on the diagram, but what is typical for baremetal hypervisors is that they are usually much smaller than hosted hypervisors, implementing less services, and delegating most tasks, such as the infamous I/O emulation to (often) unprivileged VMs.

Let's now look at the other extreme example of using Qubes – the diagram below shows an architecture of a “Qubized” Windows system that uses either a hosted VMM, such as Virtual Box or VMWare Workstation, or even the previously mentioned Windows Native Isolation mechanisms, as an isolation provider for domains.

 

Of course this architecture lacks many benefits discussed above, such as untrusted domains for networking and USB stacks, small hypervisors, etc. But it still can be used to implement multiple security domains, at a much lower “price”: better hardware compatibility, easier deployment, and in case of Windows Native Isolation – excellent performance.

And it really can be made reasonable, although it might require more effort  than it might seem at first sight. Take Windows Native Isolation – of course just creating different user accounts to represent different domains is not enough, because Windows still doesn't implement true GUI-level isolation. Nor network isolation. So, there is a challenge to do it right, and “right” in this case means to make the isolation as good as the Windows kernel can isolate processes from different users from each other.

Sure, a single kernel exploit destroys this all, but it's still better than “one application can (legally) read all my files” policy that 99% of all desktop OSes out there essentially implement today.

Now, probably the best thing with all this is that once we implement a product based on, say, Qubes for Windows, together with various cool “addons” that will take advantage of the Qubes services infrastructure, and which shall be product-specific, it should then be really easy to upgrade to another VMM, say Hyper-V to boost security. And the users shall not even notice a change in the UI, save for the performance degradation perhaps (well, clearly automatic creation of VMs to handle various users tasks would be more costly on Hyper-V than with Windows Native Isolation, where “VMs” are just... processes).

Qubes building blocks – implementation details

Let's have a look now at the repository layout for the latest Qubes OS sources – every name listed below represents a separate code repository that corresponds to a logical module, or a building block of a Qubes system:

core-admin

core-admin-linux

core-agent-linux

core-agent-windows

core-vchan-xen

desktop-linux-kde

desktop-linux-xfce4

gui-agent-linux

gui-agent-windows

gui-agent-xen-hvm-stubdom

gui-common

gui-daemon

linux-dom0-updates

linux-installer-qubes-os

linux-kernel

linux-template-builder

linux-utils

linux-yum

qubes-app-linux-pdf-converter

qubes-app-linux-split-gpg

qubes-app-linux-tor

qubes-app-thunderbird

qubes-builder

qubes-manager

vmm-xen

vmm-xen-windows-pvdrivers

Because current Qubes R2 still doesn't use HAL layer to support different hypervisors, it can currently be used with only one hypervisor, namely Xen, whose code is provided by the vmm-xen repository (in an ideal world we would be just using vanilla Xen instead of building our own from sources, but in reality we like the ability to build it ourselves, slightly modifying some things).

Once we move towards the Qubes Odyssey architecture (essentially by replacing the hardcoded calls to Xen's management stack, in the core-admin module, with libvirt calls), we could then easily switch Xen for other hypervisors, such as Hyper-V or Virtual Box. In case of Hyper-V we would not have access to the sources of the VMM, of course, so would just be using the stock binaries, although we still might want to maintain the vmm-hyperv repository that could contain various hardening scripts and configuration files for this VMM. Or might not. Also, chances are high that we would be just able to use the stock libvirt drivers for Hyper-V or Virtual Box, so no need for creating core-libvirt-hyperv or core-libvirt-virtualbox backends.

What we will need to provide, is our custom inter-domain communication library for each hypervisor supported. This means we will need to write core-vchan-hyperv or core-vchan-virtualbox. Most (all?) VMMs do provide some kind of API for inter-VM communication (or at least VM-host communication), so the main task of such component is to wrap the VMM-custom mechanism with Qubes-standarized API for vchan (and this standardization is one thing we're currently working on). All in all, in most cases this will be a simple task.

If we, on the other hand, wanted to support an “exotic” VMM, such as the previously mentioned Windows Native Isolation, which is not really a true VMM, then we will need to write our own libvirt backend to support is:

core-libvirt-windows

... as well as the corresponding vchan module (which should be especially trivial to write in this case):

core-vchan-windows

Additionally, if we're building a system where the Admin domain is not based on Linux, which would likely be the case if we used Hyper-V, or Virtual Box for Windows, or, especially, Windows Native Isolation, then we should also provide core-admin-windows module, that, among other things, should provide Qubes qrexec implementation, something that is highly OS-dependent.

As can be seen above, we currently only have core-admin-linux, which is understandable as we currently use Linux in Dom0. But the good news is that we only need to write core-admin-XXX once for each OS that is to be supported as an Admin domain, as this code should not be depend on the actual VMM used (thanks to our smart HAL).

Similarly, we also need to assure that our gui-daemon can run on the OS that is to be used as a GUI domain (again, in most cases GUI domain would be the same as Admin domain, but not always). Here the situation is generally much easier because “with just a few #ifdefs” our current GUI daemon should compile and run on most OSes, from Linux/Xorg to Windows and Macs (which is the reason we only have one gui-daemon repository, instead of several gui-daemon-XXX).

Finally we should provide some code that will gather all the components needed for our specific product and package this all into either an installable ISO, if Qubes is to be a standalone OS, like current Qubes, or into an executable, in case Qubes is to be an “application”. The installer, depending on the product, might do some cool things, such as e.g. take current user system and automatically move it into one of Qubes domains.

To summary, these would be the components needed to build “Qubes for Windows” product:

core-admin

core-admin-windows

core-agent-windows

core-vchan-windows

core-libvirt-windows

desktop-windows

gui-agent-windows

gui-common

gui-daemon

windows-installer-qubes-for-windows

qubes-builder

qubes-manager

Additionally we will likely need a few qubes-app-* modules that would implement some "addons", such as perhaps automatic links and documents opening in specific VMs, e.g.:

qubes-app-windows-mime-handlers

Here, again, the sky's the limit and this is specifically the area where each vendor can go to great lenghts and build killer apps using our Qubes framework.

Now, if we wanted to create "Qubes for Hyper-V" we would need the following components:

core-admin

core-admin-windows

core-agent-linux

core-agent-windows

core-vchan-hyperv

desktop-windows

gui-agent-linux

gui-agent-windows

gui-common

gui-daemon

windows-installer-qubes-hyperv

qubes-app-XXX

qubes-builder

qubes-manager

vmm-hyperv

Here, as an example, I also left optional core-agent-linux and gui-agent-linux components (the same that are to be used with Xen-based Qubes OS) to allow support for also Linux-based VMs – if we can get those “for free”, then why not!

It should be striking how many of those components are the same in both of those two cases – essentially the only differences are made by the use of different vmm-* components and, of course, the different installer

It should be also clear now how this framework now enables seamless upgrades from one product (say Qubes for Windows) to another (say Qubes for Hyper-V).

Licensing

Our business model assumes working with vendors, as opposed to end users, and licensing to them various code modules needed to create products based on Qubes.

All the code that comprises the base foundation needed for creation of any Qubes variant (so core-admin, gui-common, gui-daemon, qubes-builder and qubes-manager) will be kept open source, GPL specifically. Additionally all the code needed for building of Xen-based Qubes OS with Linux-based AppVMs and Linux-based GUI and Admin domains, will continue to be available as open source. This is to ensure Qubes OS R3 that will be based on this framework, can remain fully open source (GPL).

Additionally we plan to double-license this core open source code to vendors who would like to use it in proprietary products and who would not like to be forced, by the GPL license, to share the (modified) sources.

All the other modules, especially those developed to support other VMMs (Hyper-V, Virtual Box, Windows Native Isolation), as well as those to support Windows OS (gui-agent-windows, core-agent-windows, core-admin-windows, etc) will most likely be proprietary and will be available only to vendors who decide to work with us and buy a license.

So, if you want to develop an open source product that uses Qubes framework, then you can freely do that as all the required core components for this will be open sourced. But if you would like to make a proprietary product, then you should buy a license from us. I think this is a pretty fair deal.

Current status and roadmap

We're currently working on two fronts: one is rewriting current Qubes code to support Qubes HAL, while the other one is adding a backend for Windows Native Isolation (which also involves doing things such as GUI isolation right on Windows).

We believe that by implementing two such extreme backends: Xen and Windows Native Isolation we can best show the flexibility of the framework (plus our customer is especially interested in this backend;)

We should be able to publish some code, i.e. the framework together with early Qubes OS R3 that will be based on it, sometime in fall or maybe earlier.

We obviously are determined to further develop Xen-based Qubes OS, because we believe this is the most practically secure OS available today, and we believe such OS should be open source.

Qubes R2 will still be based on the Xen-hardcoded code, because it's close to the final release and we don't want to introduce such drastic changes at this stage. The only thing that Qubes R2 will get in common with Qubes Odyssey is this new source code layout as presented above (but still with hardcoded xl calls and xen-vchan).

So, this is all really exciting and a big thing, let's see if we can change the industry with this :)

Oh, and BTW, some readers might be wondering why the framework was codenamed “Odyssey” -- this is obviously because of the “HAL” which plays a central role here, and which, of course, also brings to mind the famous Kubrick's movie.

Qubes 2 Beta 2 has been released!

Qubes R2 Beta 2 with KDE 4.9 environment (click for more screenshots)

We're progressing fast and today I would like to announce the release of Qubes R2 Beta 2 ISO. The installation and upgrade instructions, as well as the ISO itself, can be found via the our wiki page. As usual, please remember to verify the digital signature of the downloaded ISO.

The major changes in this beta release include:

• Upgraded Dom0 distribution to the latest Fedora 18 (all previous releases used Fedora 13 for Dom0!)
• Upgraded default VM template also to Fedora 18
• Upgraded Dom0 kernel to 3.7.6
• Upgraded KDE environment in Dom0 (KDE 4.9)
• Introduced Xfce 4.10 environment for Dom0 as an alternative to KDE
• A few other fixes and improvements, including the recently discussed Disposable VM-based PDF converter

The upgrade of the Dom0 distribution and kernel should significantly improve hardware compatibility – one of the major problems with Qubes adoption so far, as we hear. Now, with the latest GPU drivers and Xorg packages, I hope we will be able to cover a much boarder range of hardware (especially all the newer GPUs).

The upgrade of KDE in Dom0 is mostly an eye-candy type of improvement (but then, who doesn't like eye candies!), as is the introduction of the Xfce4 as its alternative, although, arguably, Xfce4 is considered faster and lighter than KDE. In Qubes the choice of an actual desktop environment that runs in Dom0 is not as important as it is on traditional Linux systems, I think, simply because most of the functionality, typically provided by such environments, such as apps and file management, is simply... disabled, because on Qubes there are no user apps or files in Dom0.

Nevertheless people love and hate particular window managers and environments, and we hope that now, by supporting alternative environments, we could appeal to even more users.

I'm glad that we just completed this difficult phase of upgrading Qubes Dom0 distribution (for the first time since Qubes R1 Beta 1!) -- this forced us to clean up the code and prepare for some even bigger and bolder changes in the near future. But those will come only in Release 3. As far as Release 2 is considered, we do have quite a few more tickets scheduled for R2 Beta 3 milestone, but most of those represent various addons, rather than modifications to Qubes core software.

So what are those brave changes that are to happen in Release 3? That I will write about in a separate blog post... Stay tuned!

So, please enjoy this latest Qubes R2 beta release, and be sure to send all your questions and comments, as well as the HCL info, to the qubes-devel mailing list. I have already upgraded my primary laptop to this release a few days ago and everything seems to be working fine, so fear not!

Converting untrusted PDFs into trusted ones: The Qubes Way

Arguably one of the biggest challenges for desktop security is how to handle those overly complex PDFs, DOCs, and similar files, that are so often exchanged by people, or downloaded from the Web, and that often provide a way for the attacker to compromise the user's desktop system.

Today I would like to discuss a recent innovation we created for Qubes OS that allows to securely convert those pesky PDFs (as well as essentially any graphics files) into trusted PDFs. Here by a “trusted PDF” I mean a file that should be harmless to the user's system, so, a non-malicious PDF.

A few years ago, we have already introduced a mechanism in Qubes OS called Disposable VMs, that can be used to safely open any file, including PDFs, DOCs, etc. The file is being opened in a... well a dedicated disposable VM that is created within seconds (typically below 5 seconds) and all the file processing and rendering happens inside this VM. Once the document is closed, the disposable VM is automatically destroyed, and any changes to the file (e.g. if the was an editable DOC file) are automatically propagated back into the original file. This mechanism is very powerful, and I often use it for my daily work. However, it surely is a bit cumbersome – who wants to wait 5 seconds for the PDF to open, especially if I have a dozen of invoices to look through! So, today I present an alternative approach...
 
Approaches to converting PDFs

The problem of converting a potentially untrusted PDF into a harmless one is certainly not a new one. Some tools have already be created for this task.

The typical approach is to parse the original PDF, look for “potentially dangerous things” there, and remove them. As simple as that! This is, of course, a typical AntiVirus approach to the problem. And, typically as it is for most AV approaches, it's completely useless against any more skilled and determined attacker (and these are the ones we fear the most, don't we?).

A somehow better approach is to parse the original PDF, disassemble it into pieces, and then reassemble them into a new PDF only using the “trusted” pieces – this, I think, could be called a whitelisting approach.

Anyway, the fundamental problem with the approaches mentioned above, is that all of them require parsing of the original PDF file. And parsing is where the “big bang” usually happens. Parsing is where our, normally pretty decent, code, comes in close, intimate contact with some unknown complex input data, which often leads to a successful abuse or exploitation.

Parsing PDFs safely

So, how to perform parsing safely? Of course, that's simple! Let's run the parser in an isolated container – in case of Qubes we already have an ideal such container: it's the Disposable VMs.

But, before we get too excited, let's think more about it – say we run the parser safely isolated in a Disposable VM, meaning it couldn't harm any of the rest of the system, except for the Disposable VM itself, which we however won't worry much about, because it is disposable... But then what?

We want our PDF back in our original VM, to actually use it, right? But we cannot just copy the result from the Disposable VM, because if it got compromised, as a result of parsing of the malicious PDF, then we would like get... a compromised converted PDF. So, this approach gives us nothing!

(Even though our “solution” incorporates all the obligatory buzzwords: “Disposable VMs” (“Micro Disposable VMs”?), “VMs isolated using hardware Intel vPRO technology” and, of course, the “hypervisor”! Sometime just the mere fact we use “hardware virtualization” buys us nothing... People seem to forget about this sometimes.)

So, the trick to make this approach meaningful is to introduce what I will call a “Simple Representation” of the input file. More on this, straightforward concept, below. The idea is that our parser (that runs in a Disposable VM) will be expected to return the Simple Representation of the original PDF. Of course, it might very well go wild (as a result of exploitation by the PDF it parses), and don't obey our expectations, and instead return something totally different and potentially malicious. But that doesn't matter! The whole point of the Simple Representation is that it should be, well... simple to parse it safely and discard in case what we're getting doesn't look like the Simple Representation.

Ok, so what's the simplest possible representation of an arbitrary PDF file? Yes, it's the RGB format, which is essentially just a raw array of RGB values for each pixel. In fact, I'm not sure there could be anything simpler in the Known Universe to represent a PDF file...

Now this is all becoming simple: we would expect our parser to send us just two things: the dimensions (W x H) of the bitmap representation of each of the page of the PDF in question, and each of the PDF page itself converted into a raw RGB format. If the parser didn't obey, we would still interpret whatever stream of bytes we get as a RGB bitmap – in the worst case the PDF we create would look like un-tuned analog TV screen.

The diagram below summaries this idea:

Implementing this all on Qubes

Now I would like to show how easy it is to implement such PDF converter service using the Qubes advanced infrastructure that we call qrexec, and which is part of Qubes core for quite some time now.

First, let's choose the PDF and image conversion tools. The choice of PDF converter is not security critical, because it will run in an isolated Disposable VM. Here I decided to use pdftocairo converter, which is part of the poppler-utils package on Fedora. We will also use ImageMagick's “convert” command to convert the PNG files (produced by pdftocairo, one for each PDF page) to the raw RGB format. Incidentally ImageMagick supports RGB format natively. As mentioned above, in addition to sending the raw RGB file, we would also need to send the width and height of the pixmap – those can easily be obtained using ImageMagick's “identify” command. Again, all those programs discussed so far are not security critical – they might get exploited during the processing of the untrusted input PDF file, and we don't worry about that at all.

On the receiving side, however, we need to use a foolproof parser for the RGB format. Again, this is what we gain in this whole process – instead of requiring a foolproof-and-also-being-able-to-produce-non-malicious-PDFs parser, we only require a foolproof RGB parses, and that's quite a gain! The ImageMagick's convert comes to mind again here, and one might want to use it like this:

convert page.rgb page.pdf

Unfortunately this would be wrong, because the convert program would still try to detect the “real” format of the page.rgb file, and, if it looked more like, say, JPEG or PDF, it would parse it accordingly, compromising all our careful plan! What we really need is to tell our convert program to always treat the input as raw RGB file, instead of trying to be (too) smart and trying to guess the format by itself. This can be achieved by adding the “rgb:” prefix in front of the input argument, which provides explicit input format specification:

convert -size ${IMG_WIDTH}x${IMG_HEIGHT} -depth ${IMG_DEPTH} rgb:$RGB_FILE pdf:$PDF_FILE

Now also needed to add size and depth explicitly, because the raw RGB format doesn't convey such information (well, it has no header of any sort at all!). Of course we need to obtain the width and height from the parser, but we can validate such input rather easily. In addition we make sure that the received RGB file has exactly the size as indicated by width and height. With those precautions in place, there would have to be really a gapping hole in the ImageMagic's RGB parsing code for the attacker to exploit this. Perhaps instead of using the ImageMagick's convert I should have written a small script in python that would parse the received RGB file (and save it into a... RGB file, for later processing by ImageMagick), but I sincerely think this would be an overkill here. 

 

Finally we can write the following two simple bash scripts, one for client: qpdf-convert-client, and the other one, qpdf-convert-server, for the server (which runs in a Disposable VM).

Additionally we also need to create a policy file in Dom0 in /etc/qubes_rpc/policy/ to allow to use this service. The policy file content for this service should look like this:

$anyvm $dispvm allow

... which is pretty self explanatory. When I do development I also add another line to the policy file like this:

$anyvm devel-vm ask

... to allow me to run the server inside my 'devel-vm' VM, instead of running it in Disposable VM every time, which would be very inconvenient for development, as it would require me to update the Disposable VM template each time I wanted to test a new version of qpdf-convert-server.

The policy file should be placed in Dom0 in /etc/qubes_rpc/policy/qubes.PdfConvert file – here the name of the file must be the same as the name of the service, as invoked via qrexec_client_vm command, discussed below.

And, one last thing, in the destination VM we must also create a file that will map the service name (so, the qubes.PdfConvert in our example) to the actual binary that should be called in the VM when the service is invoked. So, the file should be named: /etc/qubes_rpc/qubes.PdfConvert (again, this is now in a VM, not in Dom0, also note the lack of policy/ subdir), and it is another one-liner with the following content:

/usr/lib/qubes/qpdf-convert-server

The full source code of qpdf-converter can be seen and downloaded from this git repo.

We're ready now to test our qubes.PdfConvert service: in the requesting VM, i.e. the one from which we want to initiate the conversion process we do:

[user@work Downloads]$ /usr/lib/qubes/qrexec_client_vm '$dispvm' qubes.PdfConvert /usr/lib/qubes/qpdf-convert-client ITLquote.pdf

-> Sending file to remote VM...

-> Waiting for converted samples...

-> Receving page 2 out of 2...

-> Merging pages into a single PDF document...

-> Converted PDF saved as: ITLquote.trusted.pdf

-> Original file saved as .ITLquote.pdf

Again, for development process I would replace '$dispvm' with something like 'devel-vm'.

The qrexec_client_vm command, used above, is not actually intended to be used by user directly (that's why it's installed in /usr/lib/qubes instead of /usr/bin/), and so when one creates a Qubes qrexec service, it's customary to create also a small wrapper around qrexec, like this one, that makes using the service simple.

The presented converter saves the original file as .${original_pdf} making it a hidden file to help the user avoid accidental opening. The new, converted file gets .trusted.pdf suffix appended to the base name of the original file. I discuss more issues regarding the human factor and avoiding accidental opening in one of the next paragraphs below. The converter can also be used to convert essentially any image file, such as JPEG, PNG, etc, into a PDF, using the same method.

As you can see creating client-server services in Qubes is very simple – in fact it took me just one afternoon to get the inital working version of the converter (with subsequent "polishing" over the next 2 days).

The qrexec infrastructure takes care about all the under-the-hood tasks, such as starting the necessary VMs, e.g. creating Disposable VM to handle the service request,establishing communication channels between VMs (which are ultimately implemented on top of Xen's shared memory), redirecting client and server's stdin and stdout to each other, so that writing services is very simple, even in shell, and, of course, obeying policies defined centrally in Dom0.

Most “inter-VM” features in Qubes, such as secure file copy between domains, opening files in Disposable VMs, time synchronization, appmenus synchronization, etc, are all implemented on top of qrexec. A notable exception is clipboard exchange, which is implemented as part of the GUI protocol, but still uses the same common qrexec code for policy processing (e.g. I use this policy to block clipboard and file exchanges between my work and personal domains).

Limitations, other Simple Representations

The obvious disadvantage of converting a PDF to an RGB representation is that one looses text search, as well as copy and edit capabilities (e.g. in case of PDF forms). So, converting Intel's IA32 Software Developer's Manual this way would certainly not be a good idea... But, hey, such large PDFs can always be opened in a Disposable VM – they would be fully functional then, only that you would need to wait a few seconds for the PDF window to pop up. Or, better yet, why not keep all such PDFs in a dedicated domain? E.g. I have a VM called “work-pub” where I keep tons of various, publicly available PDFs, such as the mentioned Intel's SDM, as well as various chipset docs, conferences papers and slides, and generally lots of stuff. The key point is that all in this VM is public material (and also all is related to my work), so that I don't really care if any of those PDFs compromises my work-pub domain. In the worst case, I will revert the VM from backup and download any missing PDFs again from the web. They are public after all. 
 

But the PDF conversion described above comes extremely useful in case of all the various invoices, Purchase Orders, NDAs, contracts, and god-knows-what-else PDF documents, which I'm forced to deal with in my “work” domain (where my email client runs). Most of those are one pagers, or maximum a few pages long documents, so the fact that they got converted to a bitmap provides me with very little discomfort. At the same time I gain incredible freedom in opening all those documents natively in my work VM, without fearing that one of those invoices will comrpomise my work domain (which would be a rather sad thing for me, although the really sensitive stuff is still in some other domains ;)

An interesting question is, however, can we come up with another form of Simple Representation that would allow e.g. to preserve the text searching ability of the converted PDFs (and DOCs, PPTs)? Probably... yes. The choice of the Simple Representation should be thought of as of a trade-off between security and document's features preservation. I'm not an expert on PDF and DOC formats (and I'm not sure I want to be) but it seems plausible that one could disassemble PDF into simple pieces, select the really simple ones, send those pieces as a Simple Representation back to client, and have them reassembled back into a almost-fully-functional PDF. Here, again, the point is that the PDF parsing is done in isolated Disposable VM, while the reassembly in the trusted VM. Anyway, let me leave it as a exercise for the reader :)
 

Preventing user mistakes

Being able to right-click on a PDF file and have it converted into a trusted PDF is one thing. Having this mechanism adopted by users and actually making their daily computing safer, is another story.

Users will likely have hundreds of PDF spread over their home directories, and the real challenge is how to make sure that the user never accidentally opens the unconverted, untrusted PDF. We can think of several approaches to this problem:

• We modify the Thunderbird, Firefox, etc, e.g. by providing specific plugins, to always perform PDF conversion on each file that we got via email or downloaded from the Web. Additionally we convert all the already present PDFs in the user's home directory (file system?). And, additionally, we modify Qubes file copy operation to also always do automatic PDF conversion whenever one transfers files from other domains (if Qubes qrexec policy allows for such transfer in the first place, of course).

This approach would not be optimal, because some PDFs, as we discussed above, might not be well suited for conversion-through-bitmap process – they might be large PDFs where text search is crucial, some conference papers for review, where text copy is crucial, or some editable forms. That's why it seems better to take a slightly different approach:

• We modify mime handlers for PDF files (as well as any other files that our converter supports) and then upon every opening of the file (e.g. via mouse click in a file manager) our program gets to run and its job is to determine whether the file should be opened natively, converted to a trusted PDF, or perhaps opened in a Disposable VM. Of course, upon “first” opening we should probably ask the user about the decision, if this cannot be determined automatically. E.g. if we can reliably determine the file is already converted, we can safely open it without prompting the user, but if it's not, we should ask – perhaps the user would like to open it in a Disposable VM instead of converting, or perhaps the file should be considered trusted anyway, because it was created by the user herself.

This second approach seems like a way to go, and we will likely implement it sooner or later (probably sooner, but after the upcoming R2 Beta2). It should also be noted, that typically user would need such mechanism only in some domains – e.g. I really feel the need for such protection in my “work” domain, but not in any other. But that, of course, depends on how one partitions their digital life into security domains.

One important detail worth mentioning here, is that we should unconditionally disable “Thumbnail View” in whatever file manager we use (which itself is really a stupid feature – can people not read filenames anymore or something?).

Qubes: from containers isolation down to apps protection

The mechanism introduced today, in addition to the Disposable VMs mechanism introduced earlier, represents a trend in Qubes development of “stepping down” into AppVMs in order to also make the VMs themselves somehow more secure (in addition to the isolation between the VMs).

Originally Qubes aimed at containers isolation only. This included protecting the system TCB where techniques such as deprivileged networking stacks (and optionally also deprivileged USB stacks) have been deployed, as well as custom GUI virtualization, and generally somehow “hardened” Xen configuration. This also included protecting the VMs from each other, where techniques such as secure clipboard, secure file copy and generally secure qrexec infrastructure have been introduced, as well as trusted GUI subsystem with explicit domain decorations.

But now, Qubes is stepping down into the AppVMs in order to make the VMs themselves also less prone to compromise. We surely will be working on more such mechanisms in the future. We still are only at the beginning of the quest to create a Reasonably Secure Desktop OS!

PS. The presenetd converter will be part of the Qubes R2 Beta 2, that is expected to be released... in the comming days. Experienced users of Qubes R1 and R2 Beta 1 can install the converter immediately by building the rpms from the git repo.


PS. WTF is happening with the Blogger web interface? Seriously, I don't remember being so frustrated using any software in the recent years that I am right now, when editing this post (as well as the last several ones). It sometimes honours the line breaks, sometimes do not, sometimes inserts a couple of new lines, sometime removes them, sometime mysterious spaces appear at the end of lines, sometime those cannot be removed... It doesn't allow to paste pre-formatted code-listing (at least I couldn't figure out how to make it honour tabs). And yes, I'm using the "Compose mode", because when I try to switch to the HTML mode, not only I'm overwhelmed with tons of HTML markups, nobody knows what for, but also when I switch back to the Compose mode, my article tends to get even more fucked up! Really, a shame. I wish I could go away to some other blogging service, but I'm afraid that converting all my posts would be even a bigger PITA... Sigh.

Qubes 2 Beta 1 with initial Windows support has been released!

It's my pleasure to announce the first Beta for Qubes Release 2 is now available for download.

This release introduces generic support for fully virtualized AppVMs (called HVMs in Xen parlance), and specifically initial support for Windows-based AppVMs integration. It's been quite a challenge to add support for secure HVMs to Qubes without breaking its security architecture, and I already wrote about it in the past.

Generic support for HVMs means you can now install many different OSes as Qubes VMs, such as various Linux distros, BSD systems, and, of course, Windows. Essentially all you need is an installation ISO and the whole process is similar to creating a VM in a program like Virtual Box or VMWare Workstation (although we believe the underlying architecture for this is more secure in Qubes).

Additionally we provide a set of tools for Windows-based AppVMs (Windows 7 specifically) which allow for tight integration with the rest of the Qubes system. This currently includes support for secure (and policy controllable) clipboard and file exchanges between the Windows-based AppVMs and other AppVMs, integration with Qubes advanced networking infrastructure, and PV drivers for faster operation. As of now there is still no seamless app integration for Windows applications, so Windows VMs are presented as full-desktop-within-a-window, but we're aiming to add support for this in the next Betas.

Unlike the rest of Qubes, which is distributed under a GPL v2 license, the Qubes Windows Support Tools are not open sourced and are distributed as binaries only, under a proprietary license. They are free to use for any Qubes 2 user. The tools are not part of the Qubes 2 installation ISO (which is GPL), and are down loadable on demand.

More information about creating and using HVM domains, including Windows-based AppVMs, can be found in the wiki here.

To summary, here's a quick list of some of the exciting new features that toady's release brings in:

• Support for generic fully virtualized VMs (without qemu in the TCB!)
• Support for Windows-based AppVMs integration (clipboard, file exchange, qrexec, pv drivers)
• Secure audio input to select AppVMs (Hello Skype users!)
• Clipboard is now also controlled by central policies, unified with other qrexec policies.
• Out of the box TorVM support
• Experimental support for PVUSB
• Updated Xorg packages in Dom0 to support new GPUs
• DisposableVM customization support
• ... and, as usual, various fixes and other improvements :)

Existing users of Qubes R1 can upgrade without needing to reinstall – the upgrade procedure is described here. Standard installation is described here.

Enjoy!

PS. Please send all the technical questions to the qubes-devel mailing list, instead posting them as comments to this blog. Keep the comments here for more generic discussions.

PS2. As usual, I would like to remind that we have little control over the servers that are used for Qubes ISO distributions and that the downloads should be verified according to the procedure described here. We always assume that even our own servers (git, wiki, yum) could be compromised, and yet this should not affect Qubes security in any way, because of the extensive use of digital signatures everywhere in the development and distribution process.

How is Qubes OS different from...

Many people ask how does Qubes OS differ from other approaches to desktop security. Today I'm trying to answer the most popular questions.

• Why bother with Qubes OS, if any Linux/BSD already allows to setup different user accounts, or some form of light-weight containers or sandboxes, such as chroot, LXC, SELinux?

First, if you use Xorg or similar X-based server as your GUI server, and this is what nearly all Linux, and most of the other non-Windows OSes use, then you don't have any form of GUI-level isolation, which is essential for a desktop system. I wrote more about this surprising problem some time ago. Proper GUI-level isolation was one of the main goals for Qubes.

Second, all mainstream desktop OSes, such as Windows, Linux, BSD, even OSX, are all based on a monolithic kernels, which present a significant security problem. This is because a typical monolithic kernel of a contemporary desktop OS contains tens of millions of lines of code, and to make it worse, most of this code is reachable from (untrusted) applications via all sorts of APIs, making the attack surface on the kernel huge. And it requires just one successful kernel exploit to own the whole system, bypassing any security mechanisms that might have been built on top of it, such as SELinux, LXC, etc.

Additionally, all the various drivers, networking and USB stacks, are also hosted in the kernel, making attacks via buggy networking (e.g. via buggy 802.11 stacks or buggy firmware) or USB stacks a practical possibility. And there is essentially nothing one can do about it, when using an OS based on a monolithic kernel.

In Qubes, on the other hand, we use Xen hypervisor to provide security isolation between domains, and Xen is just a few hundred of thousands lines of code. It also doesn't need to provide all sorts of APIs to applications, because the Xen hypervisor is essentially only interested in CPU scheduling, memory management and power management, and very few things beyond that. Most notably, the Xen hypervisor knows nothing about networking, disk storage, filesystems, USB stacks, etc, as all those tasks are delegated to (often untrusted) service VMs.

• How is Qubes better than just running a bunch of VMs in VMWare or Virtual Box?

First, products such as VMWare Workstation or Fusion, or Virtual Box, are all examples of type II hypervisors (sometimes called “hosted VMMs”), which means that they run inside a normal OS, such as Windows, as ordinary processes and/or kernel modules. This means that they use the OS-provided services for all sorts of things, from networking, USB stacks, to graphics output and keyboard and mouse input, which in turn implies they can be only as secure as the hosting OS is. If the hosting OS got compromised, perhaps via a bug in its DHCP client, or USB driver, then it is a game over, also for all your VMs.

Second, those popular consumer type II VMM systems have not been designed with security as a primary goal. Instead, their main focus has been on easy of use, performance, and providing seamless integration of the guest OS(es) with the host OS. Especially the latter, which involves lack of good method to identify which domain a given application belongs to (so, lack of trusted Window Manager), support for shared clipboards which every other VM can steal, insecure file sharing methods, and others, all make it not a very desirable solution when strong domain isolation is important. (This is not to imply that Qubes doesn't support clipboard or file sharing between domains, it does – it's just that we do it in a secure way, at least so we believe). On the other hand, there are many usability improvements in Qubes that are specific to multi-domain system, and which you won't find in the above mentioned products, such as trusted Window Manager that, while maintaining great seamless integration of all the applications onto a common desktop, still allows the user to always know which domain owns which window, support for advanced networking setups, per-domain policies, the just mentioned secure mechanisms for clipboard and filesystem sharing, and many other. Qubes also focuses on making the VMs light-weight so that it was possible to run really a lot of them at the same time, and also on mechanism to allow for secure filesystem sharing between domains (templates).

Finally, the commercial hosted VMMs are really bloated pieces of code. They support everything and the kitchen sink (e.g. Open GL exposed to VMs, and various additional interfaces to allow e.g. drag and drop of files to/from the VM), and so, the attack surface on such a VMM system is orders of magnitude bigger than in case of Qubes OS.

• How does Qubes compare to [your favourite academic microkernel]? 

While the Xen hypervisor can indeed be considered a microkernel if you're not a strict terminology freak, Qubes itself is much more than just the hypervisor. Qubes is everything that is needed to build a reasonably secure desktop OS on top of a baremetal hypervisor (or microkernel). Theoretically, with just a few cosmetic changes (at least architecture-wise), Qubes could perhaps swap the Xen hypervisor for some other hypervisor or microkernel, such as perhaps Hyper-V, KVM, or some more exotic one. Thus, it makes little sense to compare Qubes with a hypervisor or microkernel project. What makes sense is to compare the Xen hypervisor, as used in Qubes, with some other hypervisor or microkernel.

Ok, so how does Xen compare with other hypervisors or microkernels out there? We think Xen is unique because it combines an elegant architecture (type I, baremetal, hypervisor) with a number of practical features, such as power management, support for Intel VT-d and driver domains, support for both para-virtualizaed, and fully-virtualized VMs, and many more, not found in e.g. academic microkernels/hypervisor projects, that otherwise seem attractive from the architecture point of view.

• How is Qubes better than Google Chrome OS?

 First, Chrome OS is not a general purpose OS. Second, it's based on Linux with all its security limitation that are a result of using a monolithic kernel described above (e.g. all the networking and USB stacks in the kernel without a possibility to deprivilige them). Not being a traditional general purpose OS, Chrome is able to avoid many of the challenges of desktop computing, such as the need to define security domains, inter-domain file exchange (as there is essentially no filesystem visible to the user), and others, which is good, of course. But then again, Chrome OS is essentially just an environment to run the Chrome Browser, so the comparison to Qubes is a bit of a misunderstanding.

Technical aspects aside, there is always the privacy concern associated with running everything in a browser – why would all my private data be managed and accessible to some 3rd party organizations and their administrators?

• How is Qubes better than [your favorite commercial military-grade certified secure OS]?

You must have heard about the super secure military-grade, formally verified, 100% certified, and generally “unbreakable” operating systems made by companies such as Green Hills, Lynx Works, and others. How do they compare to Qubes OS?

Really, I have no idea. For a mere mortal like myself (and perhaps not a US citizen), it seems impossible to get any more technical documentation of those systems (anything beyond the marketing pseudo-technical gibberish), not to mention a trial copy to play with...

Thus, from my point of view, those systems are just a vaporware. If you, my dear reader, are privileged enough to have access to such system, then good for you, but don't expect me to treat seriously a security product that is not available for a wider audience to touch and play with... (And the Chineese surely have the copies already to play with ;)

• How is Qubes different than Bromium's “micro virtualization” solution?

Many people asked recently about the Bromium's upcoming product and how it differs from Qubes OS. Unfortunately there are few public information available on this product – essentially there is one not-very-technical whitepaper and there are Ian Pratt's presentation slides from the recent XenSummit about u-Xen, apparently a hypervisor that is to be ultimately used in their upcoming product.

The whitepaper suggests that Bromium is based on a hosted (type II) hypervisor running within a normal Window OS, and that this hypervisor is used to spawn a new “micro VM” for each new “task”, where apparently the task might be something as granular as opening a new tab in a Web browser, which makes it somehow similar to Google Chrome's approach. Clearly, the Bromium's main goal seem to be to automate the process of creating separation domains, which is in contrast with what we do on Qubes OS, where the user is required to define the domains explicitly.

The Pratt's slides provide also some technical insight into how Bromium intends to secure their hypervisor. As just discussed above, a hosted hypervisor must normally trust the hosting OS, in this case Windows, which, for obvious reasons, is not a good idea from the security standpoint. Pratt, however, clearly states that “host (...) can not interfere with the privacy or integrity of the hypervisor or other guests” (slide #8). This is a strong statement, so let's take a closer look at their approach to this problem.

The Bromium's idea of how to make their hypervisor (and the VMs) protected from a potentially malicious host OS is not really breakthrough: the slides suggest to “deprivilege the host into a VT-container” (I think the verb to bluepill is now an accepted term for such action ;), and to remove the host's access to the hypervisor pages (via EPT), as well as protect DMA access from devices via VT-d, plus to make this all sensible, use DRTM scheme such as Intel TXT, to load such a hypervisor from within a potentially untrusted OS.

So, what's wrong with the idea of a load-on-the-fly-secure-VMM-system? Isn't Ian Pratt correct that one could protect its memory and execution from the interference of the host? Actually that is possible – Intel TXT, VT-x, VT-d, and EPT give us means to achieve that (although there are a number of catches here). But he's missing one important point: it's the untrusted OS that still owns and manages the input devices (e.g. via USB stacks and drivers) and, most importantly, the output (via the GUI subsystem and drivers). Ensuring that the host OS cannot interfere (e.g. sniff the screen of trusted applications) might be very difficult, or even impossible, in practice.

If I ever was to break the security of such a system, I would just follow the simple way:

1) Infect the host e.g. via one of the many USB attacks (remember they cannot have sandboxed USB driver domain, as they have only a type II hosted hypervisor),

2) Hook somewhere into the GUI subsystem and keep recoding all the interesting data from the screen...

... or something like that ;)

There are also many other things that needs to be answered and which the publicly available documents are silent about, such as e.g. how does the system handle installation of new applications? How is clipboard and file exchange between (micro)VMs handled? How large are the interfaces exposed to each (micro)VM? For now, without a solid documentation available, and without any code to play with, it is just another vaporware for me. (Interestingly there seem to be Bromium's Beta program, which however doesn't seem to be working, at least not for me -- I tried to signup twice, but never got any confirmation or response...?)

• How is Qubes different from Xen Client?

In many aspects, Xen Client might be the most similar product to Qubes OS. Like Qubes, it is based on the Xen hypervisor and so it is also a standalone OS, that one must install instead of one's favorite system, and also, like Qubes, it is targeted for desktop systems, and also offers a possibility to run a few VMs at a time.

However, XenClient has been designed with a different goal in mind, namely as a “Virtual Desktops To Go” solution, while Qubes has been designed to provide seamless experience for secure multi-domain desktop system. As a result, lots of focus in Qubes has been put on creating trusted GUI subsystem, support for advanced networking configurations, secure inter-VM clipboard and file sharing, secure method to reuse the same filesystem as a basis for the AppVMs, and also to optimize the AppVMs so they start almost instantly and take little memory, so that one could easily run many of them at the same time. All those things seem to be missing from Xen Client (as well as solid technical documentation about its design).

***

I surely have missed a few other products or approaches -- feel free to point them out in the comments, and I might write a continuation post one day.