Another attempt at blogging

Moving the blog

Sat, 05 Aug 2017 00:00:00 +0100

Well, it was never going to last forever. I have abandoned this blog. The content has been migrated over to my new blog, and new posts are being added there. Move along, nothing to see here...

QEMU USB host device pass-through

Tue, 13 Dec 2016 18:09:00 +0000

Speeding up UEFI driver development

While working on the ChaosKey UEFI driver, I once again found myself in a situation of tedium: how to simplify driver development when the target for my device driver is firmware. Moving around a USB key with a FAT filesystem may be a completely valid thing to do in order to install a pre-existing driver. But when you are trying to learn how to write drivers for a new platform (and write blog posts about it), the modify -> compile -> copy -> reboot -> load -> fail -> try again cycle gets quite tedious. As usual, QEMU has/is the answer, letting you pass a USB device controlled by the host over to the guest.

Make sure your QEMU is built with libusb support

Unsurprisingly, in order to support passing through USB device access to guests, QEMU needs to know how to deal with USB - and it does this using libusb. Most Linux distributions package a version of QEMU with libusb support enabled, but if you are building your own version from source, make sure the output from configure contains

libusb            yes

If it does not, under Debian, $ sudo apt-get build-dep qemu should do the trick - anywhere else make sure you manually install libusb v1.0 or later (Debian's libusb-dev package is v0.1, which will not work), with development headers. Without libusb support, you'll get the error message 'usb-host' is not a valid device model name when following the below instructions.

Identifying your device

Once your QEMU is ready, in order to pass a device through to a guest, you must first locate it on the host.

$ lsusb
...
Bus 008 Device 002: ID 1d50:60c6 OpenMoko, Inc
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
....

The vendor ID (1d50) of ChaosKey is taken from the OpenMoko project, which has repurposed their otherwise unused device registry space for various community/homebrew devices.

If you want to be able to access the device as non-root, ensure the device node under

/dev/bus/usb/<bus>/<device>

(in this case /dev/bus/usb/008/002) has suitable read-write access for your user.

Launching QEMU

Unlike the x86 default "pc" machine, QEMU has no default machine type for ARM. The "generic virtualized platform" type virt has no USB attached by default, so we need to manually add one. ich-usb-uhci1 supports high-speed, and so is needed for some devices to work under emulation. Using the "virtual FAT" support in QEMU is the easiest way to get a driver into UEFI without recompiling. The example below uses a subdirectory called fatdir which QEMU generates an emulated FAT filesystem from.

USB_HOST_BUS=8
USB_HOST_PORT=2
AARCH64_OPTS="-pflash QEMU_EFI.fd -cpu cortex-a57 -M virt -usb -device ich9-usb-uhci1"
X86_OPTS="-pflash OVMF.fd -usb"
COMMON_OPTS="-device usb-host,bus=usb-bus.0,hostbus=$USB_HOST_BUS,hostport=$USB_HOST_PORT -nographic -net none -hda fat:fatdir/"

This can then be executed as
$ qemu-system-x86_64 $X86_OPTS $COMMON_OPTS
or
$ qemu-system-aarch64 $AARCH64_OPTS $COMMON_OPTS
respectively.

Note: Ordering of some of the above options is important.
Note2: That AARCH64 QEMU_EFI.fd needs to be extended to 64MB in order for QEMU to be happy with it. See this older post for an example.

My usage then looks like the following, after having been dropped into the UEFI shell, with a bunch of overly verbose debug messages in my last compiled version of the driver:

Shell> load fs0:\ChaosKeyDxe.efi
*** Installed ChaosKey driver! ***
Image 'FS0:\ChaosKeyDxe.efi' loaded at 47539000 - Success
Shell> y (0x1D50:0x60C6) is my homeboy!
1 supported languages
0: 0409
Manufacturer: altusmetrum.org
Product: ChaosKey-hw-1.0-sw-1.6.6
Serial: 001c00325346430b20333632

And, yes, those last three lines are actually read from the USB device connected to host QEMU is running on.

Final comments

One thing that is unsurprising, but very cool and useful, is that this works well cross-architecture. So you can test that your drivers are truly portable by building (and testing) them for AARCH64, EBC and X64 without having to move around between physical machines.

Oh, and another useful command in this cycle is reset -s in the UEFI shell, which "powers down" the machine and exits QEMU.

UEFI Driver Development - Part 1

Tue, 13 Sep 2016 22:02:00 +0100

One of the key features of UEFI is that the specification and the APIs/ABIs it guarantees provides the ability to produce portable applications, drivers and libraries (in the form of protocols). On the simpler side, by letting you compile the driver once for each architecture - and on the more space age side by letting you build a single driver that works across all architectures (using EFI Byte Code). The extra magic comes in the form of Option ROM support, which lets plug-in PCI cards keep a driver onboard, informing UEFI to load it on boot. (Any jokes about Forth drivers for Open Firmware go here.)

So, having never actually written a UEFI driver from scratch, and most of the drivers I have come across having really been platform drivers, I figured that would be a good start to write a standalone driver from scratch. And the outcome is this slightly hands-on blog post series. This part covers:

creating a new driver from scratch
building it as a standalone driver
loading it from the UEFI Shell
having it detect the presence of a device it recognizes
unloading it from the UEFI Shell

Having just come across a hardware random number generator called ChaosKey, I figured this would make an excellent candidate.

Creating a standalone UEFI driver from scratch

Since UEFI drivers are meant to be standalone buildable, they tend to be kept in separate directories. Since this driver is intended to run during/after DXE phase, let's put everything in a directory called ChaosKeyDxe (CamelCase mandatory - link takes you to the directory in git, where you can view the files directly).

Create a build information file

First of all, we need a build information file (.inf). The format of the build information file is described in the EDK II INF File Spec, which can be found in the EDK2 github pages.

Start with a [Defines] section

[Defines]
  INF_VERSION                    = 0x00010019
  BASE_NAME                      = ChaosKeyDxe
  FILE_GUID                      = 9A54122B-F5E4-40D8-AE61-A71E406ED449
  MODULE_TYPE                    = UEFI_DRIVER
  VERSION_STRING                 = 1.0
  ENTRY_POINT                    = ChaosKeyEntryPoint
  UNLOAD_IMAGE                   = ChaosKeyUnload

The INF_VERSION reflects which version of the INF file format is being followed, in this case 1.25 (0x0001.0x0019). Interestingly, nearly all build information files I have come across before specify 0x00010005 ... cargo culting from earlier examples.

Followed by BASE_NAME, the single word identifier used for this component. To keep things simple, I'm reusing the directory name, for the same reasons.

And then a FILE_GUID, generated uniquely for this file - for example through this online generator, ensuring Uppcase and Hyphens are both ticked. If this string is copied from an existing template rather than uniquely generated, really bad stuff will happen.

And then MODULE_TYPE to tell the build system we are producing a UEFI_DRIVER.

A VERSION_STRING is also required - this is simply a UCS2 string indicating the version of the driver.

The ENTRY_POINT function is automatically called at driver load time, and needs to contain code that registers the driver with the system, and do any other global setup needed to make the driver ready to set up individual devices.

UNLOAD_IMAGE points to the function cleaning up after the driver when it is to be unloaded. This is not really mandatory for a driver expected to be used for booting the system (it will be discarded by the operating system anyway, unless it takes specific actions to keep bits resident), but it comes in very handy for development.

The build information file usually includes a (commentary only) stanza stating which architectures the executable is expected to work on.

#
#  VALID_ARCHITECTURES           = AARCH64 ARM EBC IA32 IPF X64
#

The remainder of the file simply specifies which source files are used to build the driver, which declaration files it uses (MdePkg/MdePkg.dec), which library classes it needs (resolved into specific libraries by the build description file) and which protocols it consumes.

[Sources]
  ChaosKeyDriver.h
  DriverBinding.c

[Packages]
  MdePkg/MdePkg.dec

[LibraryClasses]
  UefiBootServicesTableLib
  UefiDriverEntryPoint
  UefiLib

[Protocols]
  gEfiUsbIoProtocolGuid

Adding some actual code

Since we are not yet implementing any actual functionality beyond discovery, the only C source file added at this point is DriverBinding.c. This all comes down to implementing an instance of EFI_DRIVER_BINDING_PROTOCOL. Let us go through that, function by function.

EntryPoint

All the entry point function does is register the protocol instance, as defined in the gUsbDriverBinding struct, with the system - and return EFI_SUCCESS, printing an informational message as it does so. The gUsbDriverBinding struct contains pointers to the Supported(), Start() and Stop() functions defined by the protocol, as well as a Version number which lets UEFI pick the most up to date driver if multiple are available.

Supported

When a new device is detected in the system, UEFI will ask all of the plausible drivers whether they know how to deal with it, by calling their Supported() function. This implementation is probably one of the few bits of this driver that is pretty much feature complete - all it really needs to do is to find the USB manufacturer/device IDs and see if they are ones the driver knows how to handle. It then returns EFI_UNSUPPORTED if this is a device it does not support, or EFI_SUCCESS if it is a device it supports.

Start/Stop

Start() and Stop() are left empty for now, returning EFI_UNSUPPORTED whenever they are called. This is something that will be filled in in part 2 of this series.

UnloadImage

Finally, when (if!) we unload the driver, UnloadImage() ensures that the bits that were registered by EntryPoint() are unregistered again.

Building and using the driver

Building the standalone driver

In order to build a standalone driver, you need a platform description (.dsc) file, mapping your library dependencies to actually available libraries. One way of doing this is to implement your own complete .dsc. However, this is exactly what EDK2's OptionRomPkg/OptionRomPkg.dsc provides. So a simpler way can be to simply add the .inf to the [Components] section of OptionRomPkg.dsc. After that, the build should be as easy as:

GCC5_AARCH64_PREFIX=aarch64-linux-gnu- build -a AARCH64 -t GCC5 -p OptionRomPkg/OptionRomPkg.dsc -m OpenPlatformPkg/Drivers/Usb/Misc/ChaosKeyDxe/ChaosKeyDxe.inf

Loading the driver

For this example (and because Juno's built-in magical program-over-USB filesystem is insane), let's load the driver from a USB key. For simplicity's sake, have it plugged in when powering on and drop into the UEFI Shell. In my case, the USB key ended up as filesystem FS2:.

Shell> FS2:
FS2:\> load ChaosKeyDxe.efi
add-symbol-file /home/leif/work/git/edk2/Build/OptionRomPkg/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/Usb/Misc/ChaosKeyDxe/ChaosKeyDxe/DEBUG/ChaosKeyDxe.dll 0xF85E4000
Loading driver at 0x000F85E3000 EntryPoint=0x000F85E4044 ChaosKeyDxe.efi
*** Installed ChaosKey driver! ***
Image 'FS2:\ChaosKeyDxe.efi' loaded at F85E3000 - Success
FS2:\>

At this point, you can verify that the driver has loaded by invoking the drivers command:

FS2:\> drivers
             T   D
             Y C I
             P F A
DRV VERSION  E G G #D  #C  DRIVER NAME                         IMAGE PATH
=== ======== = = = === === =================================== ==========
 23 00000030 D N N   1   0                        Fv(B73FE497-B92E-416E-8326-45AD0D270092)/FvFile(1DF18DA0-A18B-11DF-8C3A-0002A5D5C51B)
...
 6F 0000000A D N N   1   0                        Fv(B73FE497-B92E-416E-8326-45AD0D270092)/FvFile(4579B72D-7EC4-4DD4-8486-083C86B182A7)
 9E 0000000A ? N N   0   0                        FS2:\ChaosKeyDxe.efi

You can see how the driver is loaded, and has the driver handle number 9E.

Now, plug in the ChaosKey:

FS2:\> ChaosKey (0x1D50:0x60C6) is my homeboy!
UsbSelectConfig: failed to connect driver Not Found, ignored

We can see how the Supported() function is invoked, and we then see an error message. The error is triggered by our empty Start() function returning EFI_UNSUPPORTED when UEFI attempts to bring the device online. We'll fix that bit in part 2 of the series.

Unloading the driver

During development, it can be handy to be able to load/unload without a full reboot. To do so, just call unload with the driver handle you got from the drivers command. Add the -v option to get some extra output.

FS2:\> unload -v 9E
     Revision......: 0x00001000
     ParentHandle..: FD317918
     SystemTable...: FDF30018
     DeviceHandle..: FD243918
     FilePath......: FC98FF98
     OptionsSize...: 0
     LoadOptions...: 
     ImageBase.....: F85E3000
     ImageSize.....: 8000
     CodeType......: EfiBootServicesCode
     DataType......: EfiBootServicesData
     Unload........: F85E4000
Unload - Handle [FC990598].  [y/n]?
y
remove-symbol-file /home/leif/work/git/edk2/Build/OptionRomPkg/DEBUG_GCC5/AARCH64/OpenPlatformPkg/Drivers/Usb/Misc/ChaosKeyDxe/ChaosKeyDxe/DEBUG/ChaosKeyDxe.dll 0xF85E4000
Unload - Handle [FC990598] Result Success.
FS2:\>

You can also verify the unload was successful by running drivers again, and seeing this driver no longer listed.

Opening up UEFI platform support

Wed, 22 Jun 2016 16:54:00 +0100

As some of you may know, one of the main points I have been pushing for over the past couple of years has been the availability of reference code for real platforms. Given that historically, firmware source code has simply not been available for UEFI-based computers, resolving this has been no trivial exercise.

I gave an outline presentation about this at the UEFI spring plugfest in Seattle last year, announcing that I was starting a work-in-progress tree called OpenPlatformPkg (due to lack of imagination) where I was going to start staging support for multiple platforms/devices. I created this as a separate tree holding only the platform code. As a way to keep things clean, and if I am to be honest also because this enforces deeper consideration of changes to core code. (I'm not suggesting this is the way it should be done when moving platform support into Tianocore, just that it removed several potential conflict points when interacting with organisations not used to working against a single source tree for multiple platforms.)

Since then, the most visible impact of this in the EDK2 source tree has been the removal of platform code from EDK2, as I moved it into OpenPlatformPkg. I finally started to see contributions outside of the ARM ltd. platforms a couple of months ago, and have since struggled to keep up with reviewing the patches contributed:

Over the past few weeks, I have merged support for:

Hisilicon D02 and D03 platforms
AMD Overdrive (Seattle development platform)
LeMaker Cello (Seattle-based 96boards EE)

I am also currently working with Semihalf to get support for Marvell Armada 7040 in.

A warning for the purists - I have not been mandating that the entirety of the platform support is provided in source form. But we do now have a tree which can be used to rebuild against newer versions of upstream EDK2, and I do mandate that each platform adds some open source bits.

AMD

We have been working with AMD over the past few months to provide a port based on the official Seattle Firmware Development Kit (FDK) - the same that is used for the commercial firmware shipped on AMD Overdrive and Softiron 3000 platforms. Since this was never originally designed to be used in an open source context, it has included the relicensing of header files and some restructuring to permit simple rebuilding of the binary modules for new FDK releases.

So this new port comes with some blobs - BUT it does bring source code for ACPI, SMBIOS, (non-PCIe) SATA controller, Hardware RNG and overall platform initialization. And of course it reuses some existing open source drivers available in EDK2. From a personal perspective, this means I can now with a quick turnaround test new EDK2 core changes on my Overdrive.

This being a platform with all of its RAM > 4GB, it has also helped flush out some incorrect assumptions in core EDK2 code (and a Linux driver or two).

Hisilicon

We actually got most of the way towards a port for the D02 platform back in December - enough for me to agree to post it as a public branch on the main OpenPlatformPkg repository. But I still had a fair amount of things I wanted improved style wise, and the Hisilicon engineers were under pressure to provide a port for the newer D03 platform as well, so spent a few months getting this up and running.

But contrary to the story we have seen many times (with many different vendors) in the mobile space, once they completed this they came back. and put a big effort responding to review feedback to ensure both platforms could be merged simultaneously during our reference platform sprint in Cambridge (UK) last week.

Future

Well, Tianocore has now obtained a more standardised (and public) change process, and there is currently a proposal underway for how to support platforms in official repositories: http://thread.gmane.org/gmane.comp.bios.edk2.devel/11624 With a bit of luck and some hard work OpenPlatformPkg can become redundant the year after it was introduced!

I have also had a few more contacts from people looking to provide ports for upcoming (and existing) platforms, so there should be some more to show later in the year.

QEMU 2.4 and virtio-pci

Sat, 25 Jul 2015 00:57:00 +0100

I wrote in previous posts about QEMU and how to run ARM UEFI on it.