Boot firmware, included in embedded software, is a wild ecosystem that causes cascading problems (e.g. embedded silicon vendors maintaining custom distros and forked firmware). The complexity of software and hardware generally is constantly growing. This is also true of boot firmware. UEFI may eventually get adopted via ARM SystemReady, assuming embedded vendors make hardware changes (i.e. including SPI flash for firmware). Now is a great opportunity to create the right incentives for vendors to use open source distro independent firmware rather than unmaintained closed UEFI firmware. If boot firmware is available on LVFS then vendors are only responsible for bootstrapping boards (or may leave it to customers). Then LVFS/fwupd becomes the standard mechanism for updating firmware. Vendors can choose to pay a lot of money to develop and maintain the firmware internally or hopefully see OSFW on LVFS and realize it isn’t worth their time and money to produce a inferior product.
Although Linux has become the standard for less resource constrained embedded systems, the boot firmware it depends on is still a wild and complex ecosystem. The diversity of that ecosystem means that numerous devices are not easy to use because they require hardware specific knowledge to boot. In contrast, x86-based laptops, desktops and servers generally require no specialized knowledge and support installing Linux using bootable installer images.
Laptops, desktops and servers are not only less diverse (or conversely more standardized), but they include dynamic buses (i.e. PCI) where devices can be enumerated at runtime, which is not as dependent on a centralized static description of the hardware (i.e. devicetree). Additionally, they include a legacy BIOS or UEFI firmware provided by the hardware vendor (often outsourced to a third-party).
For a very long time it has been possible through floppy disks, CDs, or USB thumb drives to boot an installer on a device to install an operating system. This is so normal, we would never consider another option, but typically embedded devices (particularly reference or development boards) are supported using pre-built images (i.e. SD card images). Imagine booting an installer and being able to install Ubuntu or Fedora onto an eMMC, large SPI flash or even NVMe on an embedded board just like a laptop, desktop or server!
The largest issue holding back major Linux distributions from providing installer images (at least for the most popular development boards) is the unwieldy ecosystem of boot firmware. Nearly all embedded boards include SD, USB and Ethernet controllers. And those that do not wouldn’t likely be popular or common enough to be targeted by a major Linux distribution.
However the solution is not to repeat the mistakes of x86 platforms. The minor improvements of UEFI over legacy BIOS do not fix the core problems caused by broken, unmaintained and closed firmware that has plagued x86 platforms for years. Furthermore the standard introduces unnecessary complexity and a model that invites bugs and exploits. Finally, it is made worse by cases where hardware is not accessible in Linux. For example, the kernel has an advanced thermal subsystem that is rarely used because vendor firmware effectively blocks control of fans through additional cumbersome interfaces that require additional expertise.
The lines are increasingly getting blurred between laptops, desktops, servers and embedded devices. The incredibly popular Raspberry Pi boards have been successful in large part because they are much more approachable. Although they have have headers exposing various buses and I/O they are also often used as normal computers or servers. This is clearly evidenced by the latest Raspberry Pi 5, which includes a fully custom “I/O controller” reminiscent of north and south bridges on x86 motherboards. That controller required a significant amount of money to develop.1 Furthermore, the limited consumer supply of Raspberry Pi boards indicates a large percentage of sales going to businesses. Raspberry Pi hardware, like Ubuntu, is rarely the best tools for the job, but they are attractive because they make the difficult tasks of building or selecting hardware and an operating system much quicker. This reflects the power of brand names in both the consumer and commercial markets.
A fundraising round in 2023 led by ARM valued Raspberry Pi at $561 million and has potentially risen to $600 million. The company plans to go public in 2024.2
Linux, especially as it is today, was not inevitable. The Unix Wars did not guarantee the birth and growth of Linux. The ecosystems of boot firmware and microcontrollers are evidence of that.
Storage available on embedded boards varies, which impacts how boards are bootstrapped and how firmware is installed. This in turn impacts how software is distributed (e.g. USB bootable installer vs. pre-configured SD-Card image).
Note that eMMC includes hardware partitions not found on SD-Cards that could be used to store some firmware (see eMMC Hardware Partitioning by Pengutronix). However these are rarely used in practice, likely because they’re not supported by the chip ROM.
Oddly, the Single Board Computer Database does not include persistent storage and the List of Single-board computers lumps all storage together in one column and at least some entries do not appear to be accurate. For example, it claims the Raspberry Pi 4 has 0.5 MiB of flash.
The absence of storage for boot firmware on the Raspberry Pi 5 has been led one vendor to sell NVMe drives with Raspberry Pi OS preinstalled! Additionally, that combined with the decision to exclude an NVME connector on the board has led to all sorts of fun for end users.3
The closet thing to a standard in embedded boot firmware is currently U-Boot, which has been around for a long time and inspired many people to develop better alternatives (e.g. ChromeOS Depthcharge, Penguintronix Barebox). More recently that includes coreboot (and it’s Rust reimplementation for RISC-V named oreboot), as well as using Linux to boot Linux (e.g. LinuxBoot/u-root). Note that Depthcharge is a coreboot payload.
ARM has taken the classic approach of encouraging hardware vendors to store hardware specific firmware and configuration on the device as is done with BIOS/UEFI on x86 laptops, desktops and servers.
The standard Workstation version of Fedora is currently available as a Live ISO for x86 and aarch64 based on UEFI.
Rather than having large hardware vendors write low-quality unmaintained closed source code, the open source boot firmware community should make the decision on how boot firmware should be designed. As is often the case, software developers like to think they can solve all of the problems. Or many folks will defer to corporate implementations and standards. Instead the experts in this field focused on maintainability, usability and cross vendor support should drive the decision making. Why should boot firmware be held to a lower standard than Linux? As with Linux, vendors may take those open solutions and fork them as they see fit, but at least the foundation is solid.
Independently, BayLibre developed k-boot, which uses a
slightly modified ATF to pass a device tree to the kernel, which loads a
busybox userspace and runs a custom init program named
kbootd.
The folks behind the Austrialian free softwre group, OzLabs, and associated consulting company, Code Construct, developed petitboot that is hosted under the OpenPower Github organization. It also claims to support ARM64 with ACPI. It is a set of Linux userspace programs leveraging udev and kexec with a ncurses interface. It is designed to be built and integrated into an image with the op-build tool that wraps Buildroot. Note that petitboot is included as a package in mainline Buildroot, but also maintains a separate package.
The Libre Computer Project that develops open hardware SBCs supports something called, SNOR bootloader, that is built using their libretech-builder-simple. shell scripts.4 Their boards generally have a eMMC connector and then either SPI flash and/SD-Card slot. They appear to use U-Boot, ATF, OPTEE, EDK2 and Buildroot.
Raspberry Pi OS ships a desktop application that acts as a much more limited bootable installer. For folks who don’t want to run that program, they support some poorly documented configuration through text files on a FAT boot partition. And perhaps some configuration is possible using cloud-init. None of these solutions is comprehensive, has as many features or is as easy to use as a normal bootable installer.
0xide was heavily inspired by OSFC 2019 and the community behind it.
Standards often become the focus rather than the underlying problems that inspire them. Trying to provide high level interfaces can create an enormous amount of complexity that takes a lot of time and money to maintain, so that the interface becomes more complex than the thing it is abstracting.
There is a large spectrum of how standards can be written to address those issues. While UEFI is a good example of failure to do so, other potential standards attempt to improve upon it.
The Linux Userspace API (UAPI) group has specified a combination of kernel cmdline, initrd and kernel image in a UEFI Portable Executable (PE) format container named Universal Kernel Image (UKI).
Additionally, there is the Universal Payload (UPL) that is a FIT image (as specified by U-Boot).
Note that Coreboot maintains a Jenkins server at https://qa.coreboot.org/, has a custom make-based build system (including building custom toolchains) and uses Gerrit for code review.
Coreboot appears to be largely supported by modern technically minded hardware vendors like Google and System76 and unfortunately not by any silicon vendors.
Embedded boot firmware is designed to be built for a particular board. This presents a significant problem when trying to support multiple boards and chips in a single (or limited number) of installer images.
New logic needs to be added to try to figure out the hardware that the installer is being run on, at least to the point where a user can provide input. This reduces the number of images that distribution needs to provide.
The boot ROM in SoCs that is implemented by the chip vendors varies considerably, with some even supporting numerous and modern modes to boot the system.
Likely the best initial approach would be to build firmware from U-Boot SPL and LinuxBoot, where U-Boot can eventually be replaced with coreboot.
Each Linux distribution should not need to know the specifics of boot firmware anymore than it should know about the firmware loaded onto peripheral devices. Therefore pre-built firmware fits more naturally into LVFS.
There is a somewhat problematic analogy to the Kernel maintainers decision to create a CVE Numbering Authority (CNA). That is given it has stirred up a bit of online controversy (perhaps mostly from commentators). After years of dealing with a burdensome and at times problematic CVE process, maintainers worked with CVE employees and the board of directors to create an CNA as other projects have done in the past in order to bring some order.5
Ignoring the online drama, it is an important example of technical members of a software project doing something other than software development. Ideally, OSFW would treat LVFS in a similar way by becoming an official vendor with a Product Security Incident Response Team.
Of the 39 vendors on LVFS that support an update protocol, 14 support UEFI UpdateCapsule. Nearly all other protocols are vendor specific.6
It seems less convenient to learn a hardware vendor specific tool to deploy an OS rather than the standard workflow of downloading a bootable image for your prefered Linux distribution, writing it to a USB dongle and booting the installer. In fact, this is typically not the only special feature provided by the vendor. Using vendor specific firmware and using it to install an OS puts an enormous amount of trust in that vendor with little to no benefit.
As with laptop vendors, shipping hardware with a distribution pre-installed is completely reasonable as long as the customer has the option to install a different distribution through the above standard workflow.
Standard distro installers may not work out-of-the-box on embedded systems. For example, non-x86 architecture, device support (e.g. eMMC, SPI flash), programming firmware, and lack of a display may all be issues not common among standard laptops, desktops and servers.
The Anaconda web-based installer is primarily focused on servers, but overlaps with embedded systems, which may not have a GPU or an appropriate connector (e.g. HDMI). Note that Anaconda also supports a text-based interface.
The Anaconda Makefile supports build itself as RPMs and then using those RPMs when building a boot image (see Building installation images).
├── boot.catalog
├── EFI
│ └── BOOT
│ ├── BOOTAA64.EFI
│ ├── fonts
│ │ └── unicode.pf2
│ ├── grubaa64.efi
│ ├── grub.cfg
│ └── mmaa64.efi
├── Fedora-Legal-README.txt
├── images
│ ├── efiboot.img
│ └── pxeboot
│ ├── initrd.img
│ └── vmlinuz
├── LICENSE
└── LiveOS
└── squashfs.imgmenuentry 'Start Fedora-Workstation-Live 39' --class red --class gnu-linux --class gnu --class os {
linux /images/pxeboot/vmlinuz root=live:CDLABEL=Fedora-WS-Live-39-1-5-respin rd.live.image quiet rhgb
initrd /images/pxeboot/initrd.img
}Note that although it is not in the EFI directory and
does not have an appropriate file extension, vmlinuz is
actually a EFI application. See CONFIG_EFI_STUB and
Documentation/admin-guide/efi-stub.rst. This reminds me of
Yocto/OpenEmbedded naming files that contain a complete
.config as defconfig.
In Buildroot I generated a extlinux.conf:
label RK3399_ROCKPRO64 linux
kernel /Image
devicetree /rk3399-rockpro64.dtb
append earlycon=uart8250,mmio32,0xff1a0000 root=/dev/mmcblk1p4 rw rootwaitComplexity is constantly growing in hardware and software (e.g. complex SoCs with multiple cores, buses and fancy IP blocks).
Software developers love building abstractions, but those abstractions can create enormous complexity in systems. Often a developer must understand the fundamentals of a system as well as the architecture of the abstraction. Also, edge cases may require large changes to abstractions. And solutions may be broadly adopted without understanding the problem. For example, why do we all need to run TrustZone code and build, configure and run ATF? TrustZone was created to protect media content from being pirated by running firmware on a shadow processor below the operating system.
Silicon vendors obviously would like to sell as many chips as possible. To make those chips more appealing it is important to provide some software/firmware support to ease the integration of those chips, which thereby reduces costs for customers and makes the chips cheaper. The quality of that software can impact the ease of integration, provide important features and reduce security vulnerabilities.
Currently software support for embedded boards is vendor specific and opinionated software stack consisting of firmware, kernel, and userspace.
Additionally, there are the following open source community projects:
There are broadly several use cases for customers of boards, a development setup, a deployed production setup (analogous to a server) and a managed setup (i.e. a hobbyist setup). Unlike like headless server use cases, development often requires some access to the target hardware. When targeting servers more of the development can occur on a laptop.
A classic Linux distribution is optimal for development where there are not extreme resource constraints. Custom embedded distro tools require an enormous amount of additional skill and experience.
Is there a viable use case where classic distro can be used for deployments? Presumably this is the case for the Raspberry Pi which despite having the largest hobbyist market is still overwhelming selling to commercial customers. They still develop and maintain their own Debian-based distribution.
Secure boot is largely a marketing term. Generally, it may refer to an authenticated chain of trust and sometimes also encrypted data.
Linux distributions for laptops and desktops do not provide an authenticated chain of trust.
A chain of trust must be established from the boot ROM, which is unique to each chip vendor.
Laptops and desktops only encrypt the root filesystem with a passphrase that is stored hashed and encrypted on an unencrypted filesystem. Newer devices include a Trusted Platform Module (TPM) that can store keys such that the key never leaves the TPM. Therefore it is not possible for an attacker to use a key from one device on other devices.
For deployed embedded systems, as with servers, it is not possible to enter a passphrase when booting the system. Therefore a key (equilivant to a passphrase) needs to be stored on the device securely so that it can not be accessed by an attacker.
Note this is not a requirement for development boards that may have human input (i.e. the software developer). However for the development use case encryption may not even be necessary.
Given potential issues with early graphics, especially with UEFI, it may be worth focusing on the web-based Fedora installer since that seems more broadly applicable to embedded boards.
Problems arise when some service or an EFI framebuffer etc is expected. Let’s say we do our LinuxBoot with native Linux DRM for graphics. Then we execute a distro kernel with its initrd, and that tries to draw on the EFI framebuffer, which does not exist. So the screen goes blank. We would need to set up at least some minimal tables for that to work out. WDYT, should we meet on that medium ground? It wouldn’t be hard to write those few structs somewhere in memory, at least.
TUXEDO mentions UEFI, but not more details on their main website. They appear to optionally support 3mdeb Dasharo/coreboot. By default the presumably use the firmware provided by their ODM, Clevo.
System76 prominently uses coreboot.
Framework appears to use InsydeH20 UEFI BIOS, a proprietary firmware, which it distributes using LVFS for Linux. Currently that only includes one firmware file, which was uploaded in the last 6 months. LVFS indicates they also work with binarly to “harden firmware security”.
NovaCustom, which is based in the Netherlands, announced on April 3, 2024 new laptops with 3mdeb Dasharo/coreboot, a goal of supporting LVFS by Q3 2024 and an option to use Heads instead of TianoCore EDK II as a coreboot payload.8 Note the Heads channel on the OSFC Slack appear to be mostly abandoned.
What 0xide is to servers and Framework to laptops, a new start-up could be to embedded. There are significant differences between the servers/laptop ecosystems and embedded.