I have been an active user of various Linux distributions since college including Ubuntu, Arch Linux, Debian and most recently Fedora.

Package management


There are two different approaches for communicating between the CPU and peripheral hardware on x86, memory-mapped I/O and port-mapped I/O.


The typical boot process consists of a Boot ROM in the target chip that loads firmware from a location that may be configured using jumpers. In embedded systems that firmware often consists of the U-Boot Secondary Program Loader (SPL), which in turn loads U-Boot (see Booting ARM Linux).

The Linux boot process can be measured using tools like bootchart or systemd-analyze.


Compare-exchange (i.e. cmp_xchg()) or compare-and-swap does an atomic read-modify-write (RMW) by reading the value at the given address and comparing it to a value provided without taking a spinlock by leveraging processor specific instructions. If they are equal it will write the other provided value. It only writes a value if the current state is as the caller expects. Either way it returns the value it read. For example, the following continuously tries to increment a value while the value read at the address by cmp_xchg() doesn’t match the value read by the caller.

do {
    int old = *addr;
    int new = old + 1;
} while (cmp_xchg(addr, old, new) != old);

Note that cmp_xchg() doesn’t appear in the kernel.


Interrupt handlers need to be short because they run in a processes context. Deadlocks can occur when a interrupt handle tries to acquire a lock that is owned by the process that was interrupted.

When the kernel is running in atomic context, things like sleeping are not allowed. Code which handles hardware and software interrupts is one obvious example of atomic context.

Power management

Linux power management is divided into runtime and system frameworks, where the runtime framework manages individual devices while the system manages the entire system.

Enable system power management using the CONFIG_PM Kconfig option.

If the actions taken by a driver are the same for system operations as for runtime operations add the following:

static const struct dev_pm_ops mydrv_dev_pm_ops = {
        SET_RUNTIME_PM_OPS(driver runtime_suspend_callback,

Power domains

Generic power management domains (genpd) allow devices to be managed independent of their subsystems or drivers. Their topology of devices can be described in the device tree using power-domains = <&pm_domain 0>;. Note that genpd uses the power_on() and power_off() callbacks. Wake-up latency constraints can be enforced using the genpd governor. Additionally, the set_performance_state callback can limit the power consumption of a device by varying the voltage to power rails.

Default sysfs

Several power management attributes are created for devices in sysfs. The power/control attribute allows userspace to take control of device power management by writing on or hand back control by writing auto.

Three more attributes are automatically created and describe the power management of a device; runtime_active_time, runtime_status, runtime_suspended_time.


The power management framework maintains a usage counter that is incremented using pm_runtime_get_() and decremented using pm_runtime_put(). If the counter reaches zero the device may be put to sleep.

There is also a feature named autosuspend that allows a device to put itself to sleep after a specified period (see power/autosuspend_delay_ms).


Common mechanisms to reduce processor power usage include the following.


When debugging a driver mount debugfs and use dynamic debug to enable debug print statements.

mount -t debugfs none /sys/kernel/debug


Devices exposed in /dev are managed by a userspace program named udev. Originally developed by Greg K.H., it has since been forked and maintained under systemd and Gentoo as eudevd (see #gentoo-eudev on Freenode). There are also a few folks in #udev on Freenode.

Events are generated by the kernel, processed by udev, which in turn generates it’s own events. It is possible to see the former with udevadm monitor -k and udevadm monitor -u will output the latter.

Udev rules include an environment that is populated from various places (see IMPORT). I found this the most confusing part of udev. Rules have access to sysfs attributes through the environment, which can be printed using udevadm info -a /sys/....

Rules are typically located in /usr/lib/udev/rules.d/*.rules and /etc/udev/rules.d/*.rules.

The syntax includes operators like = for assignment and == for matching. To skip an ACTION rules will often include a match of that action with a GOTO to LABEL="foo_end" at the end of the file.

All commands are run after processing the rule regardless of their order in the file.

A good example is 60-serial.rules, which creates symlinks for serial devices under /dev/serial.

There is also a database, known as hwdb, which I didn’t investigate.

I found the following pattern helpful for debugging adding a particular device:

cd /sys/...
udevadm test --action="add" "${PWD}"

After updating a rule it may be necessary to run udevadm control --reload.

Some of the keys match upwards against parent devices until they find a match (e.g. KERNELS, SUBSYSTEMS, DRIVERS).

See also