What is an embedded OS?

An Embedded Operating System is an Operating System designed to run on specific hardware. For example, many smart TV’s these days are running an embedded OS. Because the OS is made for the specific hardware, it can be configured to start up in the blink of an eye.

What is Buildroot?

Buildroot, just like the yocto project, is a set of tools to help automate the build process of an embedded Linux based OS. Buildroot is configured through a set of configuration files, after which it builds an entire Linux based OS.

What are the benefits of creating a custom embedded OS?

There are a few great benefits to creating a custom embedded OS (and not using a prebuilt image file):

  • The kernel and root filesystem can be seperated from the bootloader and stored on a remote server, to make the local system image really small.
  • If Buildroot is configured to only create a root filesystem, the resulting OS can run from a bootloader and/or kernel that is not built by Buildroot.
  • The U-boot bootloader is very configurable and can be configured to start the right kernel for the right Raspberry Pi board if needed. This includes downloading the right kernel (and device tree) from a remote server and then booting it.
  • The embedded OS built by buildroot supports many init systems (busybox, openrc, systemd, etc.).
  • The embedded OS is very minimal. This saves space and teaches how a Linux based OS works.
  • Almost any program that would normally run on Raspberry Pi OS (formerly called Raspbian) can easily be cross-compiled during the build process. This includes using CMake.
  • There are many debugging tools that can be built into the embedded OS to debug a certain application.
  • The cross-compiling toolchain can be fully configured within Buildroot. An external cross-compiling toolchain can also be used if desired (crosstool-NG for example).

The goal

In this post, a basic Buildroot OS will be created. This will be a minimal build; it boots but is utterly useless. At the end of this post, the minimal Buildroot OS will be used for two other projects:

  • Creating an OS that serves as an access point.
  • Cross-compiling Allegro5 and Dune Dynasty and adding it to the minimal OS.

Setting up a cross-compiling toolchain

If the kernel is compiled using arm-none-eabi-gcc and the rest of the system is compiled using aarch64-linux-gnu-gcc, the system will not run. To ensure that the system runs (well) on the target board, a cross-compiling toolchain will be set up. This toolchain contains a cross-compiling gcc, ld and other tools that would normally be used to (cross-)compile a program or OS. The tools in the toolchain will be used to compile the whole system.

In this post, crosstool-NG is used.

Downloading

git clone https://github.com/crosstool-ng/crosstool-ng
Bash

Compiling

./bootstrap # only if crosstool-NG is cloned from github
./configure --enable-local
make
Bash

Configuring a cross-compiling toolchain

The command

./ct-ng list-samples
Bash

will give a list of possible targets to compile for. The entry aarch64-rpi4-linux-gnu is used by executing the following command:

./ct-ng aarch64-rpi4-linux-gnu
Bash

Because the GNU C library is quite big compared to something like uClibc, the .config file will be edited to use uClibc instead of the GNU C library. The goal is to make the target system a little smaller in disk size. The .config file is edited using the following command:

./ct-ng menuconfig
Bash

The following changes were made inside the menuconfig menu:

  • Under C library, the option C library is changed to uClibc

Then the menuconfig is exited and restarted to reload all options:

./ct-ng menuconfig
Bash
  • Under C library, the option Add support for locales is enabled
  • Under C library, the option Add support for IPv6 is enabled
  • Under C library, the option enable iconv is enabled
  • Under C library, the option Add support for fenv.h is enabled
  • Under C compiler, the option Version of gcc is set to the second to latest version (at time of writing this was set to version 10.3.0)
  • Under C compiler, the option C++ is enabled
  • Under Operating system, the option Version of linux is set to the latest version (at time of writing this was version 5.15.2)
  • Under Debug facilities, the option gdb is disabled (optional)
  • Under Toolchain options, the option Tuple's alias is set to toolchain (optional)

Building the toolchain

After the toolchain is configured, it can be built using the following command:

./ct-ng build -j`nproc`
Bash

This build process will take a long time. Therefore, it should not be done in a (misconfigured) VM, but on a pc with lots of cores and/or threads. This can reduce the build time from a day to a few hours (maybe even less).

Checking if the toolchain works

After crosstool-NG is done creating a toolchain, a folder called x-tools can be found in the home-directory of the current user (/home/$USER/x-tools). Inside the x-tools directory, all cross-compiling toolchains can be found. To test if the new toolchain is the right version, the following command is executed inside /home/$USER/x-tools/aarch64-rpi4-linux-uclibc/bin:

toolchain-gcc -v
# or, if the "Tuple's alias" option was not set when configuring the toolchain:
aarch64-rpi4-linux-uclibc-gcc -v
Bash

The executed binary states that it is version 10.3.0, just like the version that was specified in the menuconfig of crosstool-NG. For now, it is assumed that the binary will cross-compile without errors.

Creating a basic Buildroot OS

Downloading Buildroot

To get started, Buildroot can be downloaded from the Buildroot website. There are two download options to choose from; “LTS” or “stable”. For newer devices, the stable option is a good option. For this blog, the stable .tar.gz archive is used.

Extracting

To extract the .tar.gz file, the following command can be executed:

tar -xvf buildroot-<VERSION_GOES_HERE>.tar.gz
Bash

Initial setup

To view the target devices Buildroot can build systems for, the following command can be executed:

make list-defconfigs
Bash

To configure Buildroot to build for the Raspberry Pi 4 (64 bit), the following command can be issued from the Buildroot directory:

make raspberrypi4_64_defconfig
Bash

To then configure the system, run the following command:

Configuring the target system

make menuconfig
Bash

For this post, the following changed were made to the configuration:

  • Under Build options, the option Enable compiler cache is enabled
  • Under Bootloaders, all the options are disabled (the bootloader will be compiled manually)
  • Under Filesystem images, the option ext2/3/4 root filesystem is disabled
  • Under Filesystem images, the option tar the root filesystem is enabled, along with the Compression method set to gzip
  • Under Toolchain, the option Toolchain type is set to External toolchain (this post uses a toolchain built by crosstool-NG)
  • Under Toolchain, the option Toolchain is set to Custom toolchain
  • Under Toolchain, the option Toolchain origin is set to Pre-installed toolchain
  • Under Toolchain, the option Toolchain path is set to /home/<YOUR_USERNAME_GOES_HERE>/x-tools/aarch64-rpi4-linux-uclibc (the path has to be absolute and may not contain ~/ or /home/$USER/!).
  • Under Toolchain, the option Toolchain prefix is set to aarch64-rpi4-linux-uclibc
  • Under Toolchain, the option External toolchain gcc version is set to 10.x (this version has to match the version of the cross-compiling toolchain)
  • Under Toolchain, the option External toolchain kernel headers series is set to 5.15.x or later (this version has to match the version of the cross-compiling toolchain)
  • Under Toolchain, the option Toolchain has locale support? is enabled
  • Under Toolchain, the option Toolchain has C++ support? is enabled
  • Under Toolchain, the option Toolchain has SSP support? is enabled
  • Under System configuration, the option System hostname is set to embedded
  • Under System configuration, the option System banner is set to Welcome to embedded OS!
  • Under System configuration, the option Root password is set to root (for testing test builds)
  • Under System configuration, the option /dev management is set to Dynamic using devtmpfs + mdev (to load drivers automatically when the target device boots)

Most linux desktops use udev to manage device drivers, but because this system uses BusyBox as init system, so mdev is used. To enable mdev at boot, the file buildroot-<VERSION_GOES_HERE>/board/raspberrypi/post-build.sh needs the following extra lines at the bottom, leaving the rest of the file as is:

cp package/busybox/S10mdev ${TARGET_DIR}/etc/init.d/S10mdev
chmod 755 ${TARGET_DIR}/etc/init.d/S10mdev
cp package/busybox/mdev.conf ${TARGET_DIR}/etc/mdev.conf
Bash

Building

To build the embedded OS, the following command can be executed:

make clean && make
Bash

The reason for the make clean part is that Buildroot remembers a lot, and will sometimes refuse to build (correctly) because of a cached configuration.

After compiling

When buildroot is done compiling, a file called rootfs.tar.gz can be found in the output/images folder. This is the entire embedded OS, minus the bootloader. The bootloader will be compiled manually.

The same output/images folder also contains a linux image called Image. This image needs to be converted into a U-boot image (uImage)

In addition to the rootfs.tar.gz file being present, the folder output/images will also contain a folder called rpi-firmware.
The files, start4.elf and fixup4.dat are needed later. These firmware files are files needed to boot the Raspberry Pi 4.

Converting the linux Image into a uImage

Buildroot created a file called Image. However, the U-boot bootloader (which wil be built in the next chapter) cannot boot this image file as it is. In order for U-boot to boot the kernel, the image file has to be converted to a U-boot image (uImage).

To convert the linux image into a uImage, the package u-boot-tools (or uboot-tools) is required. Then the Image file can be converted using the following command:

mkimage -n 'Linux kernel' -A arm64 -O linux -C none -T kernel -a 0x00080000 -e 0x00080000 -d arch/arm64/boot/Image uImage
Bash

A file called uImage will now be created. This file contains the entire linux kernel binary, including some extra header stuff that u-boot-tools added. This file is the kernel which runs the system.

Adding a bootloader (U-boot)

Although Buildroot can include the U-boot bootloader (among others) by enabling some settings, it might be nice to go a little more in depth to learn how U-boot actually works.

Downloading U-boot

The U-boot bootloader can be downloaded from github.

git clone https://github.com/u-boot/u-boot
cd u-boot/
Bash

Configuring the U-boot source

To see which default configurations U-boot has, execute the following command:

ls configs
# or, to only see raspberry pi entries
ls configs | grep rpi
Bash

To create a basic configuration for the Raspberry Pi 4, the following command can be executed:

ARCH=arm64 CROSS_COMPILE=~/x-tools/aarch64-rpi4-linux-uclibc/bin/aarch64-rpi4-linux-uclibc- make rpi_4_defconfig
Bash

To then further configure the source, the make menuconfig command can be given:

ARCH=arm64 CROSS_COMPILE=~/x-tools/aarch64-rpi4-linux-uclibc/bin/aarch64-rpi4-linux-uclibc- make menuconfig
Bash

In this post, the autoboot timer is changed from 2 to 0 for production-builds. This option can be changed in the following region:

Boot options ->
    Autoboot options ->
        delay in seconds before automatically booting
Bash

For test builds, it is recommended to leave this value as is.

Compiling U-boot

ARCH=arm64 CROSS_COMPILE=~/x-tools/aarch64-rpi4-linux-uclibc/bin/aarch64-rpi4-linux-uclibc- make -j`nproc`
Bash

The compile process will not take long. After the compilation is complete, a file called u-boot.bin (the U-boot binary) will exist.

Creating Raspberry Pi boot configuration

Normally, the Raspberry Pi 4 boots a file called kernel.img (sometimes kernel<version number here>.img). In this post, the Raspberry Pi 4 should start the U-boot bootloader instead of the kernel. This can be done by creating a file called config.txt with the following content:

# Enable UART communication
enable_uart=1

# Enable 64 bit mode
arm_64bit=1

# Make the raspberry Pi 4 start U-boot (u-boot.bin) instead of the default kernel8.img
kernel=u-boot.bin

# If the bluetooth device interferes with any uart/serial messages, uncomment this
#dtoverlay=miniuart-bt

# Uncomment to go fast
#arm_boost=1

# Uncomment to enable DRM VC4 V3D driver
#dtoverlay=vc4-kms-v3d
#max_framebuffers=2

# Uncomment to disable overscan
#disable_overscan=1
Bash

Creating a U-boot boot script

By default, when the U-boot bootloader starts up, it will look for a file called boot.scr. The boot.scr file is a boot script. This script will be used to load the kernel (uImage) and device tree (bcm2711-rpi-4-b.dtb) into memory and boot the system.

To create a bootscript, a file called boot.txt is created. This file will be converted to boot.scr and contains the following:

sleep 1 # some SD cards are slow and will fail to list files in time

echo +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
echo +-----------------------------------------------------------+
echo +----------------Embedded OS is starting!-------------------+
echo +-----------------------------------------------------------+
echo +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

echo ++++ Setting kernel parameters ++++++++++++++++++++++++++++++
setenv bootargs 'dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 kgdboc=ttyAMA0,115200 root=/dev/mmcblk0p2 rootwait'

echo ++++ Loading kernel into memory +++++++++++++++++++++++++++++
load mmc 0 $kernel_addr_r uImage

echo ++++ Loading device tree into memory ++++++++++++++++++++++++
load mmc 0 $fdt_addr_r bcm2711-rpi-4-b.dtb

echo ++++ Starting system ++++++++++++++++++++++++++++++++++++++++
bootm $kernel_addr_r - $fdt_addr_r
Bash

The boot.txt file is then converted to boot.scr using the following command:

mkimage -A arm64 -O linux -T script -C none -a 0 -e 0 -n "bootscript" -d boot.txt boot.scr
Bash

Creating a system image

Creating an image file

To create partitions, the program parted will be used. There will be two partitions: a boot partition called BOOT and a root partition called ROOTFS. The boot partition will have a fat16 filesystem and the root filesystem will use the ext4 filesystem.

To prevent disks from being destroyed, a virtual disk image will be created using dd:

# create a file called `system.img` with a size of 600MiB
dd if=/dev/zero of=system.img bs=1MiB count=600 status=progress
Bash

Then, this virtual disk image will be partitioned using parted:

parted system.img
Bash

Within parted, run the mklabel command and set the partition table type to msdos:

(parted) mklabel msdos
Bash

Then create the boot and root partitions:

(parted) mkpart primary fat16 2048s 30MiB
(parted) mkpart primary ext4 30MiB 100%
(parted) set 1 boot on
(parted) print
Bash

When done, exit parted by entering quit:

(parted) quit
Bash

Partitioning the system image

First, mount the virtual disk image as follows:

# Create a variable for the mount drive path
SYSTEM_IMAGE=`sudo losetup -Pf system.img --show`
Bash

Normally, disk devices in linux have paths like /dev/sda, /dev/sdb, /dev/mmcblk0, etc. This virtual device will have the path /dev/loop0. Multiple virtual devices can be mounted. The next mounted virtual disks will have the path /dev/loop1, /dev/loop2, /dev/loop3, etc.

In this post, the path to the virtual disk is stored in the SYSTEM_IMAGE variable. To view the path to the virtual disk device, run the following command:

echo $SYSTEM_IMAGE
Bash

Then, partition it using the following command:

# Create actual partitions for boot and rootfs now
sudo mkfs.vfat -n BOOT `echo "${SYSTEM_IMAGE}p1"`   # Create a fat16 partition for boot files
sudo mkfs.ext4 -L ROOTFS `echo "${SYSTEM_IMAGE}p2"` # Create an ext4 partition for rootfs files
Bash

Mounting the partitions

To mount the system image, use the following commands to mount the virtual disk image to a folder called target_mnt:

mkdir ./target_mnt
sudo mount `echo "${SYSTEM_IMAGE}p2"` ./target_mnt
sudo mkdir ./target_mnt/boot
sudo mount `echo "${SYSTEM_IMAGE}p1"` ./target_mnt/boot
Bash

Copying boot files

First, all required firmware files will be copied from the buildroot output folder to the boot folder:

sudo cp ./buildroot-<VERSION_GOES_HERE>/output/images/bcm2711-rpi-4-b.dtb ./target_mnt/boot/
sudo cp ./buildroot-<VERSION_GOES_HERE>/output/images/rpi-firmware/start4.elf ./target_mnt/boot/
sudo cp ./buildroot-<VERSION_GOES_HERE>/output/images/rpi-firmware/fixup4.dat ./target_mnt/boot/
Bash

Then, the kernel is copied to the boot folder:

sudo cp ./buildroot-<VERSION_GOES_HERE>/output/images/uImage ./target_mnt/boot/
Bash

After that, the bootloader and related configurations are copied to the boot folder:

# Copy the bootloader
sudo cp ./u-boot/u-boot.bin ./target_mnt/boot/

# Copy config.txt, to the Raspberry Pi boots U-boot
sudo cp ./u-boot/config.txt ./target_mnt/boot/

# Copy boot.scr, this bootscript starts the embedded OS
sudo cp ./u-boot/boot.scr ./target_mnt/boot/
Bash

Extracting the root filesystem to the ROOTFS partition

cd ./target_mnt
sudo tar -xvf ../buildroot-<VERSION_GOES_HERE>/output/images/rootfs.tar.gz .
cd ../
Bash

Unmounting the image

sudo sync
sudo umount ./target_mnt/boot -l
sudo umount ./target_mnt -l
sudo losetup -D
sudo rm -rf ./target_mnt
Bash

Flashing the virtual disk image to an SD

The system.img file (which now contains the entire embedded OS) can be flashed to an SD card using dd as follows:

sudo dd if=system.img of=/dev/YOUR_DEVICE_GOES_HERE bs=1MiB status=progress
sudo sync
Bash

Since system.img is an image file the flashing process is exactly the same as all other “Raspberry Pi Linux distro’s”.
Therefore, if dd seems too scary, the following tools can be used to flash the system.img file:
Raspberry Pi Imager,
Etcher,
GNOME Disks or
Win32 Disk Imager.

Sources