Building Yocto Images inside Systemd containers
If you have been working with embedded linux recently you must have heard about distribution build systems such as Buildroot and the Yocto project. Basically they allow you to create custom linux images for your bleeding-edge new device i.e. connected toaster or digital flowerpot.
This post is not going to be a Yocto tutorial. I am not going to review yocto's functionalities nor explain its build process. This is not the point. Plenty of excellent resources are available online. I will assume you know the basics and focus on setting up a build container.
One of the main issues encountered while using build systems or cross compilation is setting up and keeping a proper build environment. Yocto officially supports a few mainstream linux distributions but not always their latest releases. As an Archlinux user, I frequently experience issues when yocto's dependencies are updated.
The best solution is to have a dedicated environment such as Vagrant or containers. Building a linux distribution is a very demanding process, therefore I would choose containers over virtual machines. Yocto actually provides us with a docker based solution. Here we want something simple and permanent, something that we can reuse for future builds. I present you systemd-nspawn.
Chroot on steroids
Systemd-nspawn is actually a part of systemd. True story! But what is it exactly?
Well... It is more or less a fancy chroot system with built-in integrations with systemd. For instance systemd containers can be managed as linux services.
The main advantage over chroot is that systemd-nspwan fully virtualizes the process tree, the file system hierarchy and IPC subsystems.
In terms of isolation, the containers accesses kernel interfaces such as /sys, /proc in read-only mode. Critical parameters like system clock and network interfaces cannot be modified from within systemd containers nor can be performed critical host operations such as loading kernel modules nor rebooting.
Setting up a container
A bit of practice now.
Since our goal is build yocto linux images, it makes sense to choose a supported distribution as container base. Here I am going to use Debian stretch.
To install the base system, we will need debootstrap. This tool allow us to deploy a debian base system directly into a subdirectory of our host. Basically it accesses debian repository and download every component needed to build a rootfs. Similar tools exist for other distros, ie pacstrap for archlinux.
# Download and install debootstrap (available on community repo)
yaourt -S debootstrap
# The same you are using a debian based distro
sudo apt install debootstrap
mkdir -p containers/debian
# Specify hardware architecture, debian version and target folder
sudo debootstrap --arch=amd64 stretch containers/debian
...
I: Configuring tasksel-data...
I: Configuring libc-bin...
I: Configuring systemd...
I: Base system installed successfully.
Once your installation is finished, simply run your container with:
sudo systemd-nspawn -D containers/debian --machine=yocto
Spawning container yoct on /home/xxxxxx/containers/yocto.
Press ^] three times within 1s to kill container.
root@yocto:~#
root@yocto:~# ls /
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root@yocto:~#
Exit your container either by pressing 3 times ^] or by typing the command exit. Once in your container, the next logical step is to set the root password.
root@yocto:~# passwd
The -b or --boot option will boot the container, ie run the PID1 which means looking for an init binary and executing it. In our case systemd will be launched and a logging prompt will appear.
sudo systemd-nspawn -bD containers/debian --machine yocto
Spawning container yocto on /home/arnaud/containers/debian.
Press ^] three times within 1s to kill container.
systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.
Welcome to Debian GNU/Linux 9 (stretch)!
Set hostname to <hostname>.
Failed to install release agent, ignoring: No such file or directory
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Created slice System Slice.
[ OK ] Reached target Slices.
[ OK ] Created slice system-getty.slice.
Mounting POSIX Message Queue File System...
The container can now act as a physical machine, for example it can be shut down
using poweroff.
Accessing host directories
It is possible to share files or directories between host and container using the option --bind while starting the container.
sudo systemd-nspawn --bind /host/path/to/dir/:/container/path/to/dir --machine yocto -D containers/debian
The file or directory will be accessible in read-write mode at the given path. It is possible to bind folders in read-only mode with --bind-ro.
By default the container share the host ip address.
Many other options including for example file systems or network interfaces can be specified, see man systemd-spawn.
Yocto specifics
Once inside the container let us install the required packages.
root@yocto:~# apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat cpio python python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping libsdl1.2dev xterm sudo
# building as root is discouraged, creating a user is advised
root@yocto:~# useradd -m -g sudo -s /bin/bash builder
# Edit sudoer file
root@yocto:~# visudo
# Uncomment this line
%wheel ALL=(ALL) ALL
By default locales are not configured within the container and this might cause errors while using bitbake.
To avoid this problem it is necessary to generate them properly.
# Generating en_US.UTF-8
root@yocto:~# sudo apt-get install locales
root@yocto:~# sudo dpkg-reconfigure locales
# add the following to ~/.bashrc
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8
# don't forget to source
builder@yocto:~# source ~/.bashrc
I personally like to keep yocto cache on my host system and bind it with my container. Therefore I create specific folders on my host.
mkdir -p ~/cache/{tmp,ss,dl}
sudo systemd-nspawn --bind /home/host/cache:/home/builder/cache --machine yocto -D containers/debian
Don't forget to modify your conf/local.conf accordingly by setting the corresponding
values.
DL_DIR ?= "/home/builder/cache/dl"
TMPDIR ?= "/home/builder/cache/tmp"
SSTATE_DIR ?= "/home/builder/cache/ss"
There you go! A working building container for yocto images!
builder@yocto:~/poky/build$ bitbake core-image-minimal
Parsing recipes: 100% |########################################################| Time: 0:01:01
Parsing of 814 .bb files complete (0 cached, 814 parsed). 1281 targets, 45 skipped, 0 masked, 0 errors.
....
Accessing host tty from container
Here comes a trickier part.
Sometimes it is useful to access serial, to monitor or communicate with our device. Problem: binding /dev with our container is not sufficient.
sudo systemd-nspawn -D containers/debian --machine serial --bind /dev:/dev
...
...
root@serial:~# picocom -b 115200 /dev/ttyUSB1
FATAL: cannot open /dev/ttyUSB1: Operation not permitted
Systemd-nspawn handles permissions through cgroups. By default some devices are authorized ie: /dev/nul, /dev/zero. This is not the case for /dev/tty which permissions must be granted explicitly.
In order to do so, you will need to know the type and the subtype of the device. You can find them with the command file.
file /dev/ttyUSB0
/dev/ttyUSB0: character special (188/0)
As root on the host system, we can now grant permission for the running container serial.
echo 'c 188:0 rwm' > /sys/fs/cgroup/devices/machine.slice/machine-serial.scope/devices.allow
'c 188:0 rwm' means read write modify permission for the character device of type 188 and subtype 0.
root@serial:~# picocom -b 115200 /dev/ttyUSB0
Terminal ready
However this permission will only last while the container is running.
Note: Don't forget to add the previously created user to the dialout group if you wish this user to be authorized to access /dev/ttyUSB*.
Managing containers as services
Systemd allows us to manage containers as services using the concept of machines. Machines must be installed in a specific location: /var/lib/machines/ and can be controlled with the command machinectl.
Let's create a debian machine.
sudo debootstrap --arch=amd64 stretch /var/lib/machines/debian
# We need dbus installed inside the container to be able to use machinectl tools
# Enter the conainer
sudo systemd-nspawn -D /var/lib/machines/debian
....
# Install dbus which is not present by default in debian
root@debian:~# apt install dbus
# Set a password
root@debian:~# passwd
# Go back to the host
root@debian:~# exit
# Start the debian machine and log inside the container.
machinectl start debian
machinectl login debian
# To enable it just run
machinectl enable debian
This article is by no mean complete, this is just my basic use of systemd containers. There are plenty of other possibilities and usages.
References
https://wiki.archlinux.org/index.php/Systemd-nspawn
https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html
https://blog.selectel.com/systemd-containers-introduction-systemd-nspawn/