Debian on Blog

Creating a bootable USB stick that can still be used as normal storage

Tue, 10 Dec 2024 15:53:11 +0100

As someone who daily drives rolling release operating systems, having a bootable USB stick with a live install of Debian is essential. It has saved reverting broken packages and making my device bootable again at least a couple of times. However, creating a bootable USB stick usually uses the entire storage of the USB stick, which seems unnecessary given that USB sticks easily have 64GiB or more these days while live ISOs still don’t use more than 8GiB.

In this how-to, I will explain how to create a bootable USB stick, that also holds a FAT32 partition, which allows the USB stick used as usual.

Creating the partition partble

The first step is to create the partition table on the new drive. There are several tools to do this, I recommend the ArchWiki page on this topic for details. For best compatibility, MBR should be used instead of the more modern GPT. For simplicity I just went with the GParted since it has an easy GUI, but feel free to use any other tool. The layout should look like this:

Type  │ Partition │ Suggested size
──────┼───────────┼───────────────
EFI   │ /dev/sda1 │           8GiB
FAT32 │ /dev/sda2 │      remainder

Notes:

The disk names are just an example and have to be adjusted for your system.
Don’t set disk labels, they don’t appear on the new install anyway and some UEFIs might not like it on your boot partition.
If you used GParted, create the EFI partition as FAT32 and set the esp flag.

Mounting the ISO and USB stick

The next step is to download your ISO and mount it. I personally use a Debian Live ISO for this. To mount the ISO, use the loop mounting option:

mkdir -p /tmp/mnt/iso
sudo mount -o loop /path/to/image.iso /tmp/mnt/iso

Similarily, mount your USB stick:

mkdir -p /tmp/mnt/usb
sudo mount /dev/sda1 /tmp/mnt/usb

Copy the ISO contents

To copy the contents from the ISO to the USB stick, use this command:

sudo cp -r -L /tmp/mnt/iso/. /tmp/mnt/usb

The -L option is required to deference the symbolic links present in the ISO, since FAT32 does not support symbolic links. Note that you might get warning about cyclic symbolic links. Nothing can be done to fix those, except hoping that they won’t cause a problem. For me this never was a problem though.

Finishing touches

Umnount the ISO and USB stick:

sudo umount /tmp/mnt/usb
sudo umount /tmp/mnt/iso

Now the USB stick should be bootable and have a usable data partition that can be used on essentially every operating system. I recommend testing that the stick is bootable to make sure it actually works in case of an emergency.

Setting up fast Debian package builds using sbuild, mmdebstrap and apt-cacher-ng

Wed, 08 Feb 2023 19:49:11 +0100

In this post I will give a quick tutorial on how to set up fast Debian package builds using sbuild with mmdebstrap and apt-cacher-ng.

The usual tool for building Debian packages is dpkg-buildpackage, or a user-friendly wrapper like debuild, and while these are geat tools, if you want to upload something to the Debian archive they lack the required separation from the system they are run on to ensure that your packaging also works on a different system. The usual candidate here is sbuild. But setting up a schroot is tedious and performance tuning can be annoying. There is an alternative backend for sbuild that promises to make everything simpler: unshare. In this tutorial I will show you how to set up sbuild with this backend.

Additionally to the normal performance tweaking, caching downloaded packages can be a huge performance increase when rebuilding packages. I do rebuilds quite often, mostly when a new dependency got introduced I didn’t specify in debian/control yet or lintian notices a something I can easily fix. So let’s begin with setting up this caching.

Setting up apt-cacher-ng

Install apt-cacher-ng:

sudo apt install apt-cacher-ng

A pop-up will appear, if you are unsure how to answer it select no, we don’t need it for this use-case.

To enable apt-cacher-ng on your system, create /etc/apt/apt.conf.d/02proxy and insert:

Acquire::http::proxy "http://127.0.0.1:3142";
Acquire::https::proxy "DIRECT";

In /etc/apt-cacher-ng/acng.conf you can increase the value of ExThreshold to hold packages for a shorter or longer duration. The length depends on your specific use case and resources. A longer threshold takes more disk space, a short threshold like one day effecitvely only reduces the build time for rebuilds.

If you encounter weird issues on apt update at some point the future, you can try to clean the cache from apt-cacher-ng. You can use this script:

Setting up mmdebstrap

Install mmdebstrap:

sudo apt install mmdebstrap

We will create a small helper script to ease creating a chroot. Open ~/.local/bin/mmupdate and insert:

#!/bin/sh
mmdebstrap \
  --variant=buildd \
  --aptopt='Acquire::http::proxy "http://127.0.0.1:3142";' \
  --arch=amd64 \
  --components=main,contrib,non-free \
  unstable \
  ~/.cache/sbuild/unstable-amd64.tar.xz \
  http://deb.debian.org/debian

Notes:

aptopt enables apt-cacher-ng inside the chroot.
--arch sets the CPU architecture (see Debian Wiki).
--components sets the archive components, if you don’t want non-free pacakges you might want to remove some entries here.
unstable sets the Debian release, you can also set for example bookworm-backports here.
unstable-amd64.tar.xz is the output tarball containing the chroot, change accordingly to your pick of the CPU architecture and Debian release.
http://deb.debian.org/debian is the Debian mirror, you should set this to the same one you use in your /etc.apt/sources.list.

Make mmupdate executable and run it once:

chmod +x ~/.local/bin/mmupdate
mkdir -p ~/.cache/sbuild
~/.local/bin/mmupdate

If you execute mmupdate again you can see that the downloading stage is much faster thanks to apt-cacher-ng. For me the difference is from about 115s to about 95s. Your results may vary, this depends on the speed of your internet, Debian mirror and disk.

If you have used the schroot backend and sbuild-update before, you probably notice that creating a new chroot with mmdebstrap is slower. It would be a bit annoying to do this manually before we start a new Debian packaging session, so let’s create a systemd service that does this for us.

First create a folder for user services:

mkdir -p ~/.config/systemd/user

Create ~/.config/systemd/user/mmupdate.service and add:

[Unit]
Description=Run mmupdate
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=%h/.local/bin/mmupdate

Start the service and test that it works:

systemctl --user daemon-reload
systemctl --user start mmupdate
systemctl --user status mmupdate

Create ~/.config/systemd/user/mmupdate.timer:

[Unit]
Description=Run mmupdate daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer:

systemctl --user enable mmupdate.timer

Now every day mmupdate will be run automatically. You can adjust the period if you think daily rebuilds are a bit excessive.

A neat advantage of period rebuilds is that they the base files in your apt-cacher-ng cache warm every time they run.

Setting up sbuild

Install sbuild and (optionally) autopkgtest:

sudo apt install --no-install-recommends sbuild autopkgtest

Create ~/.sbuildrc and insert:

# backend for using mmdebstrap chroots
$chroot_mode = 'unshare';

# build in tmpfs
$unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXX';

# upgrade before starting build
$apt_update = 1;
$apt_upgrade = 1;

# build everything including source for source-only uploads
$build_arch_all = 1;
$build_arch_any = 1;
$build_source = 1;
$source_only_changes = 1;

# go to shell on failure instead of exiting
$external_commands = { "build-failed-commands" => [ [ '%SBUILD_SHELL' ] ] };

# always clean build dir, even on failure
$purge_build_directory = "always";

# run lintian
$run_lintian = 1;
$lintian_opts = [ '-i', '-I', '-E', '--pedantic' ];

# do not run piuparts
$run_piuparts = 0;

# run autopkgtest
$run_autopkgtest = 1;
$autopkgtest_root_args = '';
$autopkgtest_opts = [ '--apt-upgrade', '--', 'unshare', '--release', '%r', '--arch', '%a', '--prefix=/dev/shm/tmp.autopkgtest.' ];

# set uploader for correct signing
$uploader_name = 'Stephan Lachnit <stephanlachnit@debian.org>';

You should adjust uploader_name. If you don’t want to run autopkgtest or lintian by default you can also disable it here. Note that for packages that need a lot of space for building, you might want to comment the unshare_tmpdir_template line to prevent a OOM build failure.

You can now build your Debian packages with the sbuild command :)

Finishing touches

You can add these variables to your ~/.bashrc as bonus (with adjusted name / email):

export DEBFULLNAME="<your_name>"
export DEBEMAIL="<your_email>"
export DEB_BUILD_OPTIONS="parallel=<threads>"

In particular adjust the value of parallel to ensure parallel builds.

If you are new to signing / uploading your package, first install the required tools:

sudo apt install devscripts dput-ng

Create ~/.devscripts and insert:

DEBSIGN_KEYID=<your_gpg_fingerprint>
USCAN_SYMLINK=rename

You can now sign the .changes file with:

debsign ../<pkgname_version_arch>.changes

And for source-only uploads with:

debsign -S ../<pkgname_version_arch>_source.changes

If you don’t introduce a new binary package, you always want to go with source-only changes.

You can now upload the package to Debian with

dput ../<filename>.changes

Update Feb 22: Advanced options in mmdebstrap

Jochen Sprickerhof, who originally advised me to use the unshare backend, commented that one can also use --include=auto-apt-proxy instead of the --aptopt option in mmdebstrap to detect apt proxies automatically. He also let me know that it is possible to use autopkgtest on tmpfs (config in the blog post is updated) and added an entry on the sbuild wiki page on how to setup sbuild+unshare with ccache if you often need to build a large package.

Further, using --variant=apt and --include=build-essential will produce smaller build chroots if wished. On the contrary, one can of course also use the --include option to include debhelper and lintian (or any other packages you like) to further decrease the setup time. However, staying with buildd variant is a good choice for official uploads.

Update Aug 23: Building for experimental

mmdebstrap does not allow to create tarballs with packages from experimental. Luckily, we don’t actually need this, we can simply solve this with in sbuild by adding experimental as additional repository to our existing unstable tarball and changing the dependency resolver. Let’s create a new script ~/.local/bin/sbuild-experimental:

#!/bin/sh
sbuild --dist=experimental --extra-repository='deb http://deb.debian.org/debian experimental main contrib non-free' --build-dep-resolver=aspcud $@

Let’s not forget to make the file executable:

chmod +x ~/.local/bin/sbuild-experimental

The last step is to create a symlink for the experimental tarball pointing to our existing tarball:

ln -s -r ~/.cache/sbuild/unstable-amd64.tar.xz ~/.cache/sbuild/experimental-amd64.tar.xz

Thanks to josch and smcv on IRC for helping me out with this!

Resources for further reading

https://wiki.debian.org/sbuild
https://www.unix-ag.uni-kl.de/~bloch/acng/html/index.html
https://wiki.ubuntu.com/SimpleSbuild
https://wiki.archlinux.org/title/Systemd/Timers
https://manpages.debian.org/unstable/autopkgtest/autopkgtest-virt-unshare.1.en.html
Thanks for reading!

Installing Debian on F2FS rootfs with deboostrap and systemd-boot

Tue, 07 Feb 2023 23:37:56 +0100

I recently got a new NVME drive. My plan was to create a fresh Debian install on an F2FS root partition with compression for maximum performance. As it turns out, this is not entirely trivil to accomplish. For one, the Debian installer does not support F2FS (here is my attempt to add it from 2021). And even if it did, grub does not support F2FS with the extra_attr flag that is required for compression support (at least as of grub 2.06).

Luckily, we can install Debian anyway with all these these shiny new features when we go the manual road with debootstrap and using systemd-boot as bootloader. We can break down the process into several steps:

Warning: Playing around with partitions can easily result in data if you mess up! Make sure to double check your commands and create a data backup if you don’t feel confident about the process.

Creating the partition partble

The first step is to create the GPT partition table on the new drive. There are several tools to do this, I recommend the ArchWiki page on this topic for details. For simplicity I just went with the GParted since it has an easy GUI, but feel free to use any other tool. The layout should look like this:

Type       │ Partition      │ Suggested size
───────────┼────────────────┼───────────────
EFI        │ /dev/nvme0n1p1 │           1GiB
Linux swap │ /dev/nvme0n1p2 │           1GiB
Linux fs   │ /dev/nvme0n1p3 │      remainder

Notes:

The disk names are just an example and have to be adjusted for your system.
Don’t set disk labels, they don’t appear on the new install anyway and some UEFIs might not like it on your boot partition.
The size of the EFI partition can be smaller, in practive it’s unlikely that you need more than 300 MiB. However some UEFIs might be buggy and if you ever want to install an additional kernel or something like memtest86+ you will be happy to have the extra space.
The swap partition can be omitted, it is not strictly needed. If you need more swap for some reason you can also add more using a swap file later (see ArchWiki page). If you know you want to use suspend-to-RAM, you want to increase the size to something more than the size of your memory.
If you used GParted, create the EFI partition as FAT32 and set the esp flag. For the root partition use ext4 or F2FS if available.

Creating and mounting the root partition

To create the root partition, we need to install the f2fs-tools first:

sudo apt install f2fs-tools

Now we can create the file system with the correct flags:

mkfs.f2fs -O extra_attr,inode_checksum,sb_checksum,compression,encrypt /dev/nvme0n1p3

For details on the flags visit the ArchWiki page.

Next, we need to mount the partition with the correct flags. First, create a working directory:

mkdir bootstrap
cd bootstrap
mkdir root
export DFS=$(pwd)/root

Then we can mount the partition:

sudo mount -o compress_algorithm=zstd:6,compress_chksum,gc_merge,lazytime /dev/nvme0n1p3 $DFS

Again, for details on the mount options visit the above mentioned ArchWiki page.

Bootstrapping with `debootstrap`

First we need to install the debootstrap package:

sudo apt install debootstrap

Now we can do the bootstrapping:

debootstrap --arch=amd64 --components=main,contrib,non-free,non-free-firmware unstable $DFS http://deb.debian.org/debian

Notes:

--arch sets the CPU architecture (see Debian Wiki).
--components sets the archive components, if you don’t want non-free pacakges you might want to remove some entries here.
unstable is the Debian release, you might want to change that to testing or bookworm.
$DFS points to the mounting point of the root partition.
http://deb.debian.org/debian is the Debian mirror, you might want to set that to http://ftp.de.debian.org/debian or similar if you have a fast mirror in you area.

Chrooting into the system

Before we can chroot into the newly created system, we need to prepare and mount virtual kernel file systems. First create the directories:

sudo mkdir -p $DFS/dev $DFS/dev/pts $DFS/proc $DFS/sys $DFS/run $DFS/sys/firmware/efi/efivars $DFS/boot/efi

Then bind-mount the directories from your system to the mount point of the new system:

sudo mount -v -B /dev $DFS/dev
sudo mount -v -B /dev/pts $DFS/dev/pts
sudo mount -v -B /proc $DFS/proc
sudo mount -v -B /sys $DFS/sys
sudo mount -v -B /run $DFS/run
sudo mount -v -B /sys/firmware/efi/efivars $DFS/sys/firmware/efi/efivars

As a last step, we need to mount the EFI partition:

sudo mount -v /dev/nvme0n1p1 $DFS/boot/efi

Now we can chroot into new system:

sudo chroot $DFS /bin/bash

Configure the base system

The first step in the chroot is setting the locales. We need this since we might leak the locales from our base system into the chroot and if this happens we get a lot of annoying warnings.

export LC_ALL=C.UTF-8 LANG=C.UTF-8
apt install locales console-setup

Set your locales:

dpkg-reconfigure locales

Set your keyboard layout:

dpkg-reconfigure keyboard-configuration

Set your timezone:

dpkg-reconfigure tzdata

Now you have a fully functional Debian chroot! However, it is not bootable yet, so let’s fix that.

Define static file system information

The first step is to make sure the system mounts all partitions on startup with the correct mount flags. This is done in /etc/fstab (see ArchWiki page). Open the file and change its content to:

# file system                               mount point   type   options                                                       dump   pass

# NVME efi partition
UUID=XXXX-XXXX                              /boot/efi     vfat   umask=0077                                                    0      0

# NVME swap
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   none          swap   sw                                                            0      0

# NVME main partition
UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX   /             f2fs   compress_algorithm=zstd:6,compress_chksum,gc_merge,lazytime   0      1

You need to fill in the UUIDs for the partitions. You can use

ls -lAph /dev/disk/by-uuid/

to match the UUIDs to the more readable disk name under /dev.

Installing the kernel and bootloader

First install the systemd-boot and efibootmgr packages:

apt install systemd-boot efibootmgr

Now we can install the bootloader:

bootctl install --path=/boot/efi

You can verify the procedure worked with

efibootmgr -v

The next step is to install the kernel, you can find a fitting image with:

apt search linux-image-*

In my case:

apt install linux-image-amd64

After the installation of the kernel, apt will add an entry for systemd-boot automatically. Neat!

However, since we are in a chroot the current settings are not bootable. The first reason is the boot partition, which will likely be the one from your current system. To change that, navigate to /boot/efi/loader/entries, it should contain one config file. When you open this file, it should look something like this:

title      Debian GNU/Linux bookworm/sid
version    6.1.0-3-amd64
machine-id 2967cafb6420ce7a2b99030163e2ee6a
sort-key   debian
options    root=PARTUUID=f81d4fae-7dec-11d0-a765-00a0c91e6bf6 ro systemd.machine_id=2967cafb6420ce7a2b99030163e2ee6a
linux      /2967cafb6420ce7a2b99030163e2ee6a/6.1.0-3-amd64/linux
initrd     /2967cafb6420ce7a2b99030163e2ee6a/6.1.0-3-amd64/initrd.img-6.1.0-3-amd64

The PARTUUID needs to point to the partition equivalent to /dev/nvme0n1p3 on your system. You can use

ls -lAph /dev/disk/by-partuuid/

to match the PARTUUIDs to the more readable disk name under /dev.

The second problem is the ro flag in options which tell the kernel to boot in read-only mode. The default is rw, so you can just remove the ro flag.

Once this is fixed, the new system should be bootable. You can change the boot order with:

efibootmgr --bootorder

However, before we reboot we might add well add a user and install some basic software.

Finishing touches

Add a user:

useradd -m -G sudo -s /usr/bin/bash -c 'Full Name' username

Debian provides a TUI to install Desktop Environment. To open it, run:

tasksel

Now you can finally reboot into your new system:

reboot

Resources for further reading

https://ivanb.neocities.org/blogs/y2022/debootstrap
https://www.debian.org/releases/stable/amd64/apds03.en.html
https://www.addictivetips.com/ubuntu-linux-tips/set-up-systemd-boot-on-arch-linux/
https://p5r.uk/blog/2020/using-systemd-boot-on-debian-bullseye.html
https://www.linuxfromscratch.org/lfs/view/stable/chapter07/kernfs.html
Thanks for reading!

MiniDebConfOnline #2: Gaming utilities and Debian

Sat, 21 Nov 2020 13:30:00 +0000

After DebConf20 I didn’t plan to take part in any MiniDebConf soon, but when the topic for the MDCO got announced (“Gaming”) it was clear to me that I had to give talk about my current work in Debian, which mostly consists of packaging gaming utilities.

Compared to my last talk, I think I did much better. I was a bit better prepared and didn’t had to rush to send in the video before the deadline like with my last talk. From what I saw on IRC, there have been a couple of people chatting about the tools in the talk.

tl;dw: I gave Lutris, MangoHud, vkBasalt, GOverlay and Piper a quick video preview and also briefly presented gamemode, vkdevicechooser, OpenRGB, Oversteer, GameHub, NoiseTorch and vkmark. I also talked a bit about some minor things that annoy me when packaging gaming tools.

DebConf20: Simplifying copyright using SPDX IDs

Sat, 29 Aug 2020 15:30:00 +0000

I’ve been contributing for Debian since late 2019, I submitted a talk to the DebConf20. One of the topics that annoyed me the most at that time was how tedious it was to write and read larger d/copright files.

So I’ve submitted a talk about the SPDX standard, hoping to start a discussion to get rid of the license texts inside the file. IMHO not a very good talk, but also not bad one either for my first public talk.

tl;dw: there never was a noteworthy discussion, but thanks to some hints on IRC I’ve discovered REUSE from the FSFE. It basically already implements what I had in mind, and I will surely write up a blog post about it at some point.

Debian on Blog

Creating a bootable USB stick that can still be used as normal storage

Creating the partition partble

Mounting the ISO and USB stick

Copy the ISO contents

Finishing touches

Setting up fast Debian package builds using sbuild, mmdebstrap and apt-cacher-ng

Setting up apt-cacher-ng

Setting up mmdebstrap

Setting up sbuild

Finishing touches

Update Feb 22: Advanced options in mmdebstrap

Update Aug 23: Building for experimental

Resources for further reading

Installing Debian on F2FS rootfs with deboostrap and systemd-boot

Creating the partition partble

Creating and mounting the root partition

Bootstrapping with debootstrap

Chrooting into the system

Configure the base system

Define static file system information

Installing the kernel and bootloader

Finishing touches

Resources for further reading

MiniDebConfOnline #2: Gaming utilities and Debian

DebConf20: Simplifying copyright using SPDX IDs

Bootstrapping with `debootstrap`