Using systemd credentials to pass secrets from Hashicorp Vault to systemd services

Also posted on medium

When running services on a Linux system, there is the issue of how to pass in secrets that the service needs in a secure way. Usually this is done by creating a config file with credentials and then protecting this with file system permissions and also possible a Linux security module such as Apparmour or SELinux. This still leaves the issue that the credentials are stored in a plain text file, which can be compromised.

Systemd has started to support a credentials concept for securely acquiring and passing credentials data to systems and services. The way this works is through new systemd service settings.

LoadCredential loads in credentials from a plain text file or a unix socket. The first variant is not much different from the old concept, but the second gives us a lot of possibilities which we will explore in this article.

LoadCredentialEncrypted is similar to LoadCredential, but the file it reads credentials from is encrypted with one of three options. Option 1 is to encrypt it using a system wide credential created automatically by systemd and stored in /var/lib/systemd/credential.secret The second option is to encrypt using the tpm chip if available and the third is a combination of the two.

If the server does not have tpm, either a physical or a virtual tpm, the LoadCredentialEncrypted still have many of the same challenges as the old way, but at least the risk is reduced to one file and not spread out among many config files on the system.

LoadCredential can read from a unix socket instead of a file. The way this works, is that the systemd daemon, ie pid 1, connects to the unix socket and passes the name of the service and name of the requested credential file as metadata in the socket connection. It then reads back data from the socket, which is then populated into the credential file and made available to the service. By using a systemd socket to activate a service on the unix socket, the actual credentials server can run with no privileges and the only process on the system allowed to connect to the socket is the systemd daemon itself. This makes for very interesting possibilities, of which I will describe one possible use with a credential server getting secrets from Hashicorp Vault.

LoadCredential was introduced in systemd 247, while support for encrypted credentials was introduced in systemd 250. This means that if you are running Ubuntu, you need minimum 22.10 for full support for this feature and what this article describes. If you are using Fedora, there is as of this writing still some missing pieces in the SELinux policy preventing the socket based mode to work.

The setup and configuration of Hashicorp Vault is outside the scope of this article, but for the rest of the article I will assume that we have a Vault instance running somewhere and that key value version 1 is mounted at the path secret.

For each server, you will need to create an Approle with a policy like this:

# replace SERVER_FQDN with the servers fqdn name
path "secret/servers/SERVER_FQDN/*"
{
capabilities = ["read", "list"]
}

# repeat these two for each app the servers runs
path "auth/approle/role/APPNAME/role-id"
{
capabilities = ["read"]
}
path "auth/approle/role/APPNAME/secret-id"
{
capabilities = ["create", "update"]
# require that the secret-id must be restricted to a list of CIDRs
required_parameters = ["cidr_list"]
# require the secret-id to be wrapped
min_wrapping_ttl = "1s"
max_wrapping_ttl = "5m"
}

And create an approle for the server using

vault auth/approle/role/SERVER_FQDN policies=SERVER_FQDN

For each application that we run on the server, we can create application secrets in the path secret/servers/SERVER_FQDN/APPNAME and we also allow for the possibility to create a new approle secret-id for an approle name APPNAME, which can then be passed in to a vault agent.

In order for the server to run the credentials application, we need to create a secret id for the server approle, and save the role-id and secret-id on the server. How these are transferred to the server could be a topic for another article, but I would recommend to look at wrapping to reduce risk. For now, we can set this up by running as root on the server:

mkdir -p /etc/vault-server

read -p "role-id:" role_id
echo -n "${role_id}" > /etc/vault-server/role-id
chmod 444 /etc/vault-server/role-id

read -p "secret-id:" secret_id
systemd-creds encrypt --name=secret-id - /etc/vault-server/secret-id <<<$secret_id
chmod 400 /etc/vault-server/secret-id

read -p "URL to VAULT, for example https://vault.example.com:8200:" vault_addr
echo "VAULT_ADDR=${vault_addr}" > /etc/vault-server/env

The secret-id is now stored encrypted in /etc/vault-server/secret-id and can be loaded into a service using LoadCredentialEncrypted=secret-id:/etc/vault-server/secret-id If the server has a tpm, it will be encrypted using that.

We create a socket and service for our credential server. First the socket, /etc/systemd/system/vault-credential-server.socket

[Unit]
Description=socket for vault credential server

[Socket]
ListenStream=/run/vault-credentials.socket
SocketUser=root
SocketGroup=root
SocketMode=0600

[Install]
WantedBy=sockets.target

And the service /etc/systemd/system/vault-credential-server.service

[Unit]
Description=vault credential server for systemd units
After=network-online.target
Requires=network-online.target
ConditionPathExists=/etc/vault-server/role-id
ConditionPathExists=/etc/vault-server/secret-id

[Service]
Type=simple
LoadCredential=role-id:/etc/vault-server/role-id
LoadCredentialEncrypted=secret-id:/etc/vault-server/secret-id
EnvironmentFile=/etc/vault-server/env
ExecStart=/usr/local/bin/vault-credential-server.py
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full

The socket should be started and enabled

systemctl daemon-reload
systemctl start vault-credential-server.socket
systemctl enable vault-credential-server.socket

The credentials server itself, it requires python-systemd and aiohttp python packages, to be put in /usr/local/bin/vault-credential-server.py

#!/usr/bin/env python3

import aiohttp
import asyncio
import ipaddress
import logging
from os import environ
from pathlib import Path
import socket
import sys

from systemd import journal
import systemd.daemon

log = logging.getLogger("vault-credential-server")
log.propagate = False
log.addHandler(journal.JournalHandler(SYSLOG_IDENTIFIER="vault-credential-server"))
log.setLevel(logging.INFO)


class VaultCredentialServer:
def __init__(self) -> None:
if "VAULT_ADDR" not in environ:
raise Exception("No VAULT_ADDR set")

if "CREDENTIALS_DIRECTORY" not in environ:
raise Exception("No CREDENTIALS_DIRECTORY set")

socket_fds = systemd.daemon.listen_fds()
if len(socket_fds) == 0:
log.info("No sockets passed, exiting")
sys.exit(0)
elif len(socket_fds) > 1:
log.warn("More than 1 socket passed, not supported")
sys.exit(1)
self.socket = socket.socket(fileno=socket_fds[0])

def client_connected_cb(self, reader, writer):
asyncio.create_task(self.handle_connection(reader, writer))

async def handle_connection(self, reader, writer):
_, _, service, credential = (
writer.get_extra_info("peername").decode("utf-8").split("/")
)
log.info("Got connection from %s, credential %s", service, credential)
try:
service, _ = service.split(".")
if service.startswith("vault-agent@"):
_, service = service.split("@")
if credential == "role-id":
value = await self._get_vault_approle_id(service)
elif credential == "secret-id":
value = await self._get_vault_approle_credential(service)
else:
value = await self._get_vault_server_secret(service, credential)

writer.write(str(value).encode("utf-8"))
await writer.drain()
except Exception as e:
log.exception(e)
finally:
writer.close()
await writer.wait_closed()

async def run(self):
ttl = await self._vault_login()

server = await asyncio.start_unix_server(
self.client_connected_cb, sock=self.socket
)
async with server:
await server.start_serving()
# wait until 70% of vault token ttl has gone, then exit
await asyncio.sleep(int(ttl * 0.7))
server.close()
await server.wait_closed()

async def _vault_login(self):
role_id, secret_id = self._get_vault_credentials()
self.vault_session = aiohttp.ClientSession(base_url=environ["VAULT_ADDR"])
async with self.vault_session.post(
"/v1/auth/approle/login", json={"role_id": role_id, "secret_id": secret_id}
) as resp:
if not resp.ok:
raise Exception("Unable to log in to vault %s", await resp.text())
payload = await resp.json()
self.vault_session.headers["X-Vault-Token"] = payload["auth"][
"client_token"
]
return payload["auth"].get("lease_duration", 30)

async def _get_vault_approle_credential(self, approle):
addresses = []
for family, type, proto, canonname, sockaddr in socket.getaddrinfo(
socket.getfqdn(), 8200
):
if not (ip := ipaddress.ip_address(sockaddr[0])).is_private:
addresses.append(ip)
if len(addresses) == 0:
ip_session = aiohttp.ClientSession()
async with ip_session.get("https://api.ipify.org") as resp:
public_ip = await resp.text()
addresses.append(ipaddress.ip_address(public_ip.strip()))

cidr_list = []
for addr in addresses:
if isinstance(addr, ipaddress.IPv4Address):
net = ipaddress.IPv4Network(addr)
cidr_list.append(net.with_prefixlen)
else:
net = ipaddress.IPv6Network(addr).supernet(64)
cidr_list.append(net.with_prefixlen)

path = f"/v1/auth/approle/role/{approle}/secret-id"
headers = {"X-Vault-Wrap-TTL": "5m"}

async with self.vault_session.post(
path, json={"cidr_list": cidr_list}, headers=headers
) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting approle token")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
return data["wrap_info"]["token"]

async def _get_vault_approle_id(self, approle):
path = f"/v1/auth/approle/role/{approle}/role-id"

async with self.vault_session.get(path) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting approle id")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
return data["data"]["role_id"]

async def _get_vault_server_secret(self, app, credential):
path = f"/v1/secret/servers/{socket.getfqdn()}/{app}"

async with self.vault_session.get(path) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting server secret")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
if not credential in data["data"]:
raise Exception(f"{credential} not found in server data for app {app}")
return data["data"][credential]

def _get_vault_credentials(self):
credential_path = Path(environ["CREDENTIALS_DIRECTORY"])
role_id_file = credential_path / Path("role-id")
if not role_id_file.exists():
raise Exception("No role-id file")
secret_id_file = credential_path / Path("secret-id")
if not secret_id_file.exists():
raise Exception("No secret-id file")

with role_id_file.open("r") as f:
role_id = f.read()
with secret_id_file.open("r") as f:
secret_id = f.read()

return (role_id, secret_id)


if __name__ == "__main__":
vault_credential_server = VaultCredentialServer()
try:
asyncio.run(vault_credential_server.run())
except Exception as e:
log.exception(e)
sys.exit(2)

Remember to run chmod 755 /usr/local/bin/vault-credential-server.py

We can then in a systemd service use LoadCredential=credential:/run/vault-credentials.socket where credential one of three possible values

  • role-id — will get the role-id of the approle with the same name as the calling systemd service
  • secret-id — will create and return a new secret-id of the approle with the same name as the calling systemd service
  • any other name will lookup the name in the keys stored at the key value secret secret/servers/SERVER_FQDN/APPNAME where APPNAME is the calling systemd service.

What happens when a service using the credential server is started is the following

  1. If the vault-credential-server service is not running, systemd starts it and passes in the unix socket file descriptor for /run/vault-credentials.socket
  2. When starting, the vault-credential-server will read the role-id and secret-id from CREDENTIALS_DIRECTORY which is set by systemd to point to the temporary memory directory holding decrypted credentials passed to the service. It will then try to log in to Vault using these credentials
  3. The service will read the connection metadata from the unix socket and parse the requested credential and service name (the handle_connection function) and return the credential on the socket
  4. The service will sleep for up to 70% of the time to live on the token we got from Vault when logging in. This saves us from doing a new startup and login to vault each time, which will save time when a service requests multiple credentials, or we start several services at the same time or shortly after each other.

To show how this is used, lets create a demo app called example. First, lets create the following secret in Vault at secret/servers/SERVER_FQDN/example

{
"foo": "bar"
}

We can then create the demo app as a simple embedded shell script in the server. Add in /etc/systemd/system/example.service

[Unit]
Description=example app
Requires=vault-credential-server.socket

[Service]
Type=oneshot
LoadCredential=foo:/run/vault-credentials.socket
ExecStart=/bin/bash -c 'echo "foo=$(cat ${CREDENTIALS_DIRECTORY}/foo)"'
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full

Run

systemctl-daemon-reload
systemctl start example
systemctl status example

and you should see it printing foo=bar like this:

Jun 08 17:05:22 medium systemd[1]: Starting example app...
Jun 08 17:05:22 medium vault-credential-server[771]: Got connection from example.service, credential foo
Jun 08 17:05:22 medium bash[783]: foo=bar
Jun 08 17:05:22 medium systemd[1]: example.service: Deactivated successfully.
Jun 08 17:05:22 medium systemd[1]: Finished example app.
Jun 08 17:05:22 medium systemd[1]: run-credentials-example.service.mount: Deactivated successfully.

The final part will show the usage of the application approle part of the credential server. We will first create a vault-agent template service that can be used by the application service to get and maintain secrets and credentials, for instance to renew certificates and dynamic credentials. We start by creating a new template service in /etc/systemd/system/vault-agent@.service (%i will be the part after @ so for vault-agent@example.service it will be example )

[Unit]
Description=vault agent for %i
Requires=vault-credential-service.socket
# This service will only be started by the %i.service, this
# makes this service automatically stop when %i.service is stopped
StopWhenUnneeded=yes

[Service]
# run the vault agent as the same user and group as the app
User=%i
Group=%i
# Get role-id and secret-id from the credentials server
LoadCredential=role-id:/run/vault-credentials.socket
LoadCredential=secret-id:/run/vault-credentials.socket
EnvironmentFile=/etc/vault-server/env
ExecStart=/bin/sh -c "cd ${CREDENTIALS_DIRECTORY}; exec /usr/bin/vault agent -config /etc/%i/vault-agent.hcl"
ExecReload=kill -HUP $MAINPID
# Let systemd configure the configuration directory, ie in /etc
ConfigurationDirectory=%i
# let systemd set up the runtime directory, ie in /run
RuntimeDirectory=%i
RuntimeDirectoryMode=0750
NoNewPrivileges=yes
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
LimitMEMLOCK=infinity
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
AmbientCapabilities=CAP_IPC_LOCK
Restart=always
RestartSec=5

We can create a new vault policy for our example app

path "secret/applications/example"
{
capabilities = ["read", "list"]
}

And let us write a secret to secret/applications/example in vault

{
"foo": "vault agent bar"
}

And create the approle using

vault write auth/approle/role/example policies=example

Remember to update the policy in vault for the server to allow access to the example app

path "auth/approle/role/APPNAME/role-id"
{
capabilities = ["read"]
}
path "auth/approle/role/example/secret-id"
{
capabilities = ["create", "update"]
# require that the secret-id must be restricted to a list of CIDRs
required_parameters = ["cidr_list"]
# require the secret-id to be wrapped
min_wrapping_ttl = "1s"
max_wrapping_ttl = "5m"
}

We create a vault agent config for the example app in /etc/example/vault-agent.hcl

auto_auth {
method {
type = "approle"

config = {
role_id_file_path = "role-id"
secret_id_file_path = "secret-id"
secret_id_response_wrapping_path = "auth/approle/role/example/secret-id"
# credential directory is read-only
remove_secret_id_file_after_reading = false
}
}
}

template_config {
exit_on_retry_failure = true
}

template {
contents = "{{- with secret \"secret/applications/example\" }}{{ .Data.foo }}{{- end }}"
destination = "/run/example/secret"
perms = "0600"
}

and modify our example service (the sleep is to make sure the vault agent has finished writing the file)

[Unit]
Description=example app
# Makes sure that vault-agent@example.service is started first
After=vault-agent@%i.service
# Similar to requires, but also makes sure that when this service stops,
# vault-agent@example.service
# also stops (see man systemd.unit) for more info
BindsTo=vault-agent@%i.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'sleep 5; echo -n "secret is "; cat /run/example/secret'
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full

If we now run the example service (systemctl start example), it should print secret is vault agent bar

Jun 08 18:14:11 medium systemd[1]: Starting example app...
Jun 08 18:14:11 medium systemd[1]: Started vault agent.
Jun 08 18:14:11 medium vault-credential-server[2661]: Got connection from vault-agent@example.service, credential role-id
Jun 08 18:14:11 medium vault-credential-server[2661]: Got connection from vault-agent@example.service, credential secret-id
Jun 08 18:14:12 medium vault-credential-server.py[2661]: Unclosed client session
Jun 08 18:14:12 medium vault-credential-server.py[2661]: client_session: <aiohttp.client.ClientSession object at 0x7f332bee6b00>
Jun 08 18:14:12 medium vault-credential-server.py[2661]: Unclosed connector
Jun 08 18:14:12 medium vault-credential-server.py[2661]: connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f332befebc0>, 4212.850028107)]']
Jun 08 18:14:12 medium vault-credential-server.py[2661]: connector: <aiohttp.connector.TCPConnector object at 0x7f332bee7370>
Jun 08 18:14:12 medium sh[2787]: ==> Vault Agent started! Log data will stream in below:
Jun 08 18:14:12 medium sh[2787]: ==> Vault Agent configuration:
Jun 08 18:14:12 medium sh[2787]: Api Address 1: http://bufconn
Jun 08 18:14:12 medium sh[2787]: Cgo: disabled
Jun 08 18:14:12 medium sh[2787]: Log Level:
Jun 08 18:14:12 medium sh[2787]: Version: Vault v1.13.2, built 2023-04-25T13:02:50Z
Jun 08 18:14:12 medium sh[2787]: Version Sha: b9b773f1628260423e6cc9745531fd903cae853f
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.986Z [INFO] agent.template.server: starting template server
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.986Z [INFO] (runner) creating new runner (dry: false, once: false)
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.987Z [INFO] (runner) creating watcher
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.987Z [INFO] agent.auth.handler: starting auth handler
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.988Z [INFO] agent.auth.handler: authenticating
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.988Z [INFO] agent.sink.server: starting sink server
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.223Z [INFO] agent.auth.handler: authentication successful, sending token to sinks
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.224Z [INFO] agent.auth.handler: starting renewal process
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] agent.template.server: template server received new token
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] (runner) stopping
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] (runner) creating new runner (dry: false, once: false)
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.226Z [INFO] (runner) creating watcher
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.226Z [INFO] (runner) starting
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.292Z [INFO] agent.auth.handler: renewed auth token
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.336Z [INFO] (runner) rendered "(dynamic)" => "/run/example/secret"
Jun 08 18:14:16 medium bash[2786]: secret is vault agent bar
Jun 08 18:14:16 medium systemd[1]: example.service: Deactivated successfully.
Jun 08 18:14:16 medium systemd[1]: Finished example app.
Jun 08 18:14:16 medium sh[2787]: ==> Vault Agent shutdown triggered
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.556Z [INFO] (runner) stopping
Jun 08 18:14:16 medium systemd[1]: Stopping vault agent...
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.template.server: template server stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.auth.handler: shutdown triggered, stopping lifetime watcher
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.auth.handler: auth handler stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.559Z [INFO] agent.sink.server: sink server stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.559Z [INFO] agent: sinks finished, exiting
Jun 08 18:14:16 medium systemd[1]: vault-agent@example.service: Deactivated successfully.
Jun 08 18:14:16 medium systemd[1]: Stopped vault agent.
Jun 08 18:14:16 medium systemd[1]: run-credentials-vault\x2dagent\x40example.service.mount: Deactivated successfully.

I hope this gave you some new insights and ideas in how to use systemd credentials and vault. When you add on this to use dynamic credentials, for instance dynamic database credentials, you get a really powerful and secure setup for your applications. For some ideas, if you let the vault agent write an environment file that can be sourced by the application. If your application runs in a container, you can pass the environment file in through podman using the env-file option. The vault template is a go template, so it’s easy to create all sorts of config files.

.

Full UEFI secure boot on Fedora using signed initrd and systemd-boot

Also posted on Medium.

“Security Circus” by Alexandre Dulaunoy is licensed under CC BY-SA 2.0.

Although secure boot support has been in place in most distros for a while, most, if not all, still have a missing part in the chain, namely the missing signing of the initrd image. This allows an attacker to tamper with the initrd image without detection. Lennart Pottering wrote a very good blog post about the issue and outlined a solution that can be implemented.

Until measures as outlined there or similar are implemented in the distros, there a couple of solutions ready to set up now. They do however require a little work from the end user.

Option one is to encrypt the boot partition. This works, however you need to use grub and grub still only supports luks version 1. Also you need to decrypt the boot partition using a password or passphrase, as this is the only method grub supports, not other methods such as fido2 or tpm, which is supported by systemd-cryptenroll.

The second option is the one I’m going to describe, that is to sign and verify the initrd image. That way, it does not matter that the boot partition is unencrypted, as there is no secrets in the initrd image, and any tampering will break the signature and the boot process.


In order to sign and validate the initrd image, you will first need to set up and install your own secure boot signing key. To ease setup and use, we use sbctl. First install required packages (if you don’t want to install asciidoc and go on your system, you can use toolbox)

sudo dnf install asciidoc golang

Download the sbctl release file, unpack and install (change version as needed)

VERSION=0.9
cd /tmp
curl -L "https://github.com/Foxboron/sbctl/releases/download/${VERSION}/sbctl-${VERSION}.tar.gz" | tar zxvf -
cd "sbctl-${VERSION}"
make
sudo make install

and generate the key by running

sudo sbctl create-keys

If your computer supports it, you can install the key by running

sudo sbctl enroll-keys

If successfull, reboot. If this fails, you will need to manually install the key. This can vary from system to system, but in most cases the following will work

  • Runsudo openssl x509 -in /usr/share/secureboot/keys/db/db.pem -outform DER -out /boot/efi/EFI/fedora/DB.cer
  • Reboot into BIOS. There will usually be an option under secure boot to add your own key to the DB database from file, where you can browse the EFI partition and find the DB.cer file in EFI/fedora

After adding the key, sudo sbctl status should now a checkmark for installed and secure boot, similar to

Installed:	✔ Sbctl is installed
Owner GUID: a9fbbdb7-a05f-48d5-b63a-08c5df45ee70
Setup Mode: ✔ Disabled
Secure Boot: ✔ Enabled

The next step is setting up systemd-boot and automatic signing of the initrd on kernel upgrades. First we need to change the efi mount from /boot/efi to /efi

sudo umount /boot/efi
sudo mkdir /efi
sudo sed -i `s-/boot/efi-/efi-' /etc/fstab
sudo mount /efi
sudo ln -s /efi /boot/efi

The last command is because some packages such as fwupd expects to find the efi mount at /boot/efi

Setup systemd-boot and sign it by running

sudo bootctl install
sudo sbctl sign /efi/EFI/systemd/systemd-bootx64.efi

You might want to edit /efi/loader/loader.conf and add timeout 3 or similar, in order to get a prompt to choose images when booting.

To make sure systemd-boot keeps being signed on updates, we will need the sbsigntoolspackage installed

sudo dnf install sbsigntools

Then run sudo systemctl edit systemd-boot-update.service and add the following

[Service]
ExecStart=/bin/sh -c 'sbverify --cert /usr/share/secureboot/keys/db/db.pem /efi/EFI/systemd/systemd-bootx64.efi || sbctl sign /efi/EFI/systemd/systemd-bootx64.efi'

Next, we add dracut configuration for creating a combined and signed file containing efi stub loader, kernel and initrd. Edit /etc/dracut.conf.d/local.conf and add

uefi=yes
uefi_secureboot_cert=/usr/share/secureboot/keys/db/db.pem
uefi_secureboot_key=/usr/share/secureboot/keys/db/db.key
dracut_rescue_image=no

The reason we disable the rescue image, is that this is very big and will most likely fill your EFI partition.

The last piece is to add a kernel install script to change the systemd-boot entry to load only the combined image. Edit /etc/kernel/install.d/99-use-signed-image.install with the content

#!/bin/bash
COMMAND="$1"
KERNEL_VERSION="$2"
ENTRY_DIR_ABS="$3"
if ! [[ $COMMAND == add ]]; then
exit 1
fi
MACHINE_ID="$KERNEL_INSTALL_MACHINE_ID"
BOOT_ROOT="$KERNEL_INSTALL_BOOT_ROOT"
if [[ -f /etc/kernel/tries ]]; then
read -r TRIES </etc/kernel/tries
if ! [[ "$TRIES" =~ ^[0-9]+$ ]] ; then
echo "/etc/kernel/tries does not contain an integer." >&2
exit 1
fi
LOADER_ENTRY="$BOOT_ROOT/loader/entries/$MACHINE_ID-$KERNEL_VERSION+$TRIES.conf"
else
LOADER_ENTRY="$BOOT_ROOT/loader/entries/$MACHINE_ID-$KERNEL_VERSION.conf"
fi
sed -i "/^initrd/d" "${LOADER_ENTRY}"
sed -i "/^linux/s/linux$/initrd/" "${LOADER_ENTRY}"

Make the script executable by running sudo chmod 755 /etc/kernel/install.d/99-use-signed-image.install

Everything should now be up and running, the last thing to do is re-run kernel install for the current kernel in order to get a new updated and signd initrd image by running

sudo /bin/kernel-install -v  add $(uname -r) /lib/modules/$(uname -r)/vmlinuz

Reboot and you should have a boot that is signed all the way. Run sudo bootctl status to verify. Current Boot Loader should say systemd-boot and under Default Boot Loader Entry the linux line should point to initrd (this is the combined and signed initrd).

You can now remove the grub boot entry, so that there is no way to boot using the unsigned initrd image. Run sudo efibootmgr -v to list current boot entries. Then run sudo efibootmgr -B -b XX where XX is the number, for instance 0003

Finally, edit /etc/fwupd/uefi_capsule.conf and set EnableGrubChainLoad=false

Baconday 2021

This year was our 10 year anniversary, so we had re-runs of some of the dishes from the last 10 years (with some twists). For the canapes, we this year did a contest.

Running cec-client with Raspberry PI on Debian or Ubuntu 64-bit

Using cec-client with a raspberry pi on debian or ubuntu does not work out of the box, as the cec-client in the debian and ubuntu repositories has not been compiled with support for the propriary raspberry libraries. In addition, these libraries are 32-bit only. This is a guide in how to get this working.

First you need to set up a 32-bit chroot environment. Install debootstrap and then run

debootstrap --variant=buildd --arch=armhf bullseye /opt/cec-client/

You will then need to download the raspberry pi libraries. Unpack /opt/vc from the tarball into /opt/cec-client (replace url with newer version if needed):

cd /tmp
curl -L https://github.com/raspberrypi/firmware/archive/1.20210108.tar.gz | tar zxf -
cp -r firmware*/opt/ /opt/cec-client/

Add the raspberry pi lib to ldconfig by running

echo "/opt/vc/lib" > /opt/cec-client/etc/ld.so.conf.d/rpi.conf

You can now compile libcec and cec-utils:

chroot /opt/cec-client
ldconfig
apt-get update
apt-get install cmake libudev-dev libxrandr-dev python3-dev swig git 
cd /tmp/ 
git clone https://github.com/Pulse-Eight/platform.git
mkdir platform/build
cd platform/build
cmake .. 
make 
make install
cd /tmp/
git clone https://github.com/Pulse-Eight/libcec.git
mkdir libcec/build
cd libcec/build
cmake -DRPI_INCLUDE_DIR=/opt/vc/include -DRPI_LIB_DIR=/opt/vc/lib ..
make -j4 
make install
ldconfig
cd /tmp/
mkdir libcec/src/cec-client/build/
cd libcec/src/cec-client/build/
make
make install

In order to run cec-client inside the chroot, you will also need access to devices. You can do this by mounting this up before running chroot, but an easier and better way is to use systemd. Many of the examples you will find of how to use cec-client adds the -s flag to run a single command, but without this, cec-client runs until told to quit and listens on commands on stdin. We will use this and let systemd set up a fifo for commands for us. Create the socket and service files:

[Unit]
Description=CEC client socket

[Socket]
ListenFIFO=/run/cec.fifo

[Install]
WantedBy=sockets.target
[Unit]
Description=CEC client
After=network.target

[Service]
Type=simple
Restart=no
RootDirectory=/opt/cec-client
ExecStart=/usr/local/bin/cec-client -d 1
ExecStop=/bin/bash -c "echo q > /run/cec.fifo"
StandardInput=socket
StandardOutput=journal
MountAPIVFS=yes

[Install]
WantedBy=multi-user.target

Then reload systemd, enable and start:

systemctl daemon-reload
systemctl enable cec-client.socket
systemctl start cec-client.socket

You can now send cec-client commands by writing to /run/cec.fifo, for instance to turn on the tv with address 0.0.0.0 run

echo 'on 0.0.0.0' > /run/cec.fifo