Håvards page

Securely Sending Systemd Credentials to Podman Containers

Jan 5, 2025

Also posted on medium

Systemd got the ability to securely pass credentials to services a while ago, as I have previously written about. Podman has long had the ability to run kubernetes pod and deployment manifests and with podman-systemd.unit it’s easy to automatically generate systemd services to run these. With these two building blocks, we have the tools to pass credentials to container applications, but the question is how to do so easily and securely.

One option would be to mount the systemd credentials directory as a volume mount, but it presents some issues:

Systemd sends the path to the credentials directory as an environment variable, CREDENTIALS_DIRECTORY which is not possible to reference in the pod or deployment manifest. Even though in practice this seems to be in a predictable path, it can change and thus cause our service to fail if we hardcode the path.
If our application does not have support for reading credentials from file, we need to either build that into the application, create a wrapper to start the application or we could use an init container to read the credentials and add to a config file that the application reads.

What would be a better approach is to map the credentials to environment variables, as reading from environment variables is the normal practise for container applications. A possible approach could be to run an ExecStartPre command to read the credentials and create a podman secret. There is one issue with that approach, and that is that by default, podman secret uses the file driver, which creates a file on disk for the secret. This defeats the purpose of using systemd credentials and so we need another approach. You can see this by running (as a regular user):

echo baz | podman secret create foo -
less .local/share/containers/storage/secrets/filedriver/secretsdata.json

As can be seen in the podman-secret-create man page, podman has three build in drivers, file, pass and shell. We can use the shell driver to create in-memory secrets that can be passed on to the application using kubernetes secrets.

We will need two scripts to implement this. The first script will be run by ExecStartPre and creates the podman secret if it is not already created. Create /usr/local/bin/podman-create-secret.sh with executable permissions and the following content:

#!/bin/bash

app="$1"

if ! podman secret ls --format '{{.Name}}' | grep -E -q "^credentials-${app}$"
then
  echo "${netns}" | podman secret create -d shell --driver-opts="list=/usr/local/bin/podman-secret.py list,lookup=/usr/local/bin/podman-secret.py lookup,delete=/usr/local/bin/podman-secret.py delete,store=/usr/local/bin/podman-secret.py store" "credentials-${app}" -
fi

We will then need the actual script that podman will call. This will be called by the podman process started by ExecStart, so it will have access to the CREDENTIALS_DIRECTORY environment variable, as well as the credentials. Create /usr/local/bin/podman-secret.py with executable permission and the following content:

#!/usr/bin/env python3

import argparse
from base64 import b64decode, b64encode
import binascii
import json
from os import environ
from pathlib import Path


def parse_json_secret(data: dict) -> dict:
    parsed_data = {}
    for k, v in data.items():
        if isinstance(v, str):
            # Test if value is already base64 encoded as required by
            # kubernetes secret, otherwise encode it
            try:
                b64decode(v)
                parsed_data[k] = v
            except binascii.Error:
                parsed_data[k] = b64encode(v.encode("utf-8")).decode("ascii")
        else:
            parsed_data[k] = b64encode(json.dumps(v).encode("utf-8")).decode("ascii")
    return parsed_data


def get_credentials_secrets():
    secrets = {}
    if "CREDENTIALS_DIRECTORY" not in environ:
        return secrets
    for file in Path(environ["CREDENTIALS_DIRECTORY"]).iterdir():
        with file.open() as f:
            if file.name.endswith(".json"):
                secrets.update(parse_json_secret(json.load(f)))
            else:
                # kubernetes secrets must have base64 encoded values
                secrets[file.name] = b64encode(f.read().encode("utf-8")).decode("ascii")
    return secrets


def lookup_secret():
    secrets = get_credentials_secrets()
    secret = {
        "apiVersion": "v1",
        "kind": "Secret",
        "type": "Opaque",
        "data": secrets,
    }
    print(json.dumps(secret))


def podman_secret():
    parser = argparse.ArgumentParser()
    parser.add_argument("action", choices=["delete", "list", "lookup", "store"])
    args = parser.parse_args()

    match args.action:
        case "delete":
            return
        case "list":
            return
        case "store":
            return
        case "lookup":
            lookup_secret()


if __name__ == "__main__":
    podman_secret()

The script supports two types of secrets. If the filename of the credential ends with .json, the script will parse it as json and add it directly to the secret. Otherwise, it will add the content of the credential file with the key being the name of the file.

To test and show how to use this, let us first create two secrets (if you do not have the new run0 command which came with systemd 256 in June 2024, replace run0 with sudo):

echo -n bar | run0 systemd-creds encrypt --name=foo - /etc/test.creds
echo '{"TEST_SECRET": "very secret", "TEST_SECRET_COMPLEX": ["a", "b", "c"]}' | run0 systemd-creds encrypt --name=test.json - /etc/test2.creds

We then need a pod manifest, create /tmp/test.yaml with the following content

---
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
    - name: test
      image: busybox
      args:
        - /bin/sh
        - -c
        - sleep infinity
      envFrom:
        - secretRef:
            name: credentials-test
      securityContext:
        readOnlyRootFilesystem: true
  restartPolicy: Never
  securityContext:
    # run as nobody instead of root
    runAsUser: 65384
    runAsGroup: 65384

And the .kube file needs to be created in /etc/containers/systemd/test.kube with the following content

[Unit]
Description=Test credentials

[Service]
LoadCredentialEncrypted=foo:/etc/test.creds
LoadCredentialEncrypted=test.json:/etc/test2.creds
ExecStartPre=/usr/local/bin/podman-create-secret.sh test

[Kube]
Yaml=/tmp/test.yaml
# It is good security practise to run in a user namespace
# This requires that containers has been added to
# /etc/subgid and /etc/subuid as described in
# man podman-kube-play
# This can be done by running:
# echo containers:2147483647:2147483648 | run0 tee -a /etc/subgid
# echo containers:2147483647:2147483648 | run0 tee -a /etc/subuid
UserNS=auto:uidmapping=65384:65384:65385,gidmapping=65384:65384:65385

You can now run systemctl daemon-reload and test.service will be created:

$ systemctl daemon-reload
$ systemctl cat test.service
# /run/systemd/generator/test.service
# Automatically generated by /usr/lib/systemd/system-generators/podman-system-generator
#
[Unit]
Wants=network-online.target
After=network-online.target
Description=Test credentials
SourcePath=/etc/containers/systemd/test.kube
RequiresMountsFor=%t/containers

[Service]
LoadCredentialEncrypted=foo:/etc/test.creds
LoadCredentialEncrypted=test.json:/etc/test2.creds
ExecStartPre=/usr/local/bin/podman-create-secret.sh test
KillMode=mixed
Environment=PODMAN_SYSTEMD_UNIT=%n
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/bin/podman kube play --replace --service-container=true --userns auto:uidmapping=65384:65384:65385,gidmapping=65384:65384:65385 /tmp/test.yaml
ExecStopPost=/usr/bin/podman kube down /tmp/test.yaml

[X-Kube]
Yaml=/tmp/test.yaml
# It is good security practise to run in a user namespace
# This requires that containers has been added to
# /etc/subgid and /etc/subuid as described in
# man podman-kube-play
# Example:
# echo containers:2147483647:2147483648 | run0 tee -a /etc/subgid
# echo containers:2147483647:2147483648 | run0 tee -a /etc/subuid
UserNS=auto:uidmapping=65384:65384:65385,gidmapping=65384:65384:65385

We can now start the container and verify that it works:

$ systemctl start test.service
$ run0 podman exec test-test printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
TEST_SECRET=very secret
TEST_SECRET_COMPLEX=["a", "b", "c"]
foo=bar
HOME=/root

We see that the environment variables foo, TEST_SECRET and TEST_SECRET_COMPLEX has all been set to the values we gave to systemd-creds encrypt.