Also posted on medium
When running services on a Linux system, there is the issue of how to pass in secrets that the service needs in a secure way. Usually this is done by creating a config file with credentials and then protecting this with file system permissions and also possible a Linux security module such as Apparmour or SELinux. This still leaves the issue that the credentials are stored in a plain text file, which can be compromised.
Systemd has started to support a credentials concept for securely acquiring and passing credentials data to systems and services. The way this works is through new systemd service settings.
LoadCredential loads in credentials from a plain text file or a unix socket. The first variant is not much different from the old concept, but the second gives us a lot of possibilities which we will explore in this article.
LoadCredentialEncrypted is similar to LoadCredential, but the file it reads credentials from is encrypted with one of three options. Option 1 is to encrypt it using a system wide credential created automatically by systemd and stored in /var/lib/systemd/credential.secret The second option is to encrypt using the tpm chip if available and the third is a combination of the two.
If the server does not have tpm, either a physical or a virtual tpm, the LoadCredentialEncrypted still have many of the same challenges as the old way, but at least the risk is reduced to one file and not spread out among many config files on the system.
LoadCredential can read from a unix socket instead of a file. The way this works, is that the systemd daemon, ie pid 1, connects to the unix socket and passes the name of the service and name of the requested credential file as metadata in the socket connection. It then reads back data from the socket, which is then populated into the credential file and made available to the service. By using a systemd socket to activate a service on the unix socket, the actual credentials server can run with no privileges and the only process on the system allowed to connect to the socket is the systemd daemon itself. This makes for very interesting possibilities, of which I will describe one possible use with a credential server getting secrets from Hashicorp Vault.
LoadCredential was introduced in systemd 247, while support for encrypted credentials was introduced in systemd 250. This means that if you are running Ubuntu, you need minimum 22.10 for full support for this feature and what this article describes. If you are using Fedora, there is as of this writing still some missing pieces in the SELinux policy preventing the socket based mode to work.
The setup and configuration of Hashicorp Vault is outside the scope of this article, but for the rest of the article I will assume that we have a Vault instance running somewhere and that key value version 1 is mounted at the path secret.
For each server, you will need to create an Approle with a policy like this:
# replace SERVER_FQDN with the servers fqdn name
path "secret/servers/SERVER_FQDN/*"
{
capabilities = ["read", "list"]
}
# repeat these two for each app the servers runs
path "auth/approle/role/APPNAME/role-id"
{
capabilities = ["read"]
}
path "auth/approle/role/APPNAME/secret-id"
{
capabilities = ["create", "update"]
# require that the secret-id must be restricted to a list of CIDRs
required_parameters = ["cidr_list"]
# require the secret-id to be wrapped
min_wrapping_ttl = "1s"
max_wrapping_ttl = "5m"
}
And create an approle for the server using
vault auth/approle/role/SERVER_FQDN policies=SERVER_FQDN
For each application that we run on the server, we can create application secrets in the path secret/servers/SERVER_FQDN/APPNAME and we also allow for the possibility to create a new approle secret-id for an approle name APPNAME, which can then be passed in to a vault agent.
In order for the server to run the credentials application, we need to create a secret id for the server approle, and save the role-id and secret-id on the server. How these are transferred to the server could be a topic for another article, but I would recommend to look at wrapping to reduce risk. For now, we can set this up by running as root on the server:
mkdir -p /etc/vault-server
read -p "role-id:" role_id
echo -n "${role_id}" > /etc/vault-server/role-id
chmod 444 /etc/vault-server/role-id
read -p "secret-id:" secret_id
systemd-creds encrypt --name=secret-id - /etc/vault-server/secret-id <<<$secret_id
chmod 400 /etc/vault-server/secret-id
read -p "URL to VAULT, for example https://vault.example.com:8200:" vault_addr
echo "VAULT_ADDR=${vault_addr}" > /etc/vault-server/env
The secret-id is now stored encrypted in /etc/vault-server/secret-id and can be loaded into a service using LoadCredentialEncrypted=secret-id:/etc/vault-server/secret-id If the server has a tpm, it will be encrypted using that.
We create a socket and service for our credential server. First the socket, /etc/systemd/system/vault-credential-server.socket
[Unit]
Description=socket for vault credential server
[Socket]
ListenStream=/run/vault-credentials.socket
SocketUser=root
SocketGroup=root
SocketMode=0600
[Install]
WantedBy=sockets.target
And the service /etc/systemd/system/vault-credential-server.service
[Unit]
Description=vault credential server for systemd units
After=network-online.target
Requires=network-online.target
ConditionPathExists=/etc/vault-server/role-id
ConditionPathExists=/etc/vault-server/secret-id
[Service]
Type=simple
LoadCredential=role-id:/etc/vault-server/role-id
LoadCredentialEncrypted=secret-id:/etc/vault-server/secret-id
EnvironmentFile=/etc/vault-server/env
ExecStart=/usr/local/bin/vault-credential-server.py
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full
The socket should be started and enabled
systemctl daemon-reload
systemctl start vault-credential-server.socket
systemctl enable vault-credential-server.socket
The credentials server itself, it requires python-systemd and aiohttp python packages, to be put in /usr/local/bin/vault-credential-server.py
#!/usr/bin/env python3
import aiohttp
import asyncio
import ipaddress
import logging
from os import environ
from pathlib import Path
import socket
import sys
from systemd import journal
import systemd.daemon
log = logging.getLogger("vault-credential-server")
log.propagate = False
log.addHandler(journal.JournalHandler(SYSLOG_IDENTIFIER="vault-credential-server"))
log.setLevel(logging.INFO)
class VaultCredentialServer:
def __init__(self) -> None:
if "VAULT_ADDR" not in environ:
raise Exception("No VAULT_ADDR set")
if "CREDENTIALS_DIRECTORY" not in environ:
raise Exception("No CREDENTIALS_DIRECTORY set")
socket_fds = systemd.daemon.listen_fds()
if len(socket_fds) == 0:
log.info("No sockets passed, exiting")
sys.exit(0)
elif len(socket_fds) > 1:
log.warn("More than 1 socket passed, not supported")
sys.exit(1)
self.socket = socket.socket(fileno=socket_fds[0])
def client_connected_cb(self, reader, writer):
asyncio.create_task(self.handle_connection(reader, writer))
async def handle_connection(self, reader, writer):
_, _, service, credential = (
writer.get_extra_info("peername").decode("utf-8").split("/")
)
log.info("Got connection from %s, credential %s", service, credential)
try:
service, _ = service.split(".")
if service.startswith("vault-agent@"):
_, service = service.split("@")
if credential == "role-id":
value = await self._get_vault_approle_id(service)
elif credential == "secret-id":
value = await self._get_vault_approle_credential(service)
else:
value = await self._get_vault_server_secret(service, credential)
writer.write(str(value).encode("utf-8"))
await writer.drain()
except Exception as e:
log.exception(e)
finally:
writer.close()
await writer.wait_closed()
async def run(self):
ttl = await self._vault_login()
server = await asyncio.start_unix_server(
self.client_connected_cb, sock=self.socket
)
async with server:
await server.start_serving()
# wait until 70% of vault token ttl has gone, then exit
await asyncio.sleep(int(ttl * 0.7))
server.close()
await server.wait_closed()
async def _vault_login(self):
role_id, secret_id = self._get_vault_credentials()
self.vault_session = aiohttp.ClientSession(base_url=environ["VAULT_ADDR"])
async with self.vault_session.post(
"/v1/auth/approle/login", json={"role_id": role_id, "secret_id": secret_id}
) as resp:
if not resp.ok:
raise Exception("Unable to log in to vault %s", await resp.text())
payload = await resp.json()
self.vault_session.headers["X-Vault-Token"] = payload["auth"][
"client_token"
]
return payload["auth"].get("lease_duration", 30)
async def _get_vault_approle_credential(self, approle):
addresses = []
for family, type, proto, canonname, sockaddr in socket.getaddrinfo(
socket.getfqdn(), 8200
):
if not (ip := ipaddress.ip_address(sockaddr[0])).is_private:
addresses.append(ip)
if len(addresses) == 0:
ip_session = aiohttp.ClientSession()
async with ip_session.get("https://api.ipify.org") as resp:
public_ip = await resp.text()
addresses.append(ipaddress.ip_address(public_ip.strip()))
cidr_list = []
for addr in addresses:
if isinstance(addr, ipaddress.IPv4Address):
net = ipaddress.IPv4Network(addr)
cidr_list.append(net.with_prefixlen)
else:
net = ipaddress.IPv6Network(addr).supernet(64)
cidr_list.append(net.with_prefixlen)
path = f"/v1/auth/approle/role/{approle}/secret-id"
headers = {"X-Vault-Wrap-TTL": "5m"}
async with self.vault_session.post(
path, json={"cidr_list": cidr_list}, headers=headers
) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting approle token")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
return data["wrap_info"]["token"]
async def _get_vault_approle_id(self, approle):
path = f"/v1/auth/approle/role/{approle}/role-id"
async with self.vault_session.get(path) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting approle id")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
return data["data"]["role_id"]
async def _get_vault_server_secret(self, app, credential):
path = f"/v1/secret/servers/{socket.getfqdn()}/{app}"
async with self.vault_session.get(path) as resp:
if not resp.ok:
if resp.status == 403:
raise Exception("Permission denied getting server secret")
else:
raise Exception("Unable to get vault data: %s", await resp.text())
data = await resp.json()
if not credential in data["data"]:
raise Exception(f"{credential} not found in server data for app {app}")
return data["data"][credential]
def _get_vault_credentials(self):
credential_path = Path(environ["CREDENTIALS_DIRECTORY"])
role_id_file = credential_path / Path("role-id")
if not role_id_file.exists():
raise Exception("No role-id file")
secret_id_file = credential_path / Path("secret-id")
if not secret_id_file.exists():
raise Exception("No secret-id file")
with role_id_file.open("r") as f:
role_id = f.read()
with secret_id_file.open("r") as f:
secret_id = f.read()
return (role_id, secret_id)
if __name__ == "__main__":
vault_credential_server = VaultCredentialServer()
try:
asyncio.run(vault_credential_server.run())
except Exception as e:
log.exception(e)
sys.exit(2)
Remember to run chmod 755 /usr/local/bin/vault-credential-server.py
We can then in a systemd service use LoadCredential=credential:/run/vault-credentials.socket where credential one of three possible values
role-id— will get the role-id of the approle with the same name as the calling systemd servicesecret-id— will create and return a new secret-id of the approle with the same name as the calling systemd service- any other name will lookup the name in the keys stored at the key value secret
secret/servers/SERVER_FQDN/APPNAMEwhereAPPNAMEis the calling systemd service.
What happens when a service using the credential server is started is the following
- If the vault-credential-server service is not running, systemd starts it and passes in the unix socket file descriptor for
/run/vault-credentials.socket - When starting, the vault-credential-server will read the role-id and secret-id from
CREDENTIALS_DIRECTORYwhich is set by systemd to point to the temporary memory directory holding decrypted credentials passed to the service. It will then try to log in to Vault using these credentials - The service will read the connection metadata from the unix socket and parse the requested credential and service name (the handle_connection function) and return the credential on the socket
- The service will sleep for up to 70% of the time to live on the token we got from Vault when logging in. This saves us from doing a new startup and login to vault each time, which will save time when a service requests multiple credentials, or we start several services at the same time or shortly after each other.
To show how this is used, lets create a demo app called example. First, lets create the following secret in Vault at secret/servers/SERVER_FQDN/example
{
"foo": "bar"
}
We can then create the demo app as a simple embedded shell script in the server. Add in /etc/systemd/system/example.service
[Unit]
Description=example app
Requires=vault-credential-server.socket
[Service]
Type=oneshot
LoadCredential=foo:/run/vault-credentials.socket
ExecStart=/bin/bash -c 'echo "foo=$(cat ${CREDENTIALS_DIRECTORY}/foo)"'
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full
Run
systemctl-daemon-reload
systemctl start example
systemctl status example
and you should see it printing foo=bar like this:
Jun 08 17:05:22 medium systemd[1]: Starting example app...
Jun 08 17:05:22 medium vault-credential-server[771]: Got connection from example.service, credential foo
Jun 08 17:05:22 medium bash[783]: foo=bar
Jun 08 17:05:22 medium systemd[1]: example.service: Deactivated successfully.
Jun 08 17:05:22 medium systemd[1]: Finished example app.
Jun 08 17:05:22 medium systemd[1]: run-credentials-example.service.mount: Deactivated successfully.
The final part will show the usage of the application approle part of the credential server. We will first create a vault-agent template service that can be used by the application service to get and maintain secrets and credentials, for instance to renew certificates and dynamic credentials. We start by creating a new template service in /etc/systemd/system/vault-agent@.service (%i will be the part after @ so for vault-agent@example.service it will be example )
[Unit]
Description=vault agent for %i
Requires=vault-credential-service.socket
# This service will only be started by the %i.service, this
# makes this service automatically stop when %i.service is stopped
StopWhenUnneeded=yes
[Service]
# run the vault agent as the same user and group as the app
User=%i
Group=%i
# Get role-id and secret-id from the credentials server
LoadCredential=role-id:/run/vault-credentials.socket
LoadCredential=secret-id:/run/vault-credentials.socket
EnvironmentFile=/etc/vault-server/env
ExecStart=/bin/sh -c "cd ${CREDENTIALS_DIRECTORY}; exec /usr/bin/vault agent -config /etc/%i/vault-agent.hcl"
ExecReload=kill -HUP $MAINPID
# Let systemd configure the configuration directory, ie in /etc
ConfigurationDirectory=%i
# let systemd set up the runtime directory, ie in /run
RuntimeDirectory=%i
RuntimeDirectoryMode=0750
NoNewPrivileges=yes
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
LimitMEMLOCK=infinity
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
AmbientCapabilities=CAP_IPC_LOCK
Restart=always
RestartSec=5
We can create a new vault policy for our example app
path "secret/applications/example"
{
capabilities = ["read", "list"]
}
And let us write a secret to secret/applications/example in vault
{
"foo": "vault agent bar"
}
And create the approle using
vault write auth/approle/role/example policies=example
Remember to update the policy in vault for the server to allow access to the example app
path "auth/approle/role/APPNAME/role-id"
{
capabilities = ["read"]
}
path "auth/approle/role/example/secret-id"
{
capabilities = ["create", "update"]
# require that the secret-id must be restricted to a list of CIDRs
required_parameters = ["cidr_list"]
# require the secret-id to be wrapped
min_wrapping_ttl = "1s"
max_wrapping_ttl = "5m"
}
We create a vault agent config for the example app in /etc/example/vault-agent.hcl
auto_auth {
method {
type = "approle"
config = {
role_id_file_path = "role-id"
secret_id_file_path = "secret-id"
secret_id_response_wrapping_path = "auth/approle/role/example/secret-id"
# credential directory is read-only
remove_secret_id_file_after_reading = false
}
}
}
template_config {
exit_on_retry_failure = true
}
template {
contents = "{{- with secret \"secret/applications/example\" }}{{ .Data.foo }}{{- end }}"
destination = "/run/example/secret"
perms = "0600"
}
and modify our example service (the sleep is to make sure the vault agent has finished writing the file)
[Unit]
Description=example app
# Makes sure that vault-agent@example.service is started first
After=vault-agent@%i.service
# Similar to requires, but also makes sure that when this service stops,
# vault-agent@example.service
# also stops (see man systemd.unit) for more info
BindsTo=vault-agent@%i.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c 'sleep 5; echo -n "secret is "; cat /run/example/secret'
DynamicUser=yes
NoNewPrivileges=yes
ProtectSystem=full
If we now run the example service (systemctl start example), it should print secret is vault agent bar
Jun 08 18:14:11 medium systemd[1]: Starting example app...
Jun 08 18:14:11 medium systemd[1]: Started vault agent.
Jun 08 18:14:11 medium vault-credential-server[2661]: Got connection from vault-agent@example.service, credential role-id
Jun 08 18:14:11 medium vault-credential-server[2661]: Got connection from vault-agent@example.service, credential secret-id
Jun 08 18:14:12 medium vault-credential-server.py[2661]: Unclosed client session
Jun 08 18:14:12 medium vault-credential-server.py[2661]: client_session: <aiohttp.client.ClientSession object at 0x7f332bee6b00>
Jun 08 18:14:12 medium vault-credential-server.py[2661]: Unclosed connector
Jun 08 18:14:12 medium vault-credential-server.py[2661]: connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f332befebc0>, 4212.850028107)]']
Jun 08 18:14:12 medium vault-credential-server.py[2661]: connector: <aiohttp.connector.TCPConnector object at 0x7f332bee7370>
Jun 08 18:14:12 medium sh[2787]: ==> Vault Agent started! Log data will stream in below:
Jun 08 18:14:12 medium sh[2787]: ==> Vault Agent configuration:
Jun 08 18:14:12 medium sh[2787]: Api Address 1: http://bufconn
Jun 08 18:14:12 medium sh[2787]: Cgo: disabled
Jun 08 18:14:12 medium sh[2787]: Log Level:
Jun 08 18:14:12 medium sh[2787]: Version: Vault v1.13.2, built 2023-04-25T13:02:50Z
Jun 08 18:14:12 medium sh[2787]: Version Sha: b9b773f1628260423e6cc9745531fd903cae853f
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.986Z [INFO] agent.template.server: starting template server
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.986Z [INFO] (runner) creating new runner (dry: false, once: false)
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.987Z [INFO] (runner) creating watcher
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.987Z [INFO] agent.auth.handler: starting auth handler
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.988Z [INFO] agent.auth.handler: authenticating
Jun 08 18:14:12 medium sh[2787]: 2023-06-08T18:14:12.988Z [INFO] agent.sink.server: starting sink server
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.223Z [INFO] agent.auth.handler: authentication successful, sending token to sinks
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.224Z [INFO] agent.auth.handler: starting renewal process
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] agent.template.server: template server received new token
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] (runner) stopping
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.225Z [INFO] (runner) creating new runner (dry: false, once: false)
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.226Z [INFO] (runner) creating watcher
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.226Z [INFO] (runner) starting
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.292Z [INFO] agent.auth.handler: renewed auth token
Jun 08 18:14:13 medium sh[2787]: 2023-06-08T18:14:13.336Z [INFO] (runner) rendered "(dynamic)" => "/run/example/secret"
Jun 08 18:14:16 medium bash[2786]: secret is vault agent bar
Jun 08 18:14:16 medium systemd[1]: example.service: Deactivated successfully.
Jun 08 18:14:16 medium systemd[1]: Finished example app.
Jun 08 18:14:16 medium sh[2787]: ==> Vault Agent shutdown triggered
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.556Z [INFO] (runner) stopping
Jun 08 18:14:16 medium systemd[1]: Stopping vault agent...
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.template.server: template server stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.auth.handler: shutdown triggered, stopping lifetime watcher
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.558Z [INFO] agent.auth.handler: auth handler stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.559Z [INFO] agent.sink.server: sink server stopped
Jun 08 18:14:16 medium sh[2787]: 2023-06-08T18:14:16.559Z [INFO] agent: sinks finished, exiting
Jun 08 18:14:16 medium systemd[1]: vault-agent@example.service: Deactivated successfully.
Jun 08 18:14:16 medium systemd[1]: Stopped vault agent.
Jun 08 18:14:16 medium systemd[1]: run-credentials-vault\x2dagent\x40example.service.mount: Deactivated successfully.
I hope this gave you some new insights and ideas in how to use systemd credentials and vault. When you add on this to use dynamic credentials, for instance dynamic database credentials, you get a really powerful and secure setup for your applications. For some ideas, if you let the vault agent write an environment file that can be sourced by the application. If your application runs in a container, you can pass the environment file in through podman using the env-file option. The vault template is a go template, so it’s easy to create all sorts of config files.
.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.