Isolate service secrets and verify rotation with systemd credentials

When API services, synchronization jobs, and monitoring collectors run on the same dedicated server, service tokens often end up scattered across startup scripts, environment files, and deployment records. Process argument inspection, error logs, or a configuration archive can widen that exposure. This article uses systemd's LoadCredential on Debian 12 to keep secret source files in an administrator-controlled directory, deliver runtime copies only to the designated service, and verify rotation through repeatable checks.

1. Define the boundaries of credential isolation

The problem addressed here is how a service obtains a secret—not how secrets are issued, stored centrally, or used in remote authentication protocols. Users manage the operating system and data on their CloudDDR dedicated server. After payment, view the connection credentials in the console and access the system through SSH. The following steps require administrator privileges. Complete them with a test service first, without changing how production services start.

Environment variables are not a suitable default carrier for secrets: they can appear in debug output, child process environments, or diagnostic archives. Startup arguments can also be captured by process inspection tools. With LoadCredential, the system service manager reads the source file and provides a credentials directory when the service starts. The program locates the file through CREDENTIALS_DIRECTORY.

Credential file isolation cannot protect against a process that already has administrator privileges, nor can it prevent an application from writing secrets to logs. The protection boundary must cover file permissions, the service account, and the application's output behavior.

In this example, the source file remains plaintext. Using LoadCredential does not automatically encrypt it at rest. Full-server backups, administrator access, and records of secret issuance must also be handled as sensitive data.

2. Create a dedicated account and restrict the source file

Prepare an isolated test environment

Check systemctl --version. This example uses the systemd version provided by Debian 12 and requires Python 3. Make sure the account name is not already used by an existing service; run the account creation command below only if the account does not exist.

sudo apt-get update
sudo apt-get install python3
id credential-check
sudo useradd --system --no-create-home --shell /usr/sbin/nologin credential-check
sudo install -d -o root -g root -m 0700 /etc/service-secrets
sudo sh -c 'umask 077; printf "%s" "fixture-token-alpha" > /etc/service-secrets/api-token'
sudo stat -c '%U %G %a %n' /etc/service-secrets /etc/service-secrets/api-token

Run useradd only after id reports that the account does not exist. If it already exists, check what it is used for rather than reusing an unfamiliar account. The expected permissions are 700 for the directory and 600 for the file, with root as both owner and group. The example string is test data that grants no access, so it can safely appear in command history. Never put a real token directly into a command this way.

Obtain the real value through a controlled file handoff, then install it at the destination with administrator privileges. Do not transfer secrets through editor temporary files, terminal echo, or a shell with tracing enabled. The service account should be unable to read the source file:

sudo -u credential-check test -r /etc/service-secrets/api-token
printf '%s\n' "$?"

The expected exit code is 1. If it is 0, investigate directory permissions, file ACLs, and the account's group memberships before proceeding with deployment.

3. Make the program read only runtime credentials

Save the following program as /usr/local/libexec/credential-check.py, creating the parent directory, and set its owner to root and its permissions to 755. Neither the directory nor the program must be writable by the service account. The program reports only fixed error messages. It does not print the token, its length, or a digest, so the validation process does not create another disclosure channel.

import os
from pathlib import Path

directory = os.environ.get("CREDENTIALS_DIRECTORY")
if not directory:
    raise SystemExit("Credential directory missing")

try:
    token = (Path(directory) / "api-token").read_bytes()
except OSError:
    raise SystemExit("Credential unavailable") from None

if not token or b"\n" in token or b"\r" in token:
    raise SystemExit("Credential format rejected")

This example requires a nonempty token with no line breaks. That is not a universal rule for all secret formats. A real program should validate against the upstream protocol. For example, this check cannot be reused when multiline PEM content is allowed.

Create /etc/systemd/system/credential-check.service:

[Unit]
Description=Service credential validation

[Service]
Type=oneshot
User=credential-check
Group=credential-check
LoadCredential=api-token:/etc/service-secrets/api-token
ExecStart=/usr/bin/python3 /usr/local/libexec/credential-check.py
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
UMask=0077

This is a oneshot validation unit, so an inactive state after successful completion is normal. It does not need to be enabled at boot, and the test should not inherit a production service's restart policy. A real application can read the file in the same way, but its filesystem restrictions must be checked against the directories it needs to write to. Do not copy them blindly.

4. Verify readability and authentication separately

sudo systemctl daemon-reload
sudo systemctl start credential-check.service
sudo systemctl show credential-check.service -p Result -p ExecMainStatus
sudo journalctl -u credential-check.service -n 20 --no-pager

The expected results are Result=success and ExecMainStatus=0. These show that the service account can read the credential through the runtime directory, while the earlier check established that it cannot read the source file directly. Do not use cat, a dump of all environment variables, or a process memory dump to prove this.

This check does not establish that the token is valid. After integrating with the real service, also make an authentication request that does not change business data, and verify the response status and expected permissions. If authentication fails, check the credential format, whether the upstream service has activated the new token, whether the application actually restarted, and whether the request used an old connection or cached state.

Validation records should contain only the unit version, the sequence of operations, exit codes, and the authentication outcome—not token values. If a record must be linked to a particular rotation, use an internal change identifier rather than a token digest. When reviewing logs, also confirm that error paths do not print request headers or the full configuration.

5. Rotate with an atomic replacement and test recovery

Updating the source file does not change a credential copy that is already in use. The rotation sequence should be: obtain the new value, validate its format, replace the source atomically, restart the service, verify authentication, and finally revoke the old value according to the upstream service's rules. Running only daemon-reload does not refresh secrets used by a running service.

The following commands rotate test data only. The staging file and destination file are in the same directory so that the rename takes place within the same filesystem.

sudo sh -c 'umask 077; printf "%s" "fixture-token-beta" > /etc/service-secrets/api-token.next'
sudo chown root:root /etc/service-secrets/api-token.next
sudo chmod 0600 /etc/service-secrets/api-token.next
sudo mv -f /etc/service-secrets/api-token.next /etc/service-secrets/api-token
sudo systemctl restart credential-check.service
sudo systemctl show credential-check.service -p Result -p ExecMainStatus

Next, test a failure scenario: replace the test source file with an empty file and start the validation unit. Expect a nonzero exit code and the fixed format-error message. Restore a nonempty test value and start it again; it should succeed. This confirms that the application does not silently continue when the secret is missing. Do not run destructive tests against production tokens.

Before rotating a real token, confirm whether the upstream service can accept both the old and new tokens at the same time. If not, plan a coordinated switchover. If the old value has already been revoked, restoring the old file will not restore authentication. Retain the old value only for a controlled rollback window, protect it with the same permissions, and do not leave long-lived copies in application directories.

After validation, deactivate and remove the test unit, script, and test credentials, run daemon-reload, and confirm that no staging files remain. The final deliverable should be a verifiable procedure: an ordinary account cannot read the source file, the target service can read its runtime credential, an invalid value causes an explicit failure, authentication succeeds after rotation, and logs contain no sensitive values.

Frequently asked questions