Stop Hardcoding DB Passwords: HashiCorp Vault Dynamic Credentials in Production

by

You have a .env file sitting in your repo. Or worse — a hardcoded connection string in a Kubernetes secret that hasn’t changed since 2021. Your DBA knows the password. Your CI/CD pipeline knows the password. That one contractor who left eight months ago probably still knows the password.

This is the norm in most shops, and it’s a quiet disaster waiting to happen. Static credentials are the single most common vector for database breaches — not because developers are careless, but because the tooling to do it right used to be painful. HashiCorp Vault’s database secrets engine removes that excuse entirely.

The concept is simple: instead of your app reading a password from an environment variable, it asks Vault for a credential at startup. Vault creates a real database user on the fly, hands it over with a TTL of (say) one hour, and when that TTL expires the user is dropped. No rotation scripts. No shared passwords. No leaked credential that persists after an incident.

This guide walks through the full production setup — Vault HA is out of scope here, but every decision below is made with production in mind.

Official repo: https://github.com/hashicorp/vault

Why Static Credentials Are a Structural Problem

The real issue isn’t that passwords get leaked — it’s that once they do, you often don’t know. A static credential sitting in your app’s config could have been exfiltrated six months ago. You have no way to tell.

Dynamic credentials change the threat model. The blast radius of a leaked credential is bounded by its TTL. If someone grabs a credential that expires in an hour, they have an hour. Compare that to a static password you rotate quarterly — at best.

The secondary benefit: every credential issued by Vault is tied to a lease. You can see exactly which app instances have active leases, revoke them all at once in an incident, and never have to coordinate a "go rotate the password and tell all the teams" fire drill.

The Architecture

Before touching configs, understand what’s happening at the network level:

[App Instance] ---> [Vault Agent / SDK] ---> [Vault Server]
                                                    |
                                           [Database Secrets Engine]
                                                    |
                                            [PostgreSQL / MySQL]

Vault talks directly to your database with an admin-level account (the "root" credential — more on this below). When your app requests a credential, Vault runs a CREATE ROLE statement, hands the username and password to your app, and tracks the lease. When the lease expires or is revoked, Vault runs a DROP ROLE.

Your app never sees the admin credential. Your app’s credential never lives beyond its TTL. This is the model.

Step 1: Run Vault (Docker Compose for Dev, Then Production Notes)

Start with a working local setup before you touch production.

# docker-compose.yml
version: "3.9"

services:
  vault:
    image: hashicorp/vault:1.17
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "dev-root-token"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    command: server -dev

  postgres:
    image: postgres:16
    container_name: postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: supersecret
      POSTGRES_DB: appdb
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

Dev mode stores everything in memory — fine for learning, fatal for production. The production config comes later.

Step 2: Enable the Database Secrets Engine

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='dev-root-token'

# Enable the engine at the default path
vault secrets enable database

Now configure the PostgreSQL connection. This is the admin credential Vault will use to manage roles:

vault write database/config/appdb \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-readonly,app-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/appdb?sslmode=disable" \
  username="postgres" \
  password="supersecret"

Gotcha #1: The {{username}} and {{password}} template syntax is not a typo. Vault uses it so it can rotate the root credential itself — which you should do immediately after setup (see Step 5). If you hardcode the credentials directly into the URL string, Vault can’t rotate them, and you’re back to square one.

Step 3: Define Roles — The SQL That Creates Your App Users

A Vault database role defines what SQL runs when a credential is requested. You write the CREATE ROLE statement; Vault handles the naming and lifecycle.

# Read-only role — good for reporting services, read replicas, etc.
vault write database/roles/app-readonly \
  db_name=appdb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Read-write role for your main application
vault write database/roles/app-readwrite \
  db_name=appdb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\"; \
    GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Gotcha #2: GRANT ON ALL TABLES only covers tables that exist at role-creation time. If you run migrations and create new tables, existing dynamic users won’t have access to them. Fix this with ALTER DEFAULT PRIVILEGES:

-- Run this once as the Vault admin user after initial setup
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO PUBLIC;

ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA public
  GRANT USAGE, SELECT ON SEQUENCES TO PUBLIC;

Step 4: Vault Policies — Lock Down Who Can Request What

A credential is only as safe as the policy guarding it. Create a policy that allows apps to request credentials but nothing else:

# app-policy.hcl
path "database/creds/app-readwrite" {
  capabilities = ["read"]
}

# Allow the app to renew its own lease
path "sys/leases/renew" {
  capabilities = ["update"]
}

# Allow the app to look up its own token
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

vault policy write app-policy app-policy.hcl

Now create a token tied to this policy (or use AppRole auth — see Step 7):

vault token create \
  -policy="app-policy" \
  -ttl="768h" \
  -renewable=true

Step 5: Rotate the Root Credential Immediately

After configuring the database connection, rotate Vault’s own admin credential so it’s no longer the one you typed in:

vault write -force database/rotate-root/appdb

After this, the supersecret password is gone — Vault regenerated it internally and you can’t retrieve it. This means your ops team can no longer log into the database with that password. That’s the point. If you need emergency access, use a break-glass procedure with a separately managed admin account.

Gotcha #3: If you rotate the root and then try to reconfigure the database connection with the old password, it won’t work. Keep your Vault storage backed up, or keep a separate break-glass Postgres superuser that Vault doesn’t manage.

Step 6: Test Credential Issuance

# Request a credential
vault read database/creds/app-readwrite

You’ll see something like:

Key                Value
---                -----
lease_id           database/creds/app-readwrite/AbCdEfGhIjKlMnOp
lease_duration     1h
lease_renewable    true
password           A1B2-C3D4-E5F6-G7H8
username           v-token-app-read-AbCdEfGh

Log into Postgres and verify the user exists:

psql -h localhost -U postgres -c "\du" appdb

Wait an hour (or reduce the TTL for testing) and the user disappears automatically. That’s it. That’s the whole magic.

Step 7: AppRole Auth — The Right Way for Production Apps

Tokens with TTLs are a step up, but AppRole is the proper mechanism for machine-to-machine auth. An app gets a role_id (like a username, non-secret) and a secret_id (like a password, short-lived). It exchanges these for a Vault token.

# Enable AppRole
vault auth enable approle

# Create a role for your app
vault write auth/approle/role/myapp \
  secret_id_ttl=10m \
  token_ttl=1h \
  token_max_ttl=4h \
  token_policies="app-policy"

# Get the role_id — this is static and non-secret, bake it into your image or config
vault read auth/approle/role/myapp/role-id

# Generate a secret_id — this is sensitive, inject it at deploy time
vault write -f auth/approle/role/myapp/secret-id

Your app’s startup sequence:

# Python example using hvac
import hvac

client = hvac.Client(url='http://vault:8200')

# Authenticate with AppRole
client.auth.approle.login(
    role_id=os.environ['VAULT_ROLE_ID'],
    secret_id=os.environ['VAULT_SECRET_ID'],
)

# Request database credentials
creds = client.secrets.database.generate_credentials(name='app-readwrite')
db_user = creds['data']['username']
db_pass = creds['data']['password']
lease_id = creds['lease_id']
lease_duration = creds['lease_duration']

# Connect to database
conn = psycopg2.connect(
    host='postgres',
    database='appdb',
    user=db_user,
    password=db_pass
)

Step 8: Vault Agent — The Lazy (Smart) Integration Path

Writing Vault integration into every app is tedious. Vault Agent is a sidecar process that handles auth, credential fetching, and renewal. It writes credentials to a file or environment — your app reads from the file and doesn’t know Vault exists.

# vault-agent.hcl
vault {
  address = "http://vault:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/etc/vault/role_id"
      secret_id_file_path = "/etc/vault/secret_id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}

template {
  source      = "/etc/vault/db.ctmpl"
  destination = "/run/secrets/db.env"
  perms       = "0600"
  # Reload the app when credentials change
  command     = "kill -HUP $(cat /var/run/app.pid)"
}

The template file (db.ctmpl) uses Consul Template syntax:

{{ with secret "database/creds/app-readwrite" }}
DB_USERNAME={{ .Data.username }}
DB_PASSWORD={{ .Data.password }}
{{ end }}

Vault Agent writes a fresh db.env file before the old credential expires and signals your app to reload its database connection. Zero downtime, no app changes needed.

Production Hardening Checklist

TLS everywhere. Vault without TLS in production is not a real setup. Use a proper cert — Let’s Encrypt works fine, or issue from your internal CA. Set VAULT_CACERT in all client environments.

Audit logging. Enable the file audit backend on day one:

vault audit enable file file_path=/var/log/vault/audit.log

Every request and response is logged. This is how you answer "which service accessed the database between 14:00 and 14:30 on Tuesday." Without this, you’re flying blind.

Seal/Unseal strategy. In production, use auto-unseal with a cloud KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault) or an HSM. Manual unseal with Shamir keys is operationally painful at 3 AM during an incident.

Lease TTLs. One hour is a reasonable default. Don’t go shorter than 15 minutes — the overhead of credential churn on your database (lots of CREATE/DROP ROLE statements) is real. Don’t go longer than 24 hours, or you lose the point of dynamic credentials.

Monitor lease expiry. If your app crashes and restarts, Vault Agent handles credential renewal. But if your app holds a database connection pool and the underlying credential expires without pool reconnection, you’ll get auth errors mid-request. Make sure your connection pool validates connections before use (pool_pre_ping=True in SQLAlchemy, testOnBorrow in HikariCP).

Gotcha #4: Vault’s lease expiry and PostgreSQL’s VALID UNTIL are slightly different clocks. Vault will attempt to revoke the user at lease expiration, but if revocation fails (Vault can’t reach Postgres), the user lingers until someone cleans up. Always set VALID UNTIL in your creation statement to match or slightly exceed the max_ttl. This is your safety net.

Gotcha #5: max_ttl is a hard ceiling per credential. Even if you renew the lease repeatedly, you can’t push it past max_ttl. Your app must handle credential rotation — request a new one, update the pool, drop the old connection gracefully. If you set max_ttl to 24 hours and your app doesn’t handle rotation, every app instance will fail exactly 24 hours after startup. Set a reminder, then fix the code.

Kubernetes Integration

If you’re on Kubernetes, Kubernetes auth is cleaner than AppRole. Vault validates the pod’s service account token against the Kubernetes API.

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=production \
  policies=app-policy \
  ttl=1h

Then use the Vault Secrets Operator (VSO) or the agent injector annotations on your pod:

# Pod annotation approach
annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "myapp"
  vault.hashicorp.com/agent-inject-secret-db: "database/creds/app-readwrite"
  vault.hashicorp.com/agent-inject-template-db: |
    {{- with secret "database/creds/app-readwrite" -}}
    DB_USERNAME={{ .Data.username }}
    DB_PASSWORD={{ .Data.password }}
    {{- end -}}

The Vault Agent sidecar is injected automatically, credentials land in /vault/secrets/db, your container reads them. No Vault SDK code in your app at all.

The Part Most Guides Skip: Breaking Glass

You’ve rotated the root credential. Now Vault is down. Your app instances are running with credentials that expire in 30 minutes. What do you do?

Keep a break-glass Postgres account — a separate superuser that Vault doesn’t manage — documented in a sealed envelope (literally or metaphorically). It has a different username, a strong random password stored in a hardware-backed password manager by two keyholders, and it’s only used when Vault is completely unavailable.

Also: Vault HA with three nodes across availability zones is not optional for production. A single Vault node going down during credential expiry is a full outage. The Raft integrated storage backend makes multi-node Vault deployment much simpler than the Consul-backed approach was.

What You’ve Actually Built

After following this, your database authentication looks like this:

  • Every app instance has a unique credential tied to its Vault token
  • No credential lives beyond 24 hours, most live under an hour
  • You can revoke all credentials for a specific service instantly with vault lease revoke -prefix database/creds/app-readwrite/
  • Every credential issuance is logged with timestamp, source IP, and Vault token that requested it
  • Your team doesn’t know the database password and doesn’t need to

That last point matters more than people expect. When developers don’t know the password, it can’t be in their shell history, their notes, their Slack messages, or their personal dotfiles repo. The attack surface shrinks not because you locked them out, but because there’s genuinely nothing to steal.

Static credentials are a solved problem. The tooling is mature, the client libraries exist for every language, and the operational overhead of running Vault is lower than the overhead of your next credential rotation fire drill.

Leave a comment

👁 Views: 2,289 · Unique visitors: 1,646