Secrets Management at Scale: Beyond Environment Variables

The Secrets Problem

Every application needs secrets: database passwords, API keys, encryption keys, certificates. How you manage these secrets often determines whether a breach is a minor incident or a catastrophic failure.

Common anti-patterns that lead to breaches:

Secrets in source code (found by attackers scanning GitHub)

Environment variables in container definitions (visible in orchestrator UIs)

Shared credentials across environments (dev leak exposes production)

Long-lived static credentials (compromised once, valid forever)

Manual rotation (which means never rotated in practice)

The Secret Management Hierarchy

Level 1: Encrypted Files

Better than plaintext. Tools like SOPS or git-crypt encrypt secrets in your repo. Still has versioning and distribution challenges.

Level 2: Cloud Provider Secrets

AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. Good integration with cloud services, automatic rotation for some secret types, but vendor lock-in.

Level 3: Dedicated Secret Managers

HashiCorp Vault, CyberArk Conjur. Full-featured platforms with dynamic secrets, fine-grained policies, comprehensive audit logs, and multi-cloud support.

Dynamic Secrets: The Game Changer

Static secrets are shared, long-lived, and hard to revoke. Dynamic secrets are generated on-demand, unique per client, and automatically expire.

Example: Instead of one shared database password, Vault creates a unique database user for each application instance. Credentials expire after an hour. If compromised, the blast radius is one instance for one hour, not your entire database forever.

This pattern works for databases, cloud provider credentials, PKI certificates, SSH keys, and more.

Rotation Strategies

Automated rotation is essential. Manual rotation means rotation never happens until after a breach.

Zero-downtime rotation pattern:

Generate new credential

Update application configuration

Verify new credential works

Revoke old credential

For database credentials, this often means temporary dual-credential support in your connection logic.

Audit and Access Control

Every secret access should be logged: who accessed what, when, from where. This audit trail is invaluable for incident investigation and compliance.

Implement fine-grained policies: applications should only access secrets they need. Use separate policies for development, staging, and production. Require approval workflows for highly sensitive secrets.

Best Practices

Never hardcode: Secrets belong in secret managers, not code

Rotate automatically: Manual rotation is no rotation

Use dynamic secrets: Unique, short-lived credentials per instance

Audit everything: Log all access for investigation and compliance

Encrypt at rest: Defense in depth for stored secrets

Separate by environment: Dev should never access prod secrets

Test your recovery: Can you rotate all secrets in an emergency?