LearningTree · AWS Security

AWS Secrets Manager
Secure Credential Lifecycle

How modern cloud applications securely store, retrieve, and rotate credentials — without ever hardcoding a password. The complete architecture guide.

6 Chapters ~15 min read Architecture-Focused Exam Ready

Chapter One

What is AWS Secrets Manager

AWS Secrets Manager is a secure credential storage and lifecycle service. It stores sensitive values — database passwords, API keys, tokens, certificates — and delivers them to your applications at runtime, so you never hardcode secrets in source code.

Real-World Analogy

Secrets Manager is a hotel front desk safe.

You check in your valuables (credentials). When your application needs them, it presents its room key (IAM role) and retrieves the valuables. The safe rotates the lock combination periodically without you knowing.

AWS Translation

Safe = Secrets Manager (encrypted storage)
Valuables = passwords, API keys, tokens
Room key = IAM role/policy
Lock rotation = automatic credential rotation
Front desk log = CloudTrail audit

The Problem — Why Secrets Manager Exists Beginner

🚨

The Old Way (Dangerous)

Passwords hardcoded in source code
Credentials committed to GitHub
.env files shared via Slack
Database passwords in config files on servers
Same password used for months/years
No audit trail of who accessed what

✅

The Modern Way (Secrets Manager)

Secrets stored encrypted, centrally managed
Applications retrieve secrets at runtime via API
Access controlled by IAM policies
Automatic rotation (no human touches passwords)
Full CloudTrail audit of every access
Cross-account sharing with fine-grained control

What Secrets Manager Stores Core

🗄️

Database Credentials

RDS master passwords
Aurora cluster credentials
Redshift admin passwords
DocumentDB access

🔑

API Keys & Tokens

Third-party API keys
OAuth client secrets
Webhook signing keys
Service account tokens

📜

Other Secrets

TLS private keys
SSH keys
Encryption passphrases
License keys

⚡ Secrets Manager in 30 Seconds

Store credentials in Secrets Manager → application assumes IAM role → calls GetSecretValue API → receives credential → connects to database/API. No human ever sees the password. Rotation happens automatically.

Chapter 01 — Key Takeaway

Secrets Manager replaces hardcoded credentials with dynamic, IAM-controlled, audit-logged, automatically-rotated secret retrieval. Your application code never contains a password — it asks for one at runtime.

Chapter Two

Why Secrets Management Matters

Credential exposure is the #1 cause of cloud security breaches. Understanding why centralized secrets management exists helps you design secure architectures from day one.

Real-World Security Risks Core

Risk	What Happens	Real Example
Credentials in Git	Bots scan public repos in seconds	AWS keys on GitHub → account compromised in <5 minutes
Shared passwords	No accountability, no audit trail	Entire team uses same DB password → can't trace breach
No rotation	Compromised credential works forever	Leaked API key from 2 years ago still valid
Env vars on servers	Accessible to all processes, logged in crash dumps	Debugging output exposes DATABASE_URL with password
Secrets in CI/CD	Stored plaintext in pipeline configs	Jenkins config with all production passwords readable

Modern Cloud Security Thinking Core

Never store secrets in code or config

Source code and configuration should contain zero secrets. Period. Not even "encrypted" secrets embedded in code.
Request secrets dynamically at runtime

Applications call Secrets Manager at startup. The secret is held in memory only — never written to disk.
Use IAM for access control (not knowledge)

Access is based on "who you are" (IAM role), not "what you know" (shared password). Revoke role = immediate revocation.
Rotate automatically and frequently

Credentials change regularly without human intervention. A compromised secret has a short validity window.
Audit every access

Every GetSecretValue call is logged in CloudTrail. You know exactly who accessed which credential, when, and from where.

Common Mistake

Storing secrets in SSM Parameter Store without encryption, or using the same secret across all environments. Each environment should have its own secret with its own access policy.

🎯 Exam Insight

"Application needs database password securely" → Secrets Manager (not hardcoded, not .env).
"Credential must be rotated automatically" → Secrets Manager (Parameter Store has no native rotation).
"Audit who accessed credentials" → Secrets Manager + CloudTrail.

Chapter 02 — Key Takeaway

Credentials are the keys to your kingdom. Exposed credentials = breached systems. Secrets Manager eliminates human handling of passwords, enforces IAM-based access, and creates a full audit trail.

Chapter Three

How Secrets Manager Works

Under the hood, Secrets Manager is elegantly simple: an encrypted key-value store with IAM access control, KMS encryption, versioning, and an optional rotation Lambda function.

The Core Flow Core

Application starts (ECS, Lambda, EC2)

Runs with an IAM role that has secretsmanager:GetSecretValue permission on specific secret ARNs.
Application calls GetSecretValue

SDK call with the secret name. Request goes to Secrets Manager regional endpoint.
Secrets Manager decrypts with KMS

Internally calls KMS to decrypt the secret value using the associated CMK.
Secret returned to application

Plaintext returned over TLS. Application uses it to connect, holds in memory only.
CloudTrail logs the access

Every call recorded: who, when, which secret, source IP. Full audit trail.

Diagram 1 · How Applications Retrieve Secrets

App → IAM-authenticated API call → Secrets Manager decrypts via KMS → returns credential → app connects

Key Distinction — KMS vs Secrets Manager Core

AWS KMS

Manages encryption keys
Encrypts/decrypts data
Does NOT store your credentials
Think: "the lock mechanism"

Secrets Manager

Manages secret values themselves
Stores, retrieves, rotates credentials
Uses KMS under the hood for encryption
Think: "the vault contents"

How They Work Together

When you store a secret, Secrets Manager calls KMS to encrypt it. When you retrieve, it calls KMS to decrypt. You need permissions on BOTH: secretsmanager:GetSecretValue AND kms:Decrypt on the associated key.

Critical Permission Gotcha

When GetSecretValue fails with AccessDeniedException, the error often only mentions Secrets Manager — but the hidden culprit is usually missing KMS permission. Your IAM role needs: (1) secretsmanager:GetSecretValue on the secret ARN, AND (2) kms:Decrypt on the KMS key that encrypts it. If using a custom CMK for cross-account, the key policy must also allow your role.

Secret Structure & Versioning Core

Secret Name

Logical identifier. Convention: env/service/purpose (e.g., prod/myapp/db-password).

Secret Value

The credential. Can be plaintext string or JSON (recommended for structured secrets: username + password + host + port).

Encryption Key

KMS key used. Default: aws/secretsmanager. Custom CMK for cross-account access.

Version Stages

AWSCURRENT = active. AWSPREVIOUS = last rotated. AWSPENDING = being rotated.

Resource Policy

Optional resource-based policy for cross-account sharing (like S3 bucket policy).

Diagram 2 · AWS Architecture — ECS Task Retrieving Credentials

ECS Task → VPC Endpoint (no internet) → Secrets Manager → returns credential → Task connects to RDS

VPC Endpoint (PrivateLink) — Why It Matters Deep

By default, GetSecretValue calls travel over the public internet (encrypted with TLS). For production workloads in private subnets, create a VPC Endpoint:

Private Traffic

Traffic stays within AWS network — never leaves. No internet gateway or NAT gateway required.

Endpoint Policy

Add a policy to restrict which secrets can be accessed through this endpoint — additional security layer.

Cost

~$0.01/hour + data processing charges. Worth it for any production workload handling sensitive credentials.

Setup

Create Interface Endpoint for com.amazonaws.REGION.secretsmanager in your VPC. Attach to private subnets.

Chapter 03 — Key Takeaway

Secrets Manager = encrypted vault + IAM access + KMS encryption + versioning + rotation. You need both Secrets Manager permissions AND KMS decrypt permission. Use VPC endpoints to keep traffic private.

Chapter Four

Secret Rotation

Rotation is what makes Secrets Manager more than "encrypted Parameter Store." It automatically changes credentials on a schedule — updating both the secret AND the target system — without application downtime.

How Automatic Rotation Works Core

Rotation triggered (schedule or on-demand)

Secrets Manager invokes a Lambda rotation function based on a schedule (e.g., every 30 days).
createSecret — Generate new credential

Lambda generates a new password and stores it as AWSPENDING version.
setSecret — Update the target system

Lambda connects to the database and changes the password to the new value.
testSecret — Validate it works

Lambda authenticates with the new password. If it fails, rotation stops (old credential still works).
finishSecret — Promote to AWSCURRENT

Version stage moves: AWSPENDING → AWSCURRENT. Old password becomes AWSPREVIOUS.

Diagram 3 · Secret Rotation Flow

SM triggers Lambda → Lambda creates new password → updates DB → tests → promotes to AWSCURRENT

Rotation Strategies Core

🔄

Single-User Rotation

One DB user, password rotates in place
Brief moment where old password invalid
Simple setup — good for most cases
Risk: tiny window of connection failure

👥

Alternating-User Rotation

Two DB users alternated (user-A, user-B)
Old user valid until next rotation cycle
Zero-downtime — no connection interruption
Requires additional DB user setup

Native RDS Integration

For RDS/Aurora, Secrets Manager provides managed rotation (no Lambda required for supported DBs). Just enable it — AWS handles everything. Rotation periods: 4 hours to 365 days.

Application Must Handle Rotation

If your app caches the DB password and doesn't re-fetch on connection failure, rotation will break it. Best practice: catch auth errors → call GetSecretValue again → retry with fresh credential.

Chapter 04 — Key Takeaway

Rotation is the killer feature. It automatically changes passwords in both Secrets Manager AND the target system, tests the new credential, and promotes it — zero code changes. Applications always get AWSCURRENT. Handle re-fetch on auth failure.

Chapter Five

Secrets Manager vs Parameter Store

This is the most frequently asked question in AWS interviews and certification exams. Both store values, but they serve different purposes.

Head-to-Head Comparison Core

Feature	Secrets Manager	SSM Parameter Store
Primary purpose	Secret lifecycle management	Configuration storage
Automatic rotation	✓ Built-in (Lambda-based)	✗ No native rotation
Cost	$0.40/secret/month + $0.05/10K calls	Free (standard) / $0.05/param (advanced)
Encryption	Always encrypted (KMS)	Optional (SecureString uses KMS)
Cross-account	✓ Resource policies	✗ Not natively
Versioning	✓ Version stages (CURRENT, PREVIOUS)	Version history (labels)
Max size	64 KB	8 KB
Hierarchy	Flat names (use / convention)	✓ Full hierarchical paths
Free tier	✗ No free tier	✓ 10,000 standard params free
RDS integration	✓ Native managed rotation	✗ No direct DB integration

Pricing — Real Cost Examples Core

Workload	Secrets	Calls/day	Monthly Cost
Small app (2 DB creds)	2	1,000	~$0.95
Microservices (20 secrets)	20	50,000	~$8.25
Enterprise (100 secrets)	100	500,000	~$42.50

vs Parameter Store Advanced: 100 parameters = $5/month. Trade-off: SM gives rotation + cross-account, PS gives lower cost. For credentials that need rotation, the extra cost is justified.

Decision Framework Core

🔒

Use Secrets Manager When

Storing database credentials
Need automatic credential rotation
Cross-account secret sharing
Managing API keys that must rotate
Compliance requires lifecycle management

⚙️

Use Parameter Store When

Application config (non-secret)
Feature flags, endpoint URLs
Need hierarchical paths
Budget-constrained (free tier)
Simple encrypted values (no rotation)

They Coexist

Most production systems use both: Parameter Store for application config (feature flags, URLs) and Secrets Manager for credentials (DB passwords, API keys). They're complementary, not competitors.

🎯 Exam Tips — SM vs Parameter Store

"Automatic rotation" → Secrets Manager. Parameter Store has no rotation.
"Store database password securely" → Secrets Manager (rotation + RDS integration).
"Store app config cheaply" → Parameter Store (free tier).
"Cross-account credential sharing" → Secrets Manager (resource policies).
"Hierarchical configuration" → Parameter Store (/app/prod/db/host paths).

Chapter 05 — Key Takeaway

Secrets Manager = credential lifecycle (rotation + cross-account + audit). Parameter Store = application config (hierarchy + free + simple). The deciding factor: "Does it need automatic rotation?" Yes = Secrets Manager.

Chapter Six

Architecture Patterns & Real-World Usage

How production teams actually use Secrets Manager — from simple single-app setups to multi-account enterprise deployments.

Pattern 1 — ECS/Fargate + RDS (Most Common) Core

Store RDS password in Secrets Manager

Secret: prod/myapp/rds-master. Enable 30-day managed rotation.
ECS Task Definition references secret ARN

Use valueFrom to inject secret as env var at container start.
Task Role has GetSecretValue permission

IAM: secretsmanager:GetSecretValue on the specific ARN + kms:Decrypt.
Application connects with rotated password

Each container start gets latest password. Handles auth errors by re-fetching.

Pattern 2 — Lambda + Third-Party API Keys Core

Setup

Store API key: prod/payments/stripe-key
Lambda execution role: GetSecretValue
Fetch at cold start, cache in memory
Re-fetch if API returns 401

Benefits

API key never in code or env vars
Rotate → next invocation picks it up
CloudTrail shows which function accessed
Multi-env: different secrets per stage

Secret Caching — Avoid Throttling at Scale Deep

Secrets Manager has quotas: 5,500 requests per second per region (soft limit). High-throughput apps can hit throttling without caching.

Approach	How	When to Use
In-memory cache	Store secret in app memory, refresh every X minutes	Lambda (between invocations), long-running services
AWS Caching Client	Official SDK extension with built-in TTL and refresh	Java, Python, Go, Node.js production apps
ECS valueFrom injection	Secret injected at container start as env var	Container workloads (simplest for ECS/Fargate)
Sidecar pattern	Sidecar container fetches and refreshes secrets	Kubernetes / ECS with shared volumes

Best Practice

Cache for 5–15 minutes. Always re-fetch on authentication failure (in case rotation occurred). The AWS Secrets Manager Caching Client handles this automatically with configurable TTL.

Pattern 3 — Multi-Environment Isolation Deep

Environment	Secret Name	Access	KMS Key
Development	dev/myapp/db-password	Dev team roles only	aws/secretsmanager (default)
Staging	staging/myapp/db-password	CI/CD pipeline role	Custom CMK (staging)
Production	prod/myapp/db-password	Prod ECS task role ONLY	Custom CMK (prod)

Least Privilege Pattern

Production task role can only access prod/* secrets. Dev roles cannot access production. Enforce via IAM resource ARN: arn:aws:secretsmanager:*:*:secret:prod/*.

Pattern 4 — CI/CD Pipeline Secrets Core

🚀

How It Works

CodeBuild/GitHub Actions assume IAM role
Role scoped to specific deploy secrets
Pipeline fetches credentials at build time
Never stored in pipeline config/YAML

🛡️

Security Benefits

No secrets in source control (ever)
Pipeline logs never contain creds
Full audit trail of deployments
Revoke role = immediate no access

Pattern 5 — Cross-Account Secret Sharing Deep

Secrets Manager supports resource policies (like S3 bucket policies) for cross-account sharing. Three layers must align:

Account A: Attach resource policy to secret

Allow Account B's root or specific role to call secretsmanager:GetSecretValue on this secret's ARN.
Account B: IAM role with SM permission

Role in Account B needs secretsmanager:GetSecretValue on Account A's secret ARN.
KMS key policy: Allow Account B

The CMK encrypting the secret must allow Account B to call kms:Decrypt. This is the most commonly missed step.

Common Pitfall

If using the default aws/secretsmanager key, cross-account access is impossible (AWS managed keys don't support cross-account). You must use a Customer Managed CMK and update its key policy.

🎯 Exam Tips — Architecture

"ECS needs database password" → Task Role + Secrets Manager + valueFrom.
"Lambda needs API key without hardcoding" → SM + execution role + cache in memory.
"Prevent dev from accessing prod secrets" → IAM resource condition on secret ARN prefix.
"Cross-account secret sharing" → Resource policy on secret + IAM in consumer + custom CMK.

Troubleshooting — Common Secrets Manager Errors Core

Error	Likely Cause	Fix
AccessDeniedException	Missing `kms:Decrypt` permission (hidden culprit)	Add kms:Decrypt on the CMK to IAM role. Check key policy allows role.
ResourceNotFoundException	Secret name typo OR wrong region	SM is regional. Verify ARN and region match. Check for trailing whitespace.
ThrottlingException	Too many GetSecretValue calls	Implement secret caching (5–15 min TTL). Use AWS caching client.
InvalidRequestException (rotation)	Lambda can't connect to database	Check Lambda security group, DB SG ingress, and VPC/subnet config.
Connection fails after rotation	App caching old password	App must re-fetch AWSCURRENT on auth failure. Don't cache indefinitely.
Cross-account AccessDenied	Using aws/secretsmanager key (no cross-account)	Switch to Customer Managed CMK. Update key policy for consumer account.

Debugging Tip

For AccessDenied, check three layers: (1) Secret resource policy allows principal, (2) IAM policy allows secretsmanager:GetSecretValue, (3) KMS key policy allows kms:Decrypt. Use CloudTrail to identify which layer rejected.

Complete Summary — AWS Secrets Manager

What: Secure storage + lifecycle management for credentials (passwords, API keys, tokens).
Why: Eliminates hardcoded secrets, enables rotation, provides audit trail, enforces least privilege.
How: App → IAM role → GetSecretValue → SM decrypts via KMS → returns credential.
Rotation: Lambda-based (4 steps: create, set, test, finish). Zero application code changes.
vs Parameter Store: SM = rotation + cross-account + lifecycle. PS = config + hierarchy + free.
Permissions: secretsmanager:GetSecretValue + kms:Decrypt on the associated CMK.
Cost: $0.40/secret/month + $0.05/10K calls. Worth it for any credential needing rotation.
Best practice: Naming convention (env/service/purpose), enable rotation, handle re-fetch on auth failure.

Final Key Takeaway

If a credential needs to be rotated, audited, or shared cross-account — use Secrets Manager. If it's non-sensitive config that doesn't rotate — use Parameter Store. Modern cloud apps never hardcode passwords.

← AWS KMS Security Overview →