Monopam vault: discovery, vaulting and rotation
What changed
Every content update is versioned. Select a version to compare it with the one before.
Version 9 → 10
UpdateNew module on operating the vault in production: the vault key and re-encryption, recovering from failed rotations, secret syncs and shared items, token encoding, machine identities and dynamic secret leases. The module now has diagrams and a closing list of what to check first.
3 lessons updatedduration
Content update. Enrolled learners were notified.
Duration changed from 24 to 26 minutes.
Every secret in the tenant database is encrypted with a key that each Monopam process reads from its environment when it starts. If no key is set, Monopam uses a built-in key. That built-in key ships inside the product, so it protects against casual reading but not against someone who holds both a database dump and the software. A production installation sets its own key.
| Variable | Purpose |
| --- | --- |
| `MONOPAM_VAULT_KEY` | The active key. New writes use it |
| `MONOPAM_VAULT_KEY_VERSION` | The version number of the active key. A higher number means a newer key |
| `MONOPAM_VAULT_KEYS` | The older keys, kept so that rows written under them can still be read |
Together these form the **key ring**. A super admin can check the state with the vault maintenance key status, which reports the active version and whether the installation is externally keyed.
### Three rules that explain most incidents
**1. Every process, and only at start.** The web server and every gateway decrypt secrets, so all of them need the same key ring. The environment is read once at start, so a change does nothing until the process restarts. The classic symptom: the key was set on the web server only. Revealing a password in the browser works, because the web server can decrypt, but SSH and RDP fail, because the gateway cannot. Set the key on every host, restart each one, and plan a maintenance window for it.
**2. The job follows the key, never the other way round.** After the key is in place everywhere, a super admin runs the re-encryption job, which rewrites existing rows under the active key. Running the job without an external key is not a security gain: the rows are simply rewritten under the built-in key and a warning is logged.
**3. Old keys stay until the job reports zero failures.** The job result lists, per table, how many values it scanned, upgraded and failed. A failed value is a row the job could not decrypt, usually because the key version it was written with is missing from the ring. The job leaves such rows untouched. If you remove old keys now, those rows become unreadable for good. Add the missing key to `MONOPAM_VAULT_KEYS`, restart, run the job again and remove old keys only after a run with `FailedValues` at zero everywhere.
::diagram pam-vault-key-change
### There is no recovery
A lost key cannot be recovered, and every row encrypted under it stays unreadable. Store the key outside Monopam, for example in a password manager with restricted access, before you start the change, and treat it like the most sensitive credential you own.
> Tip: read the vault maintenance key status before and after the change. Before, it tells you whether the installation is externally keyed at all. After, the active version must be the one you set.
A rotation changes the password on the target and then runs the automations that update dependent systems, such as a service or an application pool that uses the account. The setting `If Automation Fails` on the rotation policy decides what happens when one of those automations fails.
| Choice | What Monopam does | Fits when |
| --- | --- | --- |
| `Revert to the previous password` (default) | Changes the target back, so vault and target agree on the old password | The dependent system must keep working and the target tolerates a second change |
| `Keep the new password` | Leaves vault and target on the new password and flags the item for follow-up | A second change in quick succession would lock the account |
Think about what a revert really is: a second password change on the target a few seconds after the first. On a system that locks an account after rapid changes that second change is the dangerous one. There you keep the new password and repair the dependent system by hand.
### When the revert fails too
The audit event `Password Rotation Revert Error: Manual Intervention Required` means the automation failed and the attempt to go back failed as well. Vault and target may now hold different passwords, and Monopam cannot know which one the target accepts.
Do not start clicking. First establish which password the target accepts now. Then act on that: `Reconcile` when the vault holds the right value and the target must be brought in line, a revert from Password History followed by `Reconcile` when the previous value is the valid one, or a manual reset on the host. Trying passwords in the wrong order against a system with a lockout threshold locks the account and turns a rotation problem into an outage.
::diagram pam-rotation-outcomes
### Where a rotated secret travels
**Secret syncs.** A vault item can be synchronized to an external store such as AWS Secrets Manager. The sync is queued only when a password is final: after a save, or after a successful rotation that was not reverted. A rotation that failed and was reverted pushes nothing, so the external store never sees a password that lived for ten seconds.
**Shared items between tenants.** A service provider can make an item of its main tenant usable in a customer tenant without copying the secret. The mode decides what the customer's people can do.
| Mode | Customer tenant can |
| --- | --- |
| `Sessions only` (default) | Open sessions with the account. The secret is not shown, not returned by the vault API and not placed into script tokens |
| `Sessions and reveal` | Also read the value |
The secret lives only in the main tenant. Rotation, history and reconcile happen there, and every use in the customer tenant reads the current value at that moment, so a rotated password is current everywhere at once.
**Tokens in launchers and scripts.** A token such as `{{Vault.appadmin.Password}}` inserts the raw value. Raw is wrong whenever the place you paste into has its own syntax. In a URL, a password containing `&` or `#` breaks the address. Add an encoding modifier: `{{Vault.appadmin.Password:url}}` percent-encodes the value, `:sh` and `:ps` make it safe inside a single-quoted shell or PowerShell string, and `:base64` encodes it for a transport that decodes Base64. Modifiers cannot be chained, and a misspelled modifier is ignored without an error and yields the raw value, so test with a password full of special characters.
> Tip: after a failed rotation, read the audit before you touch the account. If the revert succeeded, vault and target agree on the old password and only the dependent system needs repair. Only `Password Rotation Revert Error: Manual Intervention Required` means the two may differ.
People reveal secrets in the browser. Pipelines, pods and operators read them from the Secrets Store through a **machine identity** (`Secrets` > `Identities`). The identity logs in, receives a short-lived token and reads what its **grants** on project, environment and folder allow.
### Choosing the login method
| Method | The machine proves itself with | Choose it when | Cost |
| --- | --- | --- | --- |
| Universal Auth | A client id and client secret | The caller runs anywhere, including a CI server whose token issuer is internal only | A secret to store and to rotate by hand |
| OIDC | A signed token from its platform, checked against the issuer's public keys and the bound claim | The platform's issuer is publicly reachable, as with hosted pipeline services | Nothing stored. Does not work if Monopam cannot reach the issuer's public documents |
| Kubernetes | A service account token of the cluster | The workload runs in a cluster | Monopam must reach the cluster |
Example: a team does not want a long-lived token in its repository on a hosted pipeline service. An OIDC identity whose bound claim matches the workflow's origin gives the pipeline a short-lived token and leaves nothing to leak. Their self-hosted build server issues tokens too, but its issuer is visible only inside the network, so Monopam cannot validate them. That server gets Universal Auth, with `Allowed IP ranges` set to the build network so a leaked secret is useless elsewhere.
### Behaviour worth knowing before an incident
- **Tokens outlive changes to the identity.** Disabling an identity, or `Rotate Secret`, stops new logins at once. Tokens already issued keep working until their `Token TTL (minutes)` ends. A short TTL is your real containment time.
- **`Rotate Secret` has no rollback.** The old secret dies immediately. Every caller that still has it fails with `Invalid client credentials.` until it gets the new one. Distribute first where you can, and never rotate a shared identity casually.
- **`Allowed IP ranges` is checked at login only**, not on each read. Behind a load balancer the caller address is the first IPv4 address in `X-Forwarded-For`. If the balancer does not set that header, every caller looks like the balancer and a restricted identity is refused. Fix the header. Do not add the balancer's address to the range, which would let everyone behind it through.
- **Failed logins pause an address.** From the tenth failure within five minutes, the address gets `Too many failed logins` for the rest of the window. Thirty pipelines behind one NAT address share that fate, so one job looping with an old secret stops them all.
- **Refusals carry their code in the body.** An authorization refusal on the machine API answers HTTP 400 with `"code":403` inside. Clients must read the body.
- **`Restricted (never visible)` is a screen setting.** It hides a value in the web interface. A machine with a grant still receives it, through the machine API and through the Vault-compatible API alike. Only grants limit what a machine can read.
### Dynamic secrets
A dynamic secret (`Secrets` > `Dynamic`) creates a short-lived credential on demand, for example a database login, and hands it out as a **lease**. When the lease ends, Monopam revokes the credential at the target.
| Situation | What happens | What you do |
| --- | --- | --- |
| An application renews its database lease again and again | Each renewal extends the lease, but never beyond creation time plus `Max TTL (minutes)` | Have the application request a new lease and reconnect before that limit |
| The target was unreachable when the lease expired | The lease shows `Revoke Failed`. Monopam does not try again | Remove the generated login at the target by hand. It may still exist |
| A Kubernetes token lease is renewed | The renewal is refused | Request a new lease before the token expires |
::diagram pam-machine-lease
`Revoke Failed` is the state to watch for. It is the one case where a credential that Monopam created can stay alive without anyone noticing.
### What to check first
| Symptom | Check first |
| --- | --- |
| Every caller of one identity fails with `Invalid client credentials.` | Whether somebody used `Rotate Secret`. The old secret died at once |
| Many pipelines get `Too many failed logins` at the same time | One job looping with an old secret behind the shared NAT address |
| A restricted identity is refused behind a load balancer | Whether the balancer sets `X-Forwarded-For` |
| A client reports HTTP 400 where a refusal was expected | The body: `"code":403` is the authorization refusal |
| A disabled identity still reads secrets | Its issued tokens live until `Token TTL (minutes)` ends |