Secrets Management

Viberglass keeps three different kinds of secrets:

  1. Platform infrastructure secrets — credentials the backend itself needs (database URL, webhook encryption key, Slack tokens). These live in SSM Parameter Store as SecureString parameters and are populated by Pulumi.
  2. User-managed secrets — API keys and tokens that get injected into worker environments at job time (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.). These are managed from the Secrets page in the web UI and stored either in SSM, in the encrypted secrets table, or as environment variables on the backend (for local dev).
  3. Integration credentials — secrets attached to a specific integration on a specific project (e.g. a GITHUB_TOKEN for project A's GitHub integration). Stored alongside the integration_credentials row.

Storage backends

Every user-managed secret has a secret_location (the column is on the secrets table, added in migration 009_add_secrets_table.ts):

secret_locationWhere the value livesWhen to use
ssmAWS SSM Parameter Store, SecureString under /viberator/secrets/<name>, encrypted with the per-environment KMS key.The default in production. The backend reads the value at job dispatch time using its task role.
databaseThe secret_value_encrypted column on secrets, encrypted with SECRETS_ENCRYPTION_KEY.Useful when you don't want a separate SSM round-trip, or for self-hosted installs without AWS.
envA backend process environment variable.Local development only. The secret name is matched against process.env.

The secrets row stores the location plus a secret_path (the SSM parameter name or env var name) and never holds the plaintext value when location = ssm.

Encryption keys

Two encryption keys live outside the secrets pipeline itself:

  • SECRETS_ENCRYPTION_KEY — symmetric key used by the backend to encrypt secret_value_encrypted for location = database rows. Required to be ≥ 32 chars. In production it is stored in SSM at /viberglass/<env>/backend/secrets-encryption-key and exposed to the ECS task as an env var via the task definition.
  • WEBHOOK_SECRET_ENCRYPTION_KEY — separate key used to encrypt webhook signing secrets in webhook_provider_configs. Generated by Pulumi (random.RandomPassword) and stored in SSM at /viberglass/<env>/backend/webhook-secret-encryption-key.

KMS encryption for SSM SecureString parameters is handled by the per-environment customer-managed key created in the base Pulumi stack (alias/viberglass-<env>-ssm). Both the backend execution role and the worker execution role have kms:Decrypt against this key, scoped down to the parameters they actually need.

How the backend reads secrets

When the backend dispatches a job, it builds a bootstrap payload that includes a list of secret references for the clanker. For each reference it calls a resolver that knows how to read the configured backend:

  1. Look up the secrets row by id.
  2. If secret_location = env, read process.env[secret_path].
  3. If secret_location = database, decrypt secret_value_encrypted with SECRETS_ENCRYPTION_KEY.
  4. If secret_location = ssm, call ssm:GetParameter with WithDecryption=true. The result is held in memory only.
  5. Inject the resolved value into the worker invocation as an environment variable.

For Lambda this means a LambdaInvoker env block with the secrets baked in for that single call. For ECS the values are passed through the task definition's secrets field (referencing SSM ARNs directly) when possible, or as environment overrides when not.

How the worker reads secrets

The worker never queries SSM directly for user-managed secrets. The backend has already resolved everything and passed the values through the bootstrap payload, so the worker just does process.env[name] from inside the agent harness.

The one exception is the bootstrap payload itself on ECS clankers. Because ECS task definitions don't carry large payloads gracefully, the backend writes the payload to S3, passes the bucket and key as env vars, and the worker uses its task role to fetch and JSON-parse it. The task role's S3 permission is scoped to the specific prefix.

Platform infrastructure secrets

These are created and managed by Pulumi, not by the UI:

SSM pathPurpose
/viberglass/<env>/backend/database-urlFull Postgres connection URL written by the database component.
/viberglass/<env>/backend/database-hostHostname only.
/viberglass/<env>/backend/secrets-encryption-keyThe SECRETS_ENCRYPTION_KEY env var for the backend task.
/viberglass/<env>/backend/webhook-secret-encryption-keyThe WEBHOOK_SECRET_ENCRYPTION_KEY env var.
/viberglass/<env>/backend/slack-signing-secretSlack request signing secret.
/viberglass/<env>/backend/slack-bot-tokenSlack bot OAuth token.
/viberglass/<env>/deployment/regionDefault region for deploy workflows.
/viberglass/<env>/deployment/ecrRepositoryECR repo for the backend image.
/viberglass/<env>/ecs/cluster / serviceBackend ECS pointers.
/viberglass/<env>/frontend/apiUrlBackend URL the frontend builds against.
/viberglass/<env>/amplify/appId / branchName / regionAmplify pointers.

The Pulumi platform stack creates the parameters that need to ship with predictable defaults, then ECS tasks reference them through secrets blocks in their task definitions so the values land as env vars in the running container.

For Slack secrets specifically, Pulumi seeds them with the literal not-configured if you have not set pulumi config set --secret slackBotToken .... After the first deploy, you can either rotate them in the Slack admin and re-run Pulumi, or update the SSM parameter directly via the AWS console — the Pulumi resource is intentionally written so that out-of-band edits do not get clobbered on the next pulumi up.

Integration credentials

Integration credentials are credentials like a project's GITHUB_TOKEN. Each project's integration page lets you select an existing secret or create a new one inline; the new secret is written to SSM (or the database, depending on the environment configuration) and immediately linked to the integration.

When the worker runs against a repository it merges three secret sets, in order:

  1. The integration credential for the project's SCM integration.
  2. The clanker's attached secrets.
  3. Any task-template secrets (claws only).

Later sets override earlier ones if names collide, so the most specific secret wins.

Rotation

To rotate a secret:

  1. SSM-backed — update the SSM parameter (via the console, CLI, or Pulumi). The next job dispatch reads the new value.
  2. Database-backed — open the secret in the UI, paste the new value, save. The backend re-encrypts and updates the stored secret value.
  3. Env-backed (local dev) — update the env var and restart the backend.

There is no built-in auto-rotation yet. For the Slack and webhook keys you control via Pulumi, the recommended cadence is to rotate the source secret and re-run pulumi up against the platform stack.

Local development

For docker-compose runs the easiest setup is ENV secret location for everything: define the relevant API keys in your shell or .env, list them in the Secrets page (just the name and the env var to read from), and they'll be picked up by both the backend and any workers spawned via the Docker invoker.

You also need to set the two encryption keys (SECRETS_ENCRYPTION_KEY, WEBHOOK_SECRET_ENCRYPTION_KEY) so the backend can boot and so any database-backed secrets can be decrypted. Any 32+ char string works for local dev.

Auditing

  • Reads against SSM are logged in CloudTrail; the backend's task role is the principal.
  • Secret CRUD via the API writes to the application log with the actor and the secrets.id. The plaintext value is never logged.
  • Every access from a worker is bound to a specific jobs.id, so you can trace "which job pulled which secret" by joining the application logs to the job table.

See also

  • Secrets (user docs) — how to manage secrets from the UI without worrying about the storage backend.
  • Integrations — how project integrations attach credentials to a workspace.
  • Deployment — where the platform secrets are created during the Pulumi runs.
Previous
Deployment