Kubernetes

InfraSigns ships a Helm 3 chart for production Kubernetes deployments.

Prerequisites

Requirement	Notes
Helm 3.x	`brew install helm` or helm.sh
PostgreSQL 13+	Accessible from inside the cluster (13+ is required — the schema uses `gen_random_uuid()` and declarative hash partitioning)
Prometheus	Accessible from inside the cluster
Kubernetes Secret	Containing sensitive env vars (see below)

Create the secret

All sensitive values are injected at runtime from a Kubernetes Secret referenced by existingSecret. Create it before installing the chart:

kubectl create secret generic infrasigns-secrets \
  --from-literal=DATABASE_DSN='postgres://user:password@postgres:5432/infrasigns?sslmode=require' \
  --from-literal=LLM_API_KEY='sk-...' \
  --from-literal=TELEGRAM_TOKEN='123456:ABC-...' \
  --from-literal=TELEGRAM_CHAT_ID='-100123456789' \
  --from-literal=SLACK_WEBHOOK_URL='' \
  --from-literal=SLACK_TOKEN='' \
  --from-literal=SMTP_USER='' \
  --from-literal=SMTP_PASSWORD='' \
  --from-literal=PAGERDUTY_ROUTING_KEY='' \
  --from-literal=HEARTBEAT_URL='' \
  --from-literal=API_TOKEN="$(openssl rand -hex 16)" \
  --from-literal=WEBHOOK_TOKEN="$(openssl rand -hex 16)" \
  --from-literal=INFRASIGNS_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
  --from-literal=BOT_TELEGRAM_TOKEN='' \
  --from-literal=BOT_SLACK_APP_TOKEN='' \
  --from-literal=BOT_SLACK_TOKEN='' \
  --from-literal=DO_TOKEN='' \
  --from-literal=HETZNER_TOKEN='' \
  --from-literal=PROM_TOKEN=''

Keys PAGERDUTY_ROUTING_KEY and HEARTBEAT_URL can be empty strings — empty values silently disable those channels. The two Slack keys are not in that set; see the Slack sentence below for what an empty value does there. PAGERDUTY_ROUTING_KEY supplies the PagerDuty Events API v2 integration key; empty disables the channel, and because unset keys expand to empty, a secret that omits it simply leaves PagerDuty off. PagerDuty is incident-only — it pages on incident firing/resolution (an inbound alert group, or a health check that fails and recovers) and never receives digests. Slack additionally requires choosing a transport in values via config.notify.slack.mode: set it to webhook and provide SLACK_WEBHOOK_URL, or to api and provide SLACK_TOKEN (a chat:write bot token) plus config.notify.slack.channel — the api transport posts incident updates as thread replies. With mode unset the whole Slack block is omitted and neither key is read, which is what the empty '' in the example above relies on. With mode set, an empty value is not a silent disable but a startup failure: the process refuses the config with notify.slack.webhook_url is required when mode is "webhook" (or the matching notify.slack.token message in api mode) and the pod does not start. A webhook_url that is present but not an http/https URL naming a remote host — a paste with the scheme dropped, say — is the opposite: it loads, builds no Slack channel, and is reported by a startup line and by infrasigns_notify_channel_deliverable. SMTP_USER/SMTP_PASSWORD supply email credentials for the SMTP channel (enabled via config.notify.email.smtp_host); set both to authenticate, or leave both empty for an unauthenticated relay. Setting only one is rejected at config validation — and because unset keys expand to empty, an email channel that needs auth but whose secret omits these keys silently sends unauthenticated. LLM_API_KEY can also be empty when config.llm.provider is none (the deterministic analysis engine needs no key). An empty or absent API_TOKEN disables the manual digest trigger endpoint (POST /api/digest/trigger) and the MCP server (/mcp). Both mount whether or not sign-in is configured, and the token is the whole gate: each one acts on this deployment's own organization and never on a tenant's, so whoever holds API_TOKEN is being trusted as the operator. TELEGRAM_CHAT_ID can be empty, and the pair is what enables delivery: Telegram is a delivery channel only when TELEGRAM_TOKEN and TELEGRAM_CHAT_ID are both set. A token with an empty chat ID is a deliberate, valid shape — the token is what the Q&A bot falls back to where that bot is enabled (config.bot.telegram.allowed_chat_ids), and there is no delivery channel — so the pod starts and stays ready, nothing is delivered to Telegram, and a startup line names notify.telegram.chat_id, the config key this Secret entry populates. Because unset keys expand to empty, a secret that simply forgot the chat ID lands in exactly that state. The startup line cannot tell the two apart — they are the same configuration — so it names both remedies and leaves the choice to you: set notify.telegram.chat_id (this Secret's TELEGRAM_CHAT_ID) to deliver, or move the token to bot.telegram.token, which on this chart means putting it in BOT_TELEGRAM_TOKEN and leaving TELEGRAM_TOKEN empty, if the Q&A bot is all this deployment wants. That is why the line is logged at info level; every other incomplete channel is a warning. Setting TELEGRAM_CHAT_ID while TELEGRAM_TOKEN is empty has no second reading and is a warning — leave both empty to switch Telegram off with no line and no series. BOT_TELEGRAM_TOKEN can be empty: the Q&A bot's Telegram transport (enabled via config.bot.telegram.allowed_chat_ids) falls back to TELEGRAM_TOKEN; set it only to run the bot as a separate Telegram identity. An enabled Telegram bot with BOTH keys empty fails config validation — unset keys expand to empty, so this catches a secret that forgot the token instead of silently polling with a dead credential. The Slack transport (enabled via config.bot.slack.allowed_channel_ids) needs BOT_SLACK_APP_TOKEN (the Socket Mode xapp-… app-level token — required, no fallback, so a secret that omits it fails validation) and BOT_SLACK_TOKEN (the xoxb-… reply token, which can be empty to fall back to SLACK_TOKEN).

INFRASIGNS_ENCRYPTION_KEY is the root key that encrypts credentials stored in the database — anything saved through the web UI: a source token, a basic-auth password, an SMTP password, a Slack webhook URL. It is 32 bytes of standard base64, which is what both openssl rand -base64 32 and infrasigns secrets keygen print. Like the notify keys it can be empty or absent, and unlike them the consequence is not a disabled feature: nothing is encrypted, and every stored credential sits in the clear in the org_configs table, readable from any dump, backup or read replica. The pod starts and stays ready either way, so a Secret that simply forgot this key gives no signal at all. Credentials that live in this Secret and reach the config as ${VAR} are unaffected in either direction — they never enter the database. Losing the key loses every stored credential: there is no recovery, and everything saved through the UI would have to be re-entered, so keep a copy somewhere other than beside the database backups (an archive holding both protects nothing).

Setting the key on an already-running deployment is safe and is not a breaking change: encryption engages on the key's presence, values already stored in the clear stay readable, and serve seals them on its next start (infrasigns secrets seal is the same pass on demand). Removing it again is not symmetric — run infrasigns secrets unseal --write-cleartext while the key is still configured, because only a process that can decrypt can write the values back out.

Rotating it is one restart plus one command: put the new key in INFRASIGNS_ENCRYPTION_KEY, add the old one to config.security.previous_encryption_key as a ${VAR} reference resolved from this same Secret, helm upgrade, then run infrasigns secrets rewrap and drop the old entry at the next deploy. To supply the key as a file instead — the shape the External Secrets Operator projects — leave this variable out and see below; setting both forms is refused at start-up by name rather than resolved by precedence.

DO_TOKEN is the DigitalOcean personal access token (read scope) referenced as ${DO_TOKEN} by a config.sources entry of type: digitalocean — see DigitalOcean. Unlike the notify keys, it can NOT be usefully empty: an omitted or empty DO_TOKEN expands to an empty token and that source fails to load (a required-credential fail-loud, not a silent disable). Omit the key entirely if you configure no DigitalOcean source.

HETZNER_TOKEN is the Hetzner Cloud API token (read scope) referenced as ${HETZNER_TOKEN} by a config.sources entry of type: hetzner — see Hetzner Cloud. Like DO_TOKEN, it can NOT be usefully empty: an omitted or empty HETZNER_TOKEN expands to an empty token and that source fails to load (a required-credential fail-loud, not a silent disable). Omit the key entirely if you configure no Hetzner source.

PROM_TOKEN is the bearer token for an authenticated Prometheus-compatible backend (Grafana Cloud, Dash0, Mimir, VictoriaMetrics), referenced as ${PROM_TOKEN} by a config.sources entry of type: prometheus (or loki). Unlike the DigitalOcean/Hetzner tokens it is OPTIONAL — an omitted or empty PROM_TOKEN expands to an empty token, and the source simply queries unauthenticated (the backend returns 401 if it requires auth). The env-var name is a convention: any name works as long as the source's token: '${…}' reference matches. Omit the key entirely if all your Prometheus/Loki sources are unauthenticated.

WEBHOOK_TOKEN enables the inbound alert webhook receiver (POST /webhook/alerts) and post-deploy verification (POST /webhook/deploys). Empty or absent, the receiver is disabled: since #558 the route is still mounted, but it answers 401 {"error":"unauthorized"} to every request and ingests nothing, so it is never reachable unauthenticated and its presence does not disclose whether you configured it — and a config with config.deploys.enabled: true fails validation. Non-secret webhook settings (dedupe_window, model) live under config.webhook in values; the token itself is accepted only via the secret (the chart schema rejects config.webhook.token).

DATABASE_DSN must reference a role that owns the schema (both the chart's migration job and serve's startup migration run on it). The tenant tables carry forced row-level security as defense-in-depth — see the database notes for the backup implication (pg_dump must run as a superuser or BYPASSRLS role) and the least-privilege groundwork.

External Secrets Operator

If you use AWS Secrets Manager, HashiCorp Vault, or similar, create an ExternalSecret that produces the same key set and reference it with existingSecret.

The encryption key has a second shape here, and it is the one that fits: ESO projects a secret as a mounted file, so rather than routing the key through INFRASIGNS_ENCRYPTION_KEY you can point an ExternalSecret at a Secret of its own and hand the chart its name.

encryptionKeySecret:
  name: infrasigns-encryption-key   # must already exist in the release namespace
  key: encryption-key               # the Secret key, and the file name in the pod

The chart mounts it read-only at /etc/infrasigns-keys/<key> in both the server and the migration Job — infrasigns migrate up opens the store and therefore reads the key file too, so a key mounted only into the Deployment would fail every upgrade at the pre-upgrade hook — and renders security.encryption_key_file pointing at that path. Only the named key is projected, so a Secret carrying other material does not spill it into the pod. To create the Secret by hand instead:

infrasigns secrets keygen > key
kubectl create secret generic infrasigns-encryption-key \
  --namespace infrasigns --from-file=encryption-key=key
rm key

keygen writes the key to stdout and every word of guidance to stderr, so that redirect produces a file the chart can hand over as it stands.

If something other than this chart puts the file in the pod — a Vault agent sidecar injected through podAnnotations, say — set config.security.encryption_key_file to that path and leave encryptionKeySecret.name empty. Setting both fails the render rather than choosing between them.

Install

helm install infrasigns oci://ghcr.io/infrasigns/charts/infrasigns \
  --namespace infrasigns --create-namespace \
  --set existingSecret=infrasigns-secrets \
  --set config.sources[0].name=production \
  --set config.sources[0].url=http://prometheus.monitoring.svc:9090 \
  --set config.llm.provider=openai \
  --set config.llm.model=gpt-4o-mini

Or with a values file:

# my-values.yaml
existingSecret: infrasigns-secrets

config:
  sources:
    - name: production
      url: http://prometheus.monitoring.svc:9090
    # No self-hosted Prometheus? Pull AWS CloudWatch (type: cloudwatch, region,
    # IRSA/IAM role), DigitalOcean (type: digitalocean, token: ${DO_TOKEN}), or
    # Hetzner Cloud (type: hetzner, token: ${HETZNER_TOKEN}); probe any HTTP
    # URL (type: healthcheck, endpoints — no secret needed); or run LogQL metric
    # queries against Loki (type: loki, url) instead — see
    # docs/configuration.md#sources.
  llm:
    provider: openai
    model: gpt-4o-mini
  reports:
    digest:
      schedule: "0 8 * * *"
      # stale_episode_after: "24h"   # optional; see the values table below

helm install infrasigns oci://ghcr.io/infrasigns/charts/infrasigns \
  --namespace infrasigns --create-namespace \
  -f my-values.yaml

What the chart creates

Resource	Notes
ConfigMap (`<release>-infrasigns-config`)	Non-sensitive config rendered from `config.` values; this is the copy the Deployment mounts, and it is an ordinary resource* — applied in the normal sync, updated by `helm upgrade --no-hooks`, and deleted by `helm uninstall`
ConfigMap (`<release>-infrasigns`)	The same body, rendered from the same named template, as a pre-install/pre-upgrade hook (weight -1) so it exists before the weight-0 migration Job mounts it. Nothing long-lived mounts this copy — except on a release installed before the split, whose running pod mounts this very name until the next upgrade rolls it (see Upgrading from a release installed before the split). The hook copy is the one that keeps the historical name, and which name goes where is load-bearing rather than cosmetic — see Upgrading from a release installed before the split. It renders only when the Job does (both `migration.enabled` and `preflight.enabled` off means neither exists), and `helm uninstall` leaves it behind — see Uninstall. Why two: #583
Job (migrate / preflight)	Runs the config pre-flight and then `infrasigns migrate up` before each install/upgrade (hook weight 0). The two halves have separate switches: `migration.enabled: false` drops `migrate up`, `preflight.enabled: false` drops the pre-flight. The Job is named `<release>-infrasigns-migrate` while `migrate up` runs and `<release>-infrasigns-preflight` when it does not — one or the other, never both. (Every name in this table is built from the chart's `fullname`, which is `<release>-infrasigns` unless the release name already contains `infrasigns`, in which case it is the release name alone — so a release called `infrasigns` gives `infrasigns-migrate`, which is the form the examples below use. A long release name is truncated to fit Kubernetes' 63-character label bound, and the point at which that starts differs between the two suffixes, so `kubectl get job -n <namespace>` is the reliable read.)
Deployment	Single replica — see Replica count
Service	ClusterIP on `service.port` (default 8080); routes to the container via the `http` named port
ServiceAccount	Created by default; set `serviceAccount.create: false` to bring your own
Ingress	Disabled by default
ServiceMonitor	Disabled by default; requires prometheus-operator

GitOps (Argo CD, Flux)

Nothing in this section was driven against Argo CD. No Argo installation was used to write it; what is measured is our own rendered output and the behaviour of Helm's hooks on a real cluster. Each claim below says which of the three it is, because the difference matters when it is your sync that hangs.

The config the pod runs is an ordinary resource, and that is the part that matters here. Since #583 the ConfigMap the Deployment mounts carries no helm.sh/hook annotations (measured, from helm template), so a GitOps controller treats it like the Deployment and the Service: it is applied in the ordinary sync, and it shows up in argocd app diff. Before the split, the only copy of your configuration was a Helm hook — which Argo maps to a sync hook, out of the ordinary apply set — so the diff of your application did not include the thing that decides what it does.

Flux: the helm-controller runs helm install/helm upgrade itself, so the hook ordering is Helm's own, exactly as described in Upgrade. That follows from Helm's semantics rather than from a run of ours.

Argo CD: what remains in the hook phase is the migration Job (weight 0) and the hook copy of the ConfigMap (weight -1) that only that Job mounts. Argo maps Helm's pre-install/pre-upgrade to PreSync. Whether it also honours helm.sh/hook-weight as an ordering within PreSync is the thing we have not measured. If it does not, the two are unordered and the Job's pod can be created before its ConfigMap exists. That consequence is measured, on kind: the pod sits in Init:0/1 with

Warning  FailedMount  MountVolume.SetUp failed for volume "config":
  configmap "<release>-infrasigns" not found

and the sync fails at the timeout. It is loud and recoverable — a retried sync finds the ConfigMap already applied — and it is never a silent stale config, which is the failure mode worth designing against. If you hit it, the lever is an explicit sync wave on the hook ConfigMap:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

The chart deliberately does not ship that annotation. It would be a third-party accommodation we have never tested, and shipping one on a belief is how the defect above got into the chart in the first place. If you drive this against a real Argo CD, tell us what happened and it can stop being an open question.

Upgrade

helm upgrade infrasigns oci://ghcr.io/infrasigns/charts/infrasigns \
  --namespace infrasigns \
  -f my-values.yaml

The migration Job runs automatically before the new Deployment starts. It runs two containers: a preflight that runs infrasigns config validate against the rendered config, and then infrasigns migrate up. If either fails, the upgrade stops and the previous Deployment version keeps running. With migration.enabled: false the Job renders as <release>-infrasigns-preflight and carries the pre-flight alone.

That now covers the config too, which it did not before #583. The ConfigMap the Deployment mounts used to be the hook, at weight -1 — so a refused upgrade had already written the refused config to it by the time the pre-flight failed at weight 0. helm rollback then reported success in a second without touching it, because rollback does not run hooks and the chart declares none for it, and the surviving pod kept working only until something restarted it: the next pod start read the refused config and CrashLoopBackOffed, on a config no helm command in your history asked for. (Driven on a cluster, not reasoned about.) With the mounted copy an ordinary resource, a failed upgrade leaves it holding the last config that passed, and the delete-a-pod case is an ordinary restart.

The same split fixes helm upgrade --no-hooks, which is what GitOps tooling and --atomic retries reach for. It used to roll the pod (the Deployment's checksum/config annotation changed) onto a ConfigMap that hooks never applied, so the new pod ran the old config — and the divergence was permanent, because a later ordinary helm upgrade with the same values then updated the ConfigMap and did not roll the pod. Both halves are now ordinary resources that move together.

--no-hooks is the one path the pre-flight cannot cover, and it got wider. The pre-flight lives in the hook Job, so --no-hooks skips it — while the mounted ConfigMap, now an ordinary resource, IS applied. So a config the daemon will refuse reaches the pod on that path: helm upgrade --no-hooks exits 0, the checksum/config annotation moves, strategy: Recreate stops the old pod, and the new one CrashLoopBackOffs on the refused value with no previous ReplicaSet to fall back to. Driven on a cluster, not reasoned about: three restarts in 90 seconds, and helm rollback <previous-revision> brings the pod back. Two consequences worth planning for. Run infrasigns config validate against the config you are about to apply whenever you use --no-hooks, or let the hooks run; helm template … | ... plus the binary is the same check the Job would have made. And if it does happen, the remedy is helm rollback, not a retried upgrade — a retry re-applies the same bytes. Closing this properly (moving the pre-flight onto the Deployment as an init container, so no helm flag can skip it) is a change to the pod's own shape and is tracked separately.

The pre-flight is what stops it, and that is worth knowing because it used to be migrate up's job and no longer is. Since #566 migrate up reads the file for database.dsn and the encryption key alone — deliberately, so a verdict about your monitoring configuration cannot stop a schema install — and it PRINTS the file's verdict rather than exiting on it. So a config the daemon will refuse still leaves migrate up at exit 0. config validate is the same rules with the opposite disposition: it exits non-zero, which is what fails the hook.

That ordering is what makes the next section's guarantee hold. The Deployment uses the Recreate strategy, so by the time the new pod's serve could refuse anything the old pod is already gone — a refusal there is CrashLoopBackOff with nothing monitoring. Everything decidable from the config alone is therefore decided in the hook, where a failure still leaves the previous Deployment in place — on any path that runs the hook. helm upgrade --no-hooks is the exception, and the paragraph above says what happens there and what to do about it.

Upgrading from a release installed before the split

If the release was installed by a chart older than the two-ConfigMap split, upgrade it with this chart and nothing else — do not delete or rename anything first. The split gave the Deployment's copy a new name (<release>-infrasigns-config) and left the historical <release>-infrasigns on the hook copy, and that assignment is what makes the upgrade work at all.

The reason, measured rather than argued: Helm does not store hook manifests in the release, so an older release holds a live <release>-infrasigns ConfigMap that its own stored manifest does not know about. The moment a chart emits that name as an ordinary resource, Helm refuses to adopt the live object:

Error: UPGRADE FAILED: Unable to continue with update: ConfigMap
"<release>-infrasigns" in namespace "<namespace>" exists and cannot be imported
into the current release: invalid ownership metadata; annotation validation
error: missing key "meta.helm.sh/release-name"

Adding meta.helm.sh/* annotations to the chart does not fix that — Helm reads ownership off the live object, not off the manifest being applied. Deleting the ConfigMap by hand fixes the upgrade and breaks the way back: the previous revision's Deployment mounts that same name, rollback runs no hooks, and neither revision can then start — both pods sit in FailedMount. Keeping the historical name on the hook copy avoids the whole problem, and the upgrade and a subsequent helm rollback to the pre-split revision were both driven end to end.

One consequence is worth knowing before you run it, because it is the one thing the split does not make cleaner on this path. The hook copy's delete policy is before-hook-creation, so the upgrade deletes and recreates <release>-infrasigns — and on a pre-split release that is the object the running pod has mounted. The window is short — the pre-split chart's own comment estimated ~100ms, and that figure is INHERITED rather than measured: it arrives verbatim from the comment this split replaced, where it stood beside "the probability is very low in practice" with nothing behind it, and nothing in this change timed it. Treat it as an order of magnitude. A pod only notices if it restarts inside the window, but the window is real until the upgrade rolls the pod onto <release>-infrasigns-config, and it re-opens on every later upgrade if you roll back to the pre-split revision and stay there. After the upgrade, nothing long-lived mounts the hook copy again.

Every existing release rolls its pod once on this upgrade. The rendered config.yaml is byte-identical, but the Deployment now mounts a differently-named ConfigMap, which moves the pod template. With Recreate that is one brief monitoring gap, the same as any other upgrade.

If you installed from an unreleased build of this change, the naming was briefly the other way round — the Deployment's copy was <release>-infrasigns and the hook copy <release>-infrasigns-hook — and upgrading from it is not blocked, but it leaves two traces worth knowing about. <release>-infrasigns-hook is orphaned permanently: Helm does not store hook manifests in the release, so nothing prunes it. The kubectl delete configmap,job -l app.kubernetes.io/instance=<release> command under Uninstall sweeps it, because every ConfigMap this chart renders carries that label; a by-name cleanup has to name it explicitly. And <release>-infrasigns should be deleted by that upgrade even though the pre-upgrade hook has just created it: it was an ordinary resource in the old release and is a hook in the new one, so Helm's apply phase has it in the old manifest and not the new one. That step follows from Helm's ownership rules rather than from a run of ours — unlike the released-chart path above, this one was not driven — and it costs nothing either way, because only the migration Job mounts that copy and the next upgrade's hook recreates it before that Job runs. No released chart ever produced either state.

Reading a failed migration Job

Two things about that Job are worth knowing before you need them.

kubectl logs job/<release>-infrasigns-migrate -n <namespace> prints the migrate container. The Job also carries a preflight init container, on every install (see below). When that is what failed, the plain command does not print its error: it defaults to the migrate container, which never ran, and answers that the container is still waiting to start (PodInitializing) rather than showing you the refusal. Name the init container instead:

kubectl logs job/infrasigns-migrate -n infrasigns              # migrate
kubectl logs job/infrasigns-migrate -n infrasigns -c preflight # the pre-flight

The Job's NAME depends on migration.enabled. With migrations managed out-of-band there is no migrate up container, so the Job is named <release>-infrasigns-preflight and the pre-flight is its only container — kubectl logs job/infrasigns-preflight -n infrasigns with no -c, for a release called infrasigns. kubectl get job -n <namespace> names whichever one this release rendered, which is the reliable way to read it: the name is the chart's fullname plus the suffix, truncated if that would exceed Kubernetes' 63-character label bound (see What the chart creates).

Read the logs before you retry. The Job's helm.sh/hook-delete-policy is before-hook-creation,hook-succeeded, so a failed Job is kept for debugging — and then deleted at the start of the next hook run. Re-running helm upgrade, which is the natural response to a failed one, destroys the Job and its pod logs before creating the replacement.

Configuration rules can tighten between releases

A release may make a previously-accepted configuration value invalid. Where the daemon would refuse to start on a value an earlier release accepted, the release notes call the change out as Breaking; where it warns and starts while config validate refuses — the disposition #569 takes — the note says so instead, and the value still stops a helm upgrade, because the hook runs config validate. Sweep for offending values before upgrading either way, because the two ways InfraSigns reads its configuration find out very differently.

Configuration in this chart's values is checked before the new Deployment starts: the Job's preflight init container runs the whole file-scope validator over the rendered config. A value it refuses fails the Job, the upgrade stops, and the previous Deployment keeps running. Its logs name the offending key and say why — see Reading a failed migration Job above, because retrying deletes them, and name the preflight container or you will read the one that never ran.

There is a second class of rule that gate does not cover, and on this chart it has no subject. Since #546 a deployment with sign-in configured also copies its config file's sources: into its own database document at start-up, and the store's write gate applies rules there that the file-scope validator does not — a class infrasigns config validate reports on and exits 1 over, with or without --strict-provisioning (see Pre-flight before an upgrade). That flag used to be what moved the exit code; it is accepted and implied now, so on a current binary it changes nothing. This chart renders no auth: section at all (grep templates/configmap.yaml), so nothing it can produce reaches those rules, on any values.

The preflight container carries the gate for them anyway, and what is conditional is the FLAGS rather than the container: it runs infrasigns config validate on every install, and adds --strict-provisioning only where config.auth is set — which is to say nowhere today — and --accept-unjudged only where preflight.acceptUnjudged is turned on, which is off by default. That split is deliberate on both halves, and it is about PINNED IMAGES rather than about this release's behaviour, because neither flag changes it. The bare config validate is in every released image, so the container cannot break an upgrade whose image.tag is pinned below this chart's appVersion; --strict-provisioning is new in #546, so an image pinned below it fails that container with unknown flag and stops an upgrade the deployment itself would have completed. Between those two points — an image at or after #546 and before the release that made the refusal unconditional — the flag is what moves the exit code, which is the only window in which passing it still buys anything. Confining the flag to the arm that already had it is what lets the container itself run everywhere — which it has to, because the file-scope refusals it stops (a bare $NAME, for one) reach every install and not only the ones with sign-in. A refusal in serve is the CrashLoopBackOff the section above describes, so the check belongs in the hook either way.

--accept-unjudged has a switch of its own, preflight.acceptUnjudged, defaulting off, and what it buys is on the OTHER side of the exit code. config validate answers 3 when the file is valid under every rule the run could apply and a set of rules was skipped — see Pre-flight before an upgrade for the three codes. Any non-zero exit in an init container stops the upgrade, so an unqualified 3 would fail installs over a check the command could not make rather than over anything wrong with the config. The flag turns that answer back into exit 0, leaving the explanation in the container's log where kubectl logs … -c preflight reads it. It is a flag rather than a shell || because the image is distroless and carries no shell, so the container's command: is exec form with nothing to write a || in.

It does NOT ride config.auth. The two flags carry image floors at different commits — --strict-provisioning is #546, this one is #579 — and a flag added to an arm raises that whole arm's floor, so "this arm already has a floor" is not an argument for a flag whose floor is newer. Both land in the same release here, so nothing pinned to a version tag sits between them; a pin to one of the per-commit sha-… images published on every merge to main can. Turning preflight.acceptUnjudged: true on therefore carries the #579 floor and is your decision rather than the chart's: leave it off and exit 3 stops the hook with the explanation in kubectl logs … -c preflight; turn it on once your image.tag is at or after the release carrying #579 and your gate would rather log the third verdict than stop on it. (What this does not close is --strict-provisioning's own floor: an image.tag pinned below #546 with config.auth set still fails the container with unknown flag. That trade is stated in the paragraph above and is unchanged.)

Exit 3 is unreachable on every values file this chart accepts, and it is worth knowing that it is a union rather than one condition. config validate answers 3 when the file declares auth.<provider>.client_id and this shell cannot resolve it, or when the file's own bytes are not YAML while the expanded text is — a ${VAR} written inside a flow collection, which needs no auth: key at all. The first arm has no subject here because this chart renders no auth: section. The second has none because the rendered config.yaml is hand-written template literals plus toYaml over your config: values, and toYaml is a YAML serializer: what it emits parses as written. (A file that simply fails to parse both ways is not the third verdict at all — it is exit 1, a refusal.) The flags are there so that growing an auth: section activates the first arm rather than needing it found — templates/configmap.yaml lists what such a block would owe.

Third-party CI keeps the distinction: if your own gate would rather stop on 3 than tolerate it, run config validate without the flag and read the code.

It has its own switch, preflight.enabled, defaulting true and independent of migration.enabled — which is what makes "on every install" above a statement rather than an aspiration, including for the operator who manages migrations out-of-band and is the one most likely to hand-write a config. Setting preflight.enabled: false removes it, deliberately; on such a deployment nothing in the chart stops an upgrade on a config the daemon will refuse, so run infrasigns config validate against the rendered config yourself.

Per-organization configuration stored in the database — the multi-tenant deployment — has no such gate. Those documents are validated when they are written, not when they are read, so one written before a rule tightened keeps rendering in the UI while the worker refuses to build a runtime for it: that organization goes quiet — no collection, no digests, no checks — and everything else keeps running. It surfaces in exactly two places, neither of them the tenant's own screens:

an error log, lease: cannot resolve runtime for schedule, backing off, which names the organization and the validator's key;
infrasigns_schedule_cycles_total{status="no_runtime"}, which carries no organization label — so it tells you that something is degraded, not which tenant.

The fix is always to correct the stored value, which means finding the affected organizations first. The release note for each tightened rule says what to look for.

This release (#569) treats a bare $NAME anywhere in the config file's values as a mistake: only ${NAME} is a reference, and a literal dollar is written $$. serve warns about it and starts — a rule that would refuse a file an earlier version accepted must not stop a daemon — while infrasigns config validate refuses it, so the preflight container is where you will meet it. That class reaches every install, with or without sign-in. Sweep the config this chart would render, rather than your values file — the $ may be coming from a default:

helm template infrasigns oci://ghcr.io/infrasigns/charts/infrasigns \
  -f my-values.yaml -s templates/configmap.yaml \
  | sed 's/\$\$//g' | grep -nE '\$([^{$]|$)'

The sed deletes the $$ escape before the search, so the reader who has already applied the remedy is not shown their own fix as a problem. -E and not grep -P: the Perl form is a GNU extension, so on macOS or busybox it fails to stderr and prints nothing — which reads exactly like a clean render. The pattern reads bytes and cannot see YAML, so it errs wide in two ways: a hit inside a # comment is not refused, and neither is a $ with nothing after it. It is a floor rather than a total, and A literal $ in a config-file value says what it does not see.

This release (#432) tightens two source-name rules — a length bound and a new reserved word. See Source names for both, and for what a rename leaves behind.

Breaking. A deployment with sign-in configured refuses to start unless security.encryption_key or security.encryption_key_file is set. Such a deployment holds other organizations' credentials, and without a key they sit in the clear in the database; the start-up error names the key and points at infrasigns secrets keygen.

A chart-managed install cannot reach that rule today. This chart renders no auth: section at all — grep templates/configmap.yaml — so every config it can produce is self-hosted, and the migration Job validates the rendered config before the new Deployment starts either way. The requirement binds a deployment running the image against a config file of its own. When the chart grows hosted support, the key joins the required key set beside the auth block.

Setting a key on a self-hosted install is not breaking and is worth doing — see INFRASIGNS_ENCRYPTION_KEY above for what it covers and what losing it costs.

The Deployment uses the Recreate strategy: the old pod stops before the new one starts, so expect a brief monitoring gap during upgrades. This is deliberate — the MVP runs a single scheduler worker, and Recreate avoids briefly running two.

Uninstall

helm uninstall infrasigns --namespace infrasigns

The Deployment's own ConfigMap (<release>-infrasigns-config) goes with the release — it is an ordinary resource. Hook resources are not deleted by helm uninstall, and two remain: the migration Job and the hook copy of the ConfigMap (<release>-infrasigns). Remove them manually if the namespace is not being reclaimed anyway:

# If you used a different namespace (-n flag) or release name (instance= value), update both below
kubectl delete configmap,job -n infrasigns -l app.kubernetes.io/instance=infrasigns

Replica count

replicaCount must be 0 or 1. InfraSigns schedules digest, trends, and health-check cycles from a single worker process. The chart schema enforces this — helm install fails immediately if replicaCount > 1. (The scheduler claims each cycle through a database lease that single-flights it, so multi-replica scale-out is a planned flag flip rather than a rewrite; today the chart caps at one worker.)

Use replicaCount: 0 to pause the service without uninstalling.

Graceful shutdown

On SIGTERM the process tears its components down in reverse startup order under a single 30s budget (totalShutdownTimeout). Each component gets at most 10s of that, minus whatever is still reserved for the components below it, so the ones that must finish their in-flight database writes before the process exits cannot be starved by a slow HTTP server or chat transport above them. The reservations are:

Component	Reserved	Why
Incident sink	5s	One incident write / prune call
Report sink	5s	One report write (its prune shares that budget)
Source-health recorder	5s	One health write
Lease scheduler	5s	Releasing the claimed schedule row of a cut-short cycle
Store (pool close)	2s	Closing idle connections and releasing waiters — see below

That is 22s reserved of the 30s, leaving 8s for everything unreserved (the HTTP server, the chat transports, the webhook and deploy handlers, the session reaper) to share, in LIFO order, before the reservations start binding.

The reservations make a write abandoned mid-shutdown unlikely, not impossible: a drainer that runs out of budget gives up on its background writes rather than cancelling them, and nothing downstream rescues one that it gave up on.

The store's own 2s is not a rescue and buys no waiting. Closing the pool (database/sql's DB.Close) closes the idle connections, stops new queries from starting, drops the goroutines waiting for a connection and stops the connection opener; it never blocks on a connection that is currently in use. Its documentation says otherwise — that it "waits for all queries that have started processing on the server to finish" — but the implementation does not do that, so do not size the reservation as if it did. An abandoned write survives only because closing the pool leaves its busy connection alone, and what it then races is process exit, which nothing here bounds. The value of the reservations is that they make abandonment less likely by giving each drainer its slot, not that an abandoned write is safe.

Worst case the pod needs about 35s: the 30s budget, plus up to one more incident write, because the incident sink waits for its aborted write to unwind instead of abandoning it. The chart therefore sets terminationGracePeriodSeconds: 40 — matching kubernetes' 30s default would let SIGKILL land in the middle of the final writes. If you override the grace period, keep it above 35s.

Enabling ServiceMonitor

If you run prometheus-operator or kube-prometheus-stack:

serviceMonitor:
  enabled: true
  interval: 60s
  labels:
    release: kube-prometheus-stack  # match your Prometheus CR's serviceMonitorSelector

Metrics reference

All application metrics are exported on /metrics under the infrasigns_ prefix (plus the standard Go runtime and process collectors). The full catalog — every metric name, type, label, and label value — lives on the Observability page, along with the /healthz and /readyz probe contracts.

Two things about the probes are worth knowing before you size a cluster around them. /readyz gates the pod on nothing at all — it answers 200 for every process that answers, and the fault, if there is one, is in the body's top-level status word (#507). So a 1/1 pod is not evidence that its database, sources or notification channels are reachable: the machine-readable signal is infrasigns_readiness, and if you take one alerting rule from this page, take infrasigns_readiness{kind="subsystem",name="database"} == 0. The reason it gates nothing is that this chart caps replicaCount at 1, so eviction can never move traffic to a healthy sibling — it only removes the last endpoint, taking the web UI that would explain the fault, /metrics and the webhook receiver with it.

Two alerts you may already have stop covering this pod, and this is the whole of what replaces them. The pod is now Ready from the moment its containers start, whatever it can or cannot reach, so it never enters the state either of these is written against. Both are severity: warning, both for: 15m, and both are read from the upstream kubernetes-mixin that kube-prometheus-stack ships rather than from memory:

Both are described here by the CONDITION they turn on, not quoted: the shipped expressions are longer than anything worth reproducing in a doc — KubePodNotReady is a union of three vectors wrapped in a topk that drops Job-owned pods and excludes SchedulingGated, and KubeDeploymentReplicasMismatch ands a second clause onto its comparison. Read them in the mixin; a quoted sub-expression here would go stale the first time upstream edits the rule, and it is the condition, not the text, that this section is about.

KubePodNotReady fires on a pod that is in phase Running and whose Ready condition is false. A pod held NotReady by a failing readiness probe used to satisfy it.
KubeDeploymentReplicasMismatch fires when a Deployment's desired replica count exceeds its available one, and only while nothing has rolled in the last 10 minutes (so an ordinary deploy does not trip it). An unready sole replica used to satisfy that too.

Neither can fire on an InfraSigns pod any more, and nothing in kube-prometheus-stack replaces them, because the fault is no longer expressed in any Kubernetes object. infrasigns_readiness == 0 is the replacement, and you have to add it yourself — the chart's serviceMonitor gets the series scraped and ships no PrometheusRule. Suggested alerts has a starting set, including the one rule that catches nothing polling /readyz at all.

The same fact reaches a GitOps controller. A pod that is always Ready keeps the Deployment's Available condition True and availableReplicas at 1, so helm upgrade --wait returns success and Argo CD assesses the Deployment as Healthy however broken the thing it monitors is. That follows from the Kubernetes object, which is measured; the Argo half is the mapping Argo documents, and — as GitOps says of everything in that section — was not driven against a real Argo CD installation. Before this a single unreachable source meant a correct install never became Ready at all, which --wait, an Argo health gate or an HPA all read as a failed install; first-install database reachability is still gated, by the pre-install migrate up hook and by serve migrating before it binds its listener. And the container carries a startupProbe on /healthz (periodSeconds: 5 × failureThreshold: 48 = 240s), because serve applies pending migrations before it listens: while a migration runs there is no HTTP server to answer, and without the startup probe liveness killed the pod at roughly 90 seconds and the migration restarted from the beginning. 240s is deliberately the same number as the migration Job's activeDeadlineSeconds — one budget for one piece of work — and CI asserts the two are equal. A migration slower than that is still killed; the answer for one of those is migration.enabled: false and applying it out of band, or raising both numbers together (and helm's own --timeout above them).

The ServiceMonitor above is the scrape mechanism for /metrics on a prometheus-operator cluster; on a plain Prometheus, add a static scrape config pointing at the Service.

Enabling Ingress

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: infrasigns.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: infrasigns-tls
      hosts:
        - infrasigns.example.com

Key values reference

Value	Default	Description
`existingSecret`	`""`	Required. Name of the Kubernetes Secret with sensitive env vars
`config.sources`	`[]`	Required. List of `{name, url}` Prometheus sources
`service.port`	`8080`	Kubernetes Service port; routes to the container via the `http` named port
`config.llm.provider`	`openai`	`openai`, `anthropic`, or `none` (deterministic engine, no API key)
`config.llm.model`	`gpt-4o-mini`	Model name passed to the LLM API; ignored with `none`
`config.notify.<channel>.feeds`	unset	Which producers reach one channel — any of `digest`, `trends`, `incidents`, `deploys`, `checks`. Unset means every feed, the behaviour of every release before it, and an explicit `null` in a second `-f` resets the key to that default; an empty list keeps the channel configured and silent. A repeated feed is refused at `helm install`/`helm upgrade`, matching the loader, rather than installing a config the pod then crashloops on. `config.notify.pagerduty.feeds` accepts only `incidents` and `checks` and its enum refuses the other three at install time too — a pager drops any message carrying no incident, so those scopes could never match. See Feed routing
`config.reports.digest.schedule`	`"0 8 * * *"`	Cron expression (UTC) for the daily digest
`config.reports.digest.stale_episode_after`	unset (`24h`)	Go duration. An open incident episode nothing has re-asserted for this long still appears on the digest but stops standing its verdict up. Applies to senders that repeat (Alertmanager, Grafana), never to CloudWatch alarms, which only notify on a state change; must be longer than `config.webhook.dedupe_window`. No "off" value — `0s`/negatives are rejected; set something very large (`"87600h"`) to keep the pre-#419 behaviour
`config.reports.trends.enabled`	`false`	Enable the periodic trends report
`config.reports.trends.schedule`	`"0 9 * * 1"`	Cron expression for the trends report
`config.reports.trends.window`	`"168h"`	Analysis look-back (Go duration); defaults to 7d
`config.reports.trends.timezone`	`"UTC"`	IANA timezone name for the trends schedule
`config.reports.retention_days`	unset	Prune archived reports after N days; `0`/unset keeps everything forever
`config.incidents.retention_days`	unset	Prune RESOLVED incident episodes after N days; open episodes are never pruned; `0`/unset keeps forever
`config.timeline.retention_days`	unset	Prune source timeline events (fetch failures/recoveries, check-verdict changes) after N days — and, at startup only, maintenance windows that stopped silencing anything that long ago; `0`/unset keeps forever
`config.source_health.retention_days`	unset	Prune a removed source's frozen per-source rows after N days — its health row and its severity reading (#358) alike (live sources keep updating both, so only removed ones are reaped); `0`/unset keeps forever
`config.webhook.dedupe_window`	unset	Suppress duplicate inbound alerts within this window (Go duration; app default `5m`)
`config.webhook.model`	unset	`llm.model` override for inbound-alert incident summaries
`config.heartbeat.enabled`	`false`	Set to `true` and provide `HEARTBEAT_URL` in the secret
`migration.enabled`	`true`	Run `migrate up` as a pre-install/pre-upgrade hook
`preflight.enabled`	`true`	Run `infrasigns config validate` against the rendered config as a pre-install/pre-upgrade hook, so a config the daemon will refuse stops the upgrade rather than the new pod. Independent of `migration.enabled`
`preflight.acceptUnjudged`	`false`	Pass `--accept-unjudged` to that check, so its third verdict (exit 3 — valid under every rule the run could apply, with a set of rules skipped) does not stop the upgrade. Requires an `image.tag` at or after the release carrying #579; on an older image the flag itself fails the container. See Upgrade
`replicaCount`	`1`	Must be 0 or 1
`serviceMonitor.enabled`	`false`	Create a ServiceMonitor for prometheus-operator
`image.tag`	`""`	Defaults to chart `appVersion`; override to pin a specific release