Skip to content

Post-deploy verification

Tell InfraSigns when you deploy, and after a settle delay it asks the LLM to compare your metrics from before and after: "did this deploy regress anything?" Works with any CI/CD that can send an HTTP request — no Kubernetes operator required.

Deploy verification requires a real LLM provider: the before/after verdict needs reasoning, so llm.provider: none with deploys.enabled: true is rejected at config validate.

Enabling

deploys:
  enabled: true
  source: prod        # which configured source's metrics verify deploys
  delay: "5m"         # settle delay before verifying (default 5m — let the
                      # rollout finish and crashes surface first)
  cooldown: "10m"     # per-service suppression of rapid successive deploys
  mode: alert         # alert (default — you asked for it) | monitor (record only)
  model: ""           # optional llm.model override

The endpoint shares webhook.token (it is a push ingress like alerts) — deploys.enabled without a configured token fails config validation (infrasigns config validate, the migration job, and serve alike).

Sending a deploy event

curl -X POST https://infrasigns.example.com/webhook/deploys \
  -H "Authorization: Bearer $WEBHOOK_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "api",
    "version": "v2.1.0",
    "previous_version": "v2.0.9",
    "environment": "prod-eu"
  }'

service and version are required; the rest sharpens the prompt. As a GitHub Actions step:

- name: Notify InfraSigns
  run: |
    curl -sf -X POST "$INFRASIGNS_URL/webhook/deploys" \
      -H "Authorization: Bearer ${{ secrets.INFRASIGNS_WEBHOOK_TOKEN }}" \
      -H "Content-Type: application/json" \
      -d "{\"service\":\"api\",\"version\":\"${{ github.sha }}\"}"

Responses: 202 {"status":"scheduled","verify_at":…}; 200 {"status":"cooldown_suppressed"} for rapid successive deploys of the same service (the previous verification covers the current state); 400 / 401 / 503 as usual.

All four fields are yours to fill and InfraSigns validates none of their content, so since #335 each is neutralized the moment the event is accepted: line breaks and tabs collapse to a space, other control characters are dropped, surrounding whitespace is trimmed, and anything past 500 runes is truncated with an ellipsis — the cap the prompt has always applied, now applied to the delivered notification and the archived report as well, so a value cannot forge a line in either. Two consequences worth knowing:

  • A field that survives as nothing is empty. {"service": "\n"} is a missing service, answered 400, where it used to be accepted and delivered as a blank name.
  • The cooldown is keyed on the neutralized name. Two services whose names differ only in those characters, or only past the 500th rune, share one suppression window. This is deliberate: the alternative lets a caller mint unlimited variants of one service by appending a newline.

What happens

After delay, InfraSigns fetches the source's configured queries for the 30 minutes before the deploy and the window after it, prepends the deploy facts (service, old → new version, environment — always ours to supply, never left to prompt templating), and asks for a verdict with the same rationale-first, fail-closed contract as health checks.

Each list's header states the window it covers, because the two are not the same length: the baseline is a fixed 30 minutes while the after window is your configured delay. min/max are window extremes, so without that the longer baseline's wider range reads as a change.

Both metric lists are capped the same way a health check's is: at most the first 30 series of each query name individually, the rest folded into one aggregate line spanning every folded series' min and max, and at most 40 distinct query names. Each list also gets its own 160 KiB budget for the lines it prints individually, and folds further to stay inside it — so a wide baseline cannot starve the after list, and the two sides can legitimately list different numbers of series for one query name. Their per-name totals, stated on both sides, are what the comparison rests on. A series identity ending in (+7 more labels) had labels dropped; the prompt tells the model not to pair such series across the two lists on the strength of their labels matching. See LLM providers for both limits.

Unlike a health check, a deploy prompt states each query name's total series count on both sides whenever there is a baseline to compare against — not only when something was folded. A deploy verdict is a comparison, so the total is load-bearing every time: a crashlooping rollout that leaves 3 of 25 pods reporting folds nothing at a limit of 30, and without the totals the model would have to notice the difference by counting two lists of lines. The totals are not a rendering artifact, and the prompt says so — a query name whose total drops sharply really did lose series, and that is weighed rather than explained away. A total that rose, or a name that appears only in the after list, is explicitly not a regression. (A first deploy has no baseline and nothing to compare, so there the totals ride the fold exactly as a health check's do.)

The two lists are folded independently. What cannot be paired across the lists is the hidden remainder of a folded query name — those series are not listed, so the prompt tells the model to compare that name's totals and value ranges for them instead. The series a folded name still lists individually carry their labels on both sides and are paired by label like any other, and a name listed in full on both sides is complete on both sides and compared series by series throughout. That distinction matters because a per-series regression (two of four jobs newly erroring) is invisible in a total and a range that did not move. A deploy does replace pods, so instance labels routinely differ between the two lists; a changed label set is not by itself a regression.

The verdict is tri-state, like checks:

  • pass → "Deploy verified — api v2.1.0" (info),
  • fail → "Deploy REGRESSION — api v2.1.0" (warning),
  • error → the verification could not run (fetch failed, no post-deploy data, LLM outage or exhausted budget) — also notified: an unverifiable deploy is worth knowing about.

On the channels that render structured reports (Telegram, Slack, email, and the plain-text floor) the verdict is the same layout a digest gets: the service, the version it deployed and its environment (checkout-api v2.1.0 (prod-eu)) and when the deploy was accepted — minutes before the verdict describing it — a status-first verdict, the model's rationale (collapsed behind a tap on Telegram, shown in full on the other three), then one fact row naming what the deploy moved (Version: v2.0.9 → v2.1.0). That row appears only when the event carried a previous_version; without one the identity above already names the deployed version, so there is nothing left to add and the report has no fact rows at all. A regression or an unverifiable deploy draws a neutral accent on that row rather than the healthy ✅, which would say the opposite of the verdict above it.

The version is in the identity because that identity is what an email subject line and a Slack push preview show — without it, two verifications of the same service look the same at a glance.

Every verdict lands in the Reports archive (type deploy) with delivery receipts; in monitor mode it lands there silently.

Notes

  • Pending verifications are in-memory: a restart during the settle delay drops them (the deploy is still in the archive of your CI, not ours).
  • Keep cooldown at one second or more: verification IDs have second granularity, and a sub-second cooldown could let two verifications of one service land in the same second — the second verdict is dropped.
  • Each verification is one billable LLM call against llm.max_calls_per_day; on an exhausted budget the verdict is error, never a synthesized answer.
  • A slow-burn re-check (~24h after deploy) is a planned follow-up.