Webhook receiver

POST /webhook/alerts accepts alert payloads pushed by external systems — AlertManager, Grafana, and CloudWatch (via SNS) — deduplicates them, asks the LLM for an incident summary, and delivers it to your notification channels. It is the push counterpart to the pull-based digest: use it when your alerting system should drive notifications the moment something fires.

Enabling

The endpoint exists only when a token is configured — without one the route is not mounted at all (the daemon logs a warning at startup):

webhook:
  token: <bearer token, min 16 chars>   # required to enable the endpoint
  dedupe_window: "5m"                   # optional; Go duration, default 5m
  model: ""                             # optional llm.model override for summaries

Every request must carry Authorization: Bearer <token> (constant-time comparison; 401 otherwise).

Helm deployments provide the token as WEBHOOK_TOKEN in existingSecret — the chart deliberately rejects a config.webhook.token value; dedupe_window and model go under config.webhook (see Kubernetes).

Supported formats

The payload format is auto-detected; you can also force it with the X-Infrasigns-Source header (alertmanager, grafana, or cloudwatch).

Format	Detection	Fields read
AlertManager	`alerts[]` array present	`labels.alertname`, `status` (`firing`/`resolved`), `labels.severity`, all labels, annotations, `startsAt`/`endsAt`
Grafana 9+	same payload as AlertManager	identical (the `grafana` header hint maps to the AlertManager parser)
CloudWatch SNS	SNS envelope `{"Type":"Notification","Message":"…"}`	`AlarmName`, `AlarmDescription`, `NewStateValue` (`OK`→resolved, `ALARM`→firing critical, other→firing warning), `NewStateReason`, `StateChangeTime`, `Region`

Notes: CloudWatch INSUFFICIENT_DATA transitions are skipped entirely (they would poison the dedupe window); other SNS message types (SubscriptionConfirmation etc.) are acknowledged and ignored; an unrecognizable body is a 400.

Deduplication

Repeat deliveries of the same firing alert within dedupe_window are suppressed. The dedupe key is the source plus the alert name plus all of its labels, so alerts that differ in any label are distinct events. Resolved alerts always pass through — a resolution is never suppressed — and a resolution clears the alert's dedup entry: the episode provably ended, so a re-fire inside the window is a new episode and notifies again.

A suppressed repeat never reaches the incident archive either, so it does not refresh the open episode's last-re-assert either. That is why reports.digest.stale_episode_after must be longer than dedupe_window — a config where it is not is rejected at load.

Deduplication happens asynchronously after the request is accepted, so a fully-duplicate delivery still receives 202 — suppression is visible in the logs and in the absence of a notification, not in the HTTP response.

The window is in-memory: a restart clears dedup state (worst case, one duplicate notification). Keep the window shorter than the legitimate recurrence period of distinct episodes; re-sends of one long-firing episode are better rate-limited at the sender (AlertManager repeat_interval — remember each accepted delivery costs an LLM summary).

What happens after a payload is accepted

Alerts are split into a firing group (deduplicated) and a resolved group (always delivered). For each non-empty group InfraSigns asks the LLM for a summary (using webhook.model if set) and sends Incident Alert — <source> (<N> alert(s)) — with a — RESOLVED suffix for the resolved group — to every configured channel. If the LLM call fails, a plain-text fallback (<N> alert(s) from <source>) is sent instead: delivery never depends on the LLM.

On the channels that render structured reports (Telegram, Slack, email, and the plain-text floor) the message is the same layout a digest gets: the source and when the batch fired, a status-first verdict, the summary in a collapsible block, then one row per alert — its delivered severity, its identifying labels (everything except alertname and severity, which the row already says), and its summary annotation, falling back to description. Because the alerts reach the reader as rows, the model is asked for 2-3 sentences of analysis rather than a recap of the batch.

When an alert's own severity: label is not the tier it was delivered at — a p1 under the default reading, or any label your severity_aliases table folds — the row names it: DiskFull {instance=db-01:9100} (sender reported: p1). Without that, two alerts on a non-standard scheme that fold to the same tier are indistinguishable in the notification. A label that spells its tier's own name is not repeated — the comparison ignores case and padding, so Critical is silent beside critical. An abbreviation the product understands but that is not the name, warn, still shows: it is a different word, and the archive names it the same way.

The verdict comes from the receiver, not from counting the summary's claims. A firing group counts its alerts — 3 alerts firing, never "All healthy", and the count is the whole batch whatever mix of severities it holds, so it always matches the list beside it. Alerts at severity: info are drawn with a neutral accent rather than the green one an info-tier observation gets in a digest: they are firing. A resolved group reads Resolved, and its rows are drawn as ended (green) even when the alerts they describe were critical — a resolution should not carry the visual weight of a live incident.

The period is the span the batch covers: a single moment when every alert reported the same one, and a range (3 Aug 2026, 04:12 UTC – 6 Aug 2026, 09:00 UTC) when they differ, which is what a flapping alert re-firing alongside fresh ones looks like. A batch where no alert reported a startsAt renders no period rather than claiming the time the message was built.

Each channel bounds the row list against its own budget and appends a …and N more signals note. PagerDuty renders no report — its push notification is the summary prose, unchanged — but the alert rows, the verdict and the period reach it as payload.custom_details, so the on-call sees the batch in the alert itself (see Notifications).

Severity of the firing group is the maximum across its alerts: critical → Critical, warning/warn → Warning, info → Info. The label is read case-insensitively and whitespace-trimmed, so Critical and CRITICAL count. Unknown non-empty values (e.g. p2, high) and an alert carrying no severity: label at all are treated as Warning — conservative rather than silenced. Resolved notifications are always Info.

If your fleet uses a scheme InfraSigns does not recognize, this is the path where it costs the most, because it is the only one that carries a SENDER-SUPPLIED severity to PagerDuty (a health check pages too, but at the tier its own severity: sets — your config, not a pushed label) and PagerDuty maps only Critical to a PagerDuty critical: a fleet on severity: p1 pages at warning urgency, on every alert, until the vocabulary is fixed. Fixing it does not mean rewriting your alert rules: declare the scheme once with severity_aliases and a pushed p1 is read as critical here, pages as critical, and stops being reported as unrecognized. This receiver is built once per process, so the table it reads is the deployment's own — on the hosted service that is the platform's, not your organization's. Everything below is about the labels you have not aliased. You do not have to notice those by hand — a request carrying such a label logs a WARN line prefixed webhook: and containing the phrase unrecognized severity labels rank as warning, with the labels themselves, and increments infrasigns_alerts_unrecognized_severity_total{path="webhook",source} on every such request. The counter is not throttled and is the surface to alert on; the log line is throttled to at most once per payload format per 24h, which is coarser than it sounds — two AlertManager fleets pushing to one endpoint share that slot, because the signal is keyed on the payload format rather than on the sender (#350 tracks the sharper key). In practice source is always alertmanager: a CloudWatch payload carries no severity: label at all — the receiver derives the tier from the alarm state — so that format cannot produce this series, and an unrecognized payload yields no alerts to signal about. See the severity: label for the full reading, including the two labels it deliberately stays quiet about.

That same missing key is why these labels are operator-facing only. The Unmapped severity labels your sources send card on Settings (#358) reports unmapped labels per source, but it can only report what a collection cycle saw: a pushed alert is not attributed to a source, so nothing arriving here reaches that card. For a fleet that only pushes, an empty card says nothing about whether its vocabulary is understood — the WARN line and the counter above remain the way to find out.

One thing a pushed alert does surface: the label it reported is recorded with the episode and shown beside the severity badge on the Incidents rows and detail page whenever it differs from the tier — so a p1 fleet reads warning sender reported: p1 there rather than a bare warning. That names your vocabulary back to you; it does not diagnose it, because a label you deliberately aliased renders the same way (critical sender reported: p1). Whether a label is unmapped is still the WARN line and the counter.

Every notified group is also mirrored into incident history (the dashboard's "Recent incidents" card): one episode per alert identity, opened by the firing notification and closed in place by the matching resolved one, together with the LLM summary and the per-channel delivery results (failed sends included). Only notified events are recorded — dedupe-suppressed re-fires never touch history — and the write is asynchronous and ordered, so a slow LLM or notifier never costs the history row.

Each recorded incident is additionally attributed to a configured source when its labels match that source's alert_labels (or automatically, in a single-source install) — attributed incidents appear on the source's detail-page timeline under the Incidents chip. See incident attribution.

Responses

Code	Meaning
`202 {"status":"accepted","received":"N"}`	N parsed alerts accepted for async processing (duplicates may then be suppressed silently)
`200 {"status":"ok","received":"0"}`	payload valid but yielded no alerts (e.g. `INSUFFICIENT_DATA`, non-Notification SNS types, empty `alerts[]`)
`400`	unreadable body or unrecognized payload structure
`401`	missing/invalid bearer token
`413`	body over 1 MiB
`503`	daemon is shutting down

AlertManager example

route:
  receiver: infrasigns
  group_by: [job]        # group related alerts into a single delivery
  group_wait: 10s
  group_interval: 1m
  # repeat_interval: keep generous (default 4h) — every re-send costs an LLM summary

receivers:
  - name: infrasigns
    webhook_configs:
      - url: http://infrasigns:8080/webhook/alerts
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: <your webhook.token>

For Grafana, point a webhook contact point at the same URL with the same bearer token. For CloudWatch, subscribe the endpoint to the SNS topic your alarms publish to.

Observability

The receiver feeds three Prometheus counters (see the metrics catalog):

infrasigns_webhook_requests_total{status} — one per inbound request; status is accepted, unauthorized, invalid, empty, or rejected (shutting down).
infrasigns_webhook_alerts_total{outcome} — one per alert inside accepted requests; outcome is firing, resolved, or deduped.
infrasigns_alerts_unrecognized_severity_total{path="webhook",source} — one per accepted request carrying a severity: label InfraSigns does not recognize (see Severity above). Shared with the collection path, which is what the path attribute separates.

The receiver also reports readiness under the webhook key in /readyz.