Webhook receiver

POST /webhook/alerts accepts alert payloads pushed by external systems — AlertManager, Grafana, and CloudWatch (via SNS) — deduplicates them, asks the LLM for an incident summary, and delivers it to your notification channels. It is the push counterpart to the pull-based digest: use it when your alerting system should drive notifications the moment something fires.

What it is not

Alerts only. The receiver takes alert payloads. It is not a way to get metrics into InfraSigns, and there is no other one — series are read by dialing out to a source's url (how InfraSigns reaches your metrics). So a deployment whose only ingest is this endpoint gets incident notifications and nothing else: no digest, no trends, no health cards, because there is no source to collect from.

Process-level, not per-tenant. One webhook.token enables the route for the whole process, and every incident it records belongs to the default organization (RecordIncident(ctx, store.DefaultOrgID, …) in internal/cli/incidentsink.go). On a self-hosted deployment that is exactly right — the operator owns the process and the organization is theirs. On the hosted service it means there is no per-organization receiver: a tenant can neither set the token nor have their pushed alerts land in their own history, so the add-source wizard does not offer this route there.

Enabling

The endpoint accepts alerts only when a token is configured. The ROUTE is always mounted (#558): without a token it answers every request 401 {"error":"unauthorized"} — byte for byte what a configured receiver answers a wrong bearer with, so the mux does not report whether you have enabled the receiver. Nothing is processed on that arm. The daemon logs a warning at startup naming the key, which is where you diagnose it:

webhook:
  token: <bearer token, min 16 chars>   # required to enable the endpoint
  dedupe_window: "5m"                   # optional; Go duration, default 5m
  model: ""                             # optional llm.model override for summaries

Every request must carry Authorization: Bearer <token> (constant-time comparison; 401 otherwise).

Helm deployments provide the token as WEBHOOK_TOKEN in existingSecret — the chart deliberately rejects a config.webhook.token value; dedupe_window and model go under config.webhook (see Kubernetes).

Supported formats

The payload format is auto-detected; you can also force it with the X-Infrasigns-Source header (alertmanager, grafana, or cloudwatch).

Format	Detection	Fields read
AlertManager	`alerts[]` array present	`labels.alertname`, `status` (`firing`/`resolved`), `labels.severity`, all labels, annotations, `startsAt`/`endsAt`
Grafana 9+	same payload as AlertManager	identical (the `grafana` header hint maps to the AlertManager parser)
CloudWatch SNS	SNS envelope `{"Type":"Notification","Message":"…"}`	`AlarmName`, `AlarmDescription`, `NewStateValue` (`OK`→resolved, `ALARM`→firing critical, other→firing warning), `NewStateReason`, `StateChangeTime`, `Region`

Notes: CloudWatch INSUFFICIENT_DATA transitions are skipped entirely (they would poison the dedupe window); other SNS message types (SubscriptionConfirmation etc.) are acknowledged and ignored; an unrecognizable body is a 400.

Deduplication

Repeat deliveries of the same firing alert within dedupe_window are suppressed. The dedupe key is the source plus the alert name plus all of its labels, so alerts that differ in any label are distinct events. Resolved alerts always pass through — a resolution is never suppressed — and a resolution clears the alert's dedup entry: the episode provably ended, so a re-fire inside the window is a new episode and notifies again.

A suppressed repeat never reaches the incident archive either, so it does not refresh the open episode's last-re-assert either. That is why reports.digest.stale_episode_after must be longer than dedupe_window — a config where it is not is rejected at load.

Deduplication happens asynchronously after the request is accepted, so a fully-duplicate delivery still receives 202 — suppression is visible in the logs and in the absence of a notification, not in the HTTP response.

The window is in-memory: a restart clears dedup state (worst case, one duplicate notification). Keep the window shorter than the legitimate recurrence period of distinct episodes; re-sends of one long-firing episode are better rate-limited at the sender (AlertManager repeat_interval — remember each accepted delivery costs an LLM summary).

What happens after a payload is accepted

Alerts are split into a firing group (deduplicated) and a resolved group (always delivered). For each non-empty group InfraSigns asks the LLM for a summary (using webhook.model if set) and sends Incident Alert — <source> (<N> alert(s)) — with a — RESOLVED suffix for the resolved group — to every channel routed to the incidents feed, which by default is every configured channel (feed routing). If the LLM call fails, a plain-text fallback (<N> alert(s) from <source>) is sent instead: delivery never depends on the LLM.

On the channels that render structured reports (Telegram, Slack, email, and the plain-text floor) the message is the same layout a digest gets: the source and when the batch fired, a status-first verdict, the summary in a collapsible block, then one row per alert — its delivered severity, its identifying labels (everything except alertname and severity, which the row already says), and its summary annotation, falling back to description. Because the alerts reach the reader as rows, the model is asked for 2-3 sentences of analysis rather than a recap of the batch.

When an alert's own severity: label is not the tier it was delivered at — a p1 under the default reading, or any label your severity_aliases table folds — the row names it: DiskFull {instance=db-01:9100} (sender reported: p1). Without that, two alerts on a non-standard scheme that fold to the same tier are indistinguishable in the notification. A label that spells its tier's own name is not repeated — the comparison ignores case and padding, so Critical is silent beside critical. An abbreviation the product understands but that is not the name, warn, still shows: it is a different word, and the archive names it the same way.

The verdict comes from the receiver, not from counting the summary's claims. A firing group counts its alerts — 3 alerts firing, never "All healthy", and the count is the whole batch whatever mix of severities it holds, so it always matches the list beside it. Alerts at severity: info are drawn with a neutral accent rather than the green one an info-tier observation gets in a digest: they are firing. A resolved group reads Resolved, and its rows are drawn as ended (green) even when the alerts they describe were critical — a resolution should not carry the visual weight of a live incident.

The period is the span the batch covers: a single moment when every alert reported the same one, and a range (3 Aug 2026, 04:12 UTC – 6 Aug 2026, 09:00 UTC) when they differ, which is what a flapping alert re-firing alongside fresh ones looks like. A batch where no alert reported a startsAt renders no period rather than claiming the time the message was built.

Each channel bounds the row list against its own budget and appends a …and N more signals note. PagerDuty renders no report — its push notification is the summary prose, unchanged — but the alert rows, the verdict and the period reach it as payload.custom_details, so the on-call sees the batch in the alert itself (see Notifications).

Severity of the firing group is the maximum across its alerts: critical → Critical, warning/warn → Warning, info → Info. The label is read case-insensitively and whitespace-trimmed, so Critical and CRITICAL count. Unknown non-empty values (e.g. p2, high) and an alert carrying no severity: label at all are treated as Warning — conservative rather than silenced. Resolved notifications are always Info.

If your fleet uses a scheme InfraSigns does not recognize, this is the path where it costs the most, because it is the only one that carries a SENDER-SUPPLIED severity to PagerDuty (a health check pages too, but at the tier its own severity: sets — your config, not a pushed label) and PagerDuty maps only Critical to a PagerDuty critical: a fleet on severity: p1 pages at warning urgency, on every alert, until the vocabulary is fixed. Fixing it does not mean rewriting your alert rules: declare the scheme once with severity_aliases and a pushed p1 is read as critical here, pages as critical, and stops being reported as unrecognized. This receiver is built once per process, so the table it reads is the deployment's own — on the hosted service that is the platform's, not your organization's. Everything below is about the labels you have not aliased. You do not have to notice those by hand — a request carrying such a label logs a WARN line prefixed webhook: and containing the phrase unrecognized severity labels rank as warning, with the labels themselves, and increments infrasigns_alerts_unrecognized_severity_total{path="webhook",source}.

source names the fleet, not the payload format (#350): it is the configured source name the alerts attribute to, resolved by the same alert_labels matching that attributes an incident. So a request carrying two fleets' alerts raises the line once for each, naming each fleet's own vocabulary, and increments the counter once per fleet — which means summing the counter over source counts request-slices rather than requests, and is not comparable with infrasigns_webhook_requests_total. The counter is not throttled and is the surface to alert on; the log line is throttled to at most once per source per 24h, the same cadence the collection path uses.

An alert whose labels match no source's alert_labels, or match several, falls back to the detected payload format — in practice alertmanager, since a CloudWatch payload carries no severity: label at all (the receiver derives the tier from the alarm state) and an unrecognized payload yields no alerts to signal about.

This signal names a fleet only on a real alert_labels match. Incident attribution has a shortcut — one configured source with no alert_labels anywhere claims every alert, so a zero-config install still gets a populated timeline — and this signal deliberately does not inherit it. Naming a fleet that no label identified would print a scrape source's name over labels it never sent, and it would fold two AlertManager fleets pushing to a single-source install back onto one key, which is the very thing #350 exists to separate. So an install that configures no alert_labels keeps reporting alertmanager here, exactly as before — this attribute does not change for you; configure alert_labels on the sources you want named. See the severity: label for the full reading, including the two labels it deliberately stays quiet about.

These labels are still operator-facing only. The Unmapped severity labels your sources send card on Settings (#358) reports unmapped labels per source, but it can only report what a collection cycle saw: #350 gave this signal the fleet's name, not a persisted per-source reading, and the receiver is process-global rather than part of an organization's runtime. For a fleet that only pushes, an empty card says nothing about whether its vocabulary is understood — the WARN line and the counter above remain the way to find out.

One thing a pushed alert does surface: the label it reported is recorded with the episode and shown beside the severity badge on the Incidents rows and detail page whenever it differs from the tier — so a p1 fleet reads warning sender reported: p1 there rather than a bare warning. That names your vocabulary back to you; it does not diagnose it, because a label you deliberately aliased renders the same way (critical sender reported: p1). Whether a label is unmapped is still the WARN line and the counter.

Every notified group is also mirrored into incident history (the dashboard's "Recent incidents" card): one episode per alert identity, opened by the firing notification and closed in place by the matching resolved one, together with the LLM summary and the per-channel delivery results (failed sends included). Only notified events are recorded — dedupe-suppressed re-fires never touch history — and the write is asynchronous and ordered, so a slow LLM or notifier never costs the history row.

A resolved notification with no open episode to close still lands. When it reports no start of its own — a CloudWatch alarm returning to OK carries only the moment it cleared — it is read as the end of the most recent recorded episode of the same alert, provided that episode ended within the last 7 days. That covers the two ways such a notification turns up: the episode was closed by hand and the alarm cleared afterwards, or the same resolution was delivered more than once (resolved alerts are deliberately never deduplicated, so every retry reaches the store). The recorded episode keeps its severity, the label the sender reported, its name and its real start, and takes the resolution's own end time, summary and delivery receipts.

The row remembers that this happened, and the web archive says so: such an episode's duration is shown as an upper bound ("lasted at most …") rather than as a measurement, because the alert stopped firing at some unknown point between the two recorded moments. See durations the archive is not measuring.

A resolution that does report a start is never read as another episode's end. It goes to the episode it names, and if it names none — an Alertmanager restart re-derives an alert's start time, so a resolution arriving after one can name an episode nothing recorded — it is written as its own row. That is a duplicate row in the archive, and it is the deliberate choice: for a sender that reports a start the row is at least correct, where reading it as an older episode's end would overwrite an episode that was recorded correctly and lose this one entirely.

Two more bounds on the merge. A resolution arriving more than 7 days after the alert's last recorded episode ended is written as its own row, so the archive shows an episode of unknown length rather than a week-old one silently stretched. And a resolution that reports no end time either is never read as another episode's end: with nothing to measure the 7 days from except the moment we received it, the window would measure delivery latency instead of the gap between two episodes.

An archived episode absorbs at most one such resolution. The 7 days are measured from the episode's recorded end, and absorbing a resolution moves that end forward — so without a limit a run of alerts whose firings never arrived would keep extending one row indefinitely, and the archive would show a single episode that appeared to last for weeks. Worse, retention deletes archived episodes by when they ended, so that row would keep sliding out of reach of it: the one row misrepresenting its own duration would also be the one row your retention setting never removed. Once an episode has absorbed a resolution, a later one starts a row of its own — it is never read as the end of some older episode further back, which would nest one recorded episode inside another. Repeat deliveries of the same resolution are not affected: they carry the same end time, so they keep landing on the same row however many times the sender retries, and they do not use up that row's one merge.

A merge never moves a recorded end backwards. If the episode already ended later than the resolution reporting it, the recorded end stands and only the summary and receipts refresh — so a delivery that arrives late, or out of order, cannot make an episode look shorter than the store already knows it was.

Each recorded incident is additionally attributed to a configured source when its labels match that source's alert_labels (or automatically, in a single-source install) — attributed incidents appear on the source's detail-page timeline under the Incidents chip. See incident attribution.

Responses

Code	Meaning
`202 {"status":"accepted","received":"N"}`	N parsed alerts accepted for async processing (duplicates may then be suppressed silently)
`200 {"status":"ok","received":"0"}`	payload valid but yielded no alerts (e.g. `INSUFFICIENT_DATA`, non-Notification SNS types, empty `alerts[]`)
`400`	unreadable body or unrecognized payload structure
`401`	missing/invalid bearer token
`413`	body over 1 MiB
`503`	daemon is shutting down

AlertManager example

route:
  receiver: infrasigns
  group_by: [job]        # group related alerts into a single delivery
  group_wait: 10s
  group_interval: 1m
  # repeat_interval: keep generous (default 4h) — every re-send costs an LLM summary

receivers:
  - name: infrasigns
    webhook_configs:
      - url: http://infrasigns:8080/webhook/alerts
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials: <your webhook.token>

The URL above is a compose service name on a private network, which is why it is http. The rule, not the example: the daemon itself serves plain HTTP on port 8080 and terminates no TLS of its own, so http://…:8080/webhook/alerts is the right address wherever the hop stays inside a network you control — and anything crossing one you do not needs a TLS terminator in front of it, addressed as https://<your-host>/webhook/alerts with no port. The bearer token travels with every alert, so on that path the scheme is the only thing keeping it off the wire in clear.

For Grafana, point a webhook contact point at the same URL with the same bearer token. For CloudWatch, subscribe the endpoint to the SNS topic your alarms publish to.

Observability

The receiver feeds three Prometheus counters (see the metrics catalog):

infrasigns_webhook_requests_total{status} — one per inbound request; status is accepted, unauthorized, invalid, empty, or rejected (shutting down).
infrasigns_webhook_alerts_total{outcome} — one per alert inside accepted requests; outcome is firing, resolved, or deduped.
infrasigns_alerts_unrecognized_severity_total{path="webhook",source} — one per accepted request per attributed source in it carrying a severity: label InfraSigns does not recognize (see Severity above), so a request from two fleets increments two series and the sum over source is not a request count. Shared with the collection path, which is what the path attribute separates.

The receiver reports no /readyz subsystem, and deliberately not (#558). It used to carry a webhook key there, and that entry could only ever read ok while the endpoint was reachable — the receiver stops in shutdown, when /readyz has already gone — so it gated nothing; what it did do is appear only when webhook.token was set, which made an unauthenticated poll of readiness a read of whether you had enabled the receiver, the same bit the unconditional mount had just removed from the mux. The counters above are where receiver traffic is visible; status="unauthorized" is now recorded on the tokenless arm too, so /metrics does not report the bit either.