Skip to content

Notifications

Telegram

Create a bot

  1. Open Telegram and start a chat with @BotFather
  2. Send /newbot and follow the prompts
  3. Copy the token — it looks like 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11

Get your chat ID

Personal chat:

  1. Start a chat with your bot and send any message
  2. Open https://api.telegram.org/bot<TOKEN>/getUpdates
  3. Find "chat":{"id":123456789} — that's your chat ID

Group or channel:

  1. Add the bot to the group/channel and send a message
  2. Open the same URL — group IDs are negative (e.g. -100123456789)

Configure

notify:
  telegram:
    token: "123456:ABC-..."
    chat_id: "-100123456789"

Message format

Digests, incident alerts, trends reports and inbound-webhook alert summaries render as HTML (parse_mode=HTML) — a status-first verdict header, worst-first severity-accented signals, and a footer. On the webhook path each signal is one inbound alert, carrying the severity it was delivered at, its identifying labels and its summary annotation (falling back to description) — see Alert webhooks. A signal grounded in a single engine finding may also show an inline value column. It comes in two shapes: a movement, e.g. 2.4% → 10.1% ↑, and a threshold breach's reading-against-its-bar, described below. A movement's two numbers are that finding's own — a baseline anomaly shows its typical baseline → its recent peak (or trough), a trend shows its early → late window average — never a separate reading of the series taken at render time, which is what used to let the column disagree with the text beside it (#309). Two consequences worth knowing. The pair mixes aggregates of different kinds on an anomaly row (a baseline mean on the left, a recent extreme on the right), so read it as "normally about X, recently as far as Y" rather than as two readings a day apart; the claim's own prose names both. And with a real LLM the model chooses which finding a claim cites, so the column is only as well-matched as that choice — the numbers themselves are always an engine finding's, never the model's. Under llm.provider: none the match is exact, one claim per finding.

A static threshold breach shows a column of a different shape — 95.4% (limit 90%): one reading against the bar it met, with no arrow, because a breach observes a single sample and drawing it as 90 → 95.4 ↑ would invent a movement over time that nothing measured (#320). The bar shown is always the threshold that actually fired — crit: when a value breaches both — so on a digest it may be the critical bar beside a row the model chose to label a warning (the severity is the model's assessment; the numbers are the engine's).

A signal renders prose-only — no column — when it cannot be tied to a single finding: it spans several metrics, cites none, or is about a series as a whole rather than one observation. Incident alerts can show the same columns as a digest (#323): the engine's findings are grounded evidence there too, so a breach that coincides with a firing incident arrives with its reading and its bar rather than with prose alone. An incident row about an alert rather than a finding should carry no column — an alert payload holds no engine-computed numbers — and the prompt asks the model to cite nothing for such a claim; as everywhere else, which finding a claim cites is the model's choice, so that is an instruction rather than a guarantee. Two paths are prose-only by construction, having nothing citable at all: the inbound-webhook alert summary, and a critical cycle whose source produced no metric series and therefore no engine findings.

The unit suffix comes from a query's unit: or a cloud source's curated metric definition (#237); values with no known unit render bare. When there are more signals than fit the 4096-character limit, the least severe are dropped and a …and N more signals note is appended (never a mid-tag cut). At most 100 signals are considered for a single message even when more would fit — a bound on the fit search rather than on the report, since a webhook batch is sized by the sender; the same note carries the rest. A message the API cannot parse falls back to the plain-text floor. Other notifications (heartbeat, ad-hoc alerts) are plain text, truncated with ... if over the limit.

Threaded incidents

A resolution arrives as a reply to the message that announced the incident, so a busy chat reads as episodes rather than as a flat run of firing and resolved messages with no way to tell which resolution belongs to which alert. It needs no configuration and works in every chat type, including a private chat with the bot.

Threading is per episode, and an episode is the sender's own grouping — an Alertmanager alert group (its groupKey, or its group labels when the sender omits one), a CloudWatch alarm, or a single health check. It is not per individual alert: a group is delivered as one message per phase, so the alerts inside one group share a thread. A partial resolution — one alert of the group recovering while others still fire — replies into the thread without ending it, so the episode stays open until the whole group is clear. A sender that supplies no grouping at all degrades to one thread per payload format.

Digests, trend reports and deploy verdicts have no lifecycle and are always posted standalone.

A resolution ends the episode: the next incident on that group opens a fresh top-level message instead of being buried as a reply under the resolved one. A firing whose resolution is never delivered is reset after 24h idle.

The roots are held in memory, so a restart re-roots open episodes — and in the hosted service, so does saving the organization's configuration, a plan change, or an hour of inactivity. If the message being replied to was deleted, the resolution is still delivered; it simply arrives standalone. Durable thread tracking, and forum topics (a richer grouping that needs a supergroup with Topics enabled), are future work.

Tip

The same bot can also answer questions — see the Q&A bot, which reuses notify.telegram.token (Telegram) or notify.slack.token (Slack) by default.

Slack

Slack has two transports. Pick ONE with mode:

mode Delivery Threading Needs
webhook Incoming webhook no a webhook URL
api Web API (chat.postMessage) yes a bot token (chat:write) + channel ID

mode is required whenever any Slack field is set; the fields for the unselected transport must be empty. Omit the whole slack block to disable it.

webhook — incoming webhook (simple)

  1. Create an incoming webhook for your workspace: https://api.slack.com/messaging/webhooks (choose the channel that should receive digests and incident alerts).
  2. Configure:
notify:
  slack:
    mode: webhook
    webhook_url: '${SLACK_WEBHOOK_URL}'

webhook_url must be an absolute http/https URL with a host. A value with no host (https://:443/services/…) is not inert — it posts to the daemon's own localhost instead of to Slack — so the channel refuses it at send, and self-hosted /readyz reports it at the probe. config validate does not: this key is checked for presence and never parsed. In the hosted service neither the org-config save nor readiness reports it either, so a tenant's hostless webhook is refused only at send. The error never echoes the URL — an incoming-webhook URL is itself a credential, since anyone holding it can post to the channel.

api — Web API (threaded incidents)

The Web API transport posts an episode's follow-ups — later firing batches and the resolution — as thread replies to the first firing message, keeping one incident's activity in one thread instead of scattered across the channel. An episode is the sender's own grouping (an Alertmanager alert group, a CloudWatch alarm) or a single health check, not an individual alert — see Threaded incidents for what that means in practice. It needs a bot token, not a webhook:

  1. Create a Slack app, add the chat:write bot scope, install it to your workspace, and copy the bot token (xoxb-…).
  2. Invite the bot to the target channel and copy the channel ID (e.g. C0123ABCD).
  3. Configure:
notify:
  slack:
    mode: api
    token: '${SLACK_TOKEN}'
    channel: 'C0123ABCD'

Threads are keyed per episode and held in memory, so a restart re-roots open episodes — and in the hosted service, so does saving the organization's configuration, a plan change, or an hour of inactivity. A resolution ends the thread, so the next incident on that episode opens a fresh top-level message; a firing whose resolution is never delivered is reset after 24h idle. Durable thread tracking is future work. Telegram threads the same way, through replies.

Email (SMTP)

Email is a fallback channel for teams that do not use Telegram or Slack. It sends an HTML message with a plain-text fallback over SMTP with STARTTLS.

notify:
  email:
    smtp_host: smtp.example.com
    smtp_port: 587                     # optional; defaults to 587 (STARTTLS submission)
    smtp_user: '${SMTP_USER}'          # optional; omit both user and password for an unauthenticated relay
    smtp_password: '${SMTP_PASSWORD}'
    from: "InfraSigns <[email protected]>"
    to:
      - [email protected]
      - [email protected]
  • The channel is enabled when smtp_host is set; an empty smtp_host disables it.
  • smtp_user and smtp_password must be set together, or both left empty (an unauthenticated/localhost relay). Credentials are only sent after STARTTLS succeeds — InfraSigns never authenticates over a cleartext connection.
  • from and every to entry must be valid RFC 5322 addresses (a display name like InfraSigns <[email protected]> is fine); the config is rejected at load otherwise.
  • Digests, incident alerts, trends reports and inbound-webhook alert summaries are rendered natively as a theme-aware HTML email (with a plain-text alternative), and their subject is the status-first summary line, e.g. ✅ production healthy — Infrastructure Digest or ⚠️ production: 1 warning — Infrastructure Digest. The count in it is the report's own verdict, so an inbound-webhook batch reads 🔴 alertmanager: 3 alerts firing — Incident Alert rather than a severity breakdown. The healthy phrasing is reserved for a report that actually asserts health: a resolved alert group reads ✅ alertmanager: Resolved — Incident Alert (the episode is over, so the glyph is the healthy one even though the phrasing is not).
  • At most 50 signal rows are drawn, with the same …and N more signals note the other channels use. This is the only rendering the message size depends on directly, and a webhook batch is sized by the sender.
  • Other notifications — health-check and deploy-verification transitions, heartbeat, ad-hoc alerts — keep the [InfraSigns] <title> subject, e.g. [InfraSigns] 🔴 Health check FAILED — api-latency.

PagerDuty

PagerDuty is an incident-only channel: it pages your on-call rotation when an incident fires and clears the page when it resolves, using the Events API v2. Unlike the digest channels, it never receives daily digests, trends reports or deploy verdicts — a pager is for incidents.

Two things are incidents: an inbound alert group from a webhook, and a health-check transition — a check that fails pages, and its recovery clears the page (health checks in alert mode). A check that errors does not page: "could not verify" is not "the thing is broken", and an LLM or fetch outage would otherwise open one incident per check at once. If you added PagerDuty for alert webhooks and also run checks, expect pages from them too.

notify:
  pagerduty:
    routing_key: '${PAGERDUTY_ROUTING_KEY}'  # Events API v2 integration key
    severity: critical                        # optional; default for unspecified severity
  1. In PagerDuty, open the service that should receive alerts → Integrations → add an Events API v2 integration. Copy its Integration Key (a 32-char routing key) into routing_key.
  2. The channel is enabled when routing_key is set; empty disables it.
  3. A firing incident sends a trigger event; its resolution sends a resolve event. The two are correlated by a dedup key — the episode: an alert group (Alertmanager's own grouping, a CloudWatch alarm) or a check (check|<name>). So a resolution clears exactly the incident its firing opened, re-firings within one episode reuse the same PagerDuty alert, and two unrelated alert groups page as two incidents rather than merging into one. The event's payload.summary — the incident title in the PagerDuty UI and the text of the push — is the first line of the message, because that field is single-line: the summarizer's opening sentence for an alert group, the pass → fail transition line for a check. payload.source, the "affected system" column, is the report's identity — the payload format for an alert group, the check's name for a transition — not the correlation key. When the message carries a structured report (the webhook receiver and health checks; deploy reports are structured too, but this channel never sees them), the facts behind that prose ride along in payload.custom_details: the verdict, the source, the period, the model's full analysis (which no longer fits in the one-line summary), and up to 20 alert rows with their severity badges, plus a signals_omitted count when the batch was larger. This is the one channel that cannot render a report, so it gets the structure as JSON instead.
  4. severity sets the PagerDuty severity for messages whose own severity is info or unspecified (one of info, warning, error, critical; defaults to info). A message's own critical/warning severity always maps through directly, so an incident pages at its real urgency. Note this default escalates info-level incidents: with severity: critical, an all-info incident pages as critical — leave it at info (or unset) if you don't want that escalation.

Telegram, Slack, and email are optional and independent; configure any combination, and every one of them receives every notification (digests, incident alerts, trends reports, webhook-receiver summaries). PagerDuty is incident-only — it receives incident trigger/resolve events, not digests or trends.

In the hosted (cloud) service this applies to the feeds that run per-organization (digests, trends, health checks); incident alerts and deploy verification are operator-level and not delivered to a tenant's channels yet — see the Notifications page for how those are surfaced. A per-organization PagerDuty channel pages on exactly one of those feeds: health checks, which are per-org and are incidents. Inbound alert groups still page at the operator level, and digests, trends and deploy verdicts are not paged at all. It has no per-org edit form — a routing key is operator-seeded — so what a tenant sees is the channel and what reaches it, not a way to change it.