Notifications

Telegram

Create a bot

Open Telegram and start a chat with @BotFather
Send /newbot and follow the prompts
Copy the token — it looks like 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11

Get your chat ID

Personal chat:

Start a chat with your bot and send any message
Open https://api.telegram.org/bot<TOKEN>/getUpdates
Find "chat":{"id":123456789} — that's your chat ID

Group or channel:

Add the bot to the group/channel and send a message
Open the same URL — group IDs are negative (e.g. -100123456789)

Configure

notify:
  telegram:
    token: "123456:ABC-..."
    chat_id: "-100123456789"

Message format

Digests, incident alerts, trends reports and inbound-webhook alert summaries render as HTML (parse_mode=HTML) — a status-first verdict header, worst-first severity-accented signals, and a footer. On the webhook path each signal is one inbound alert, carrying the severity it was delivered at, its identifying labels and its summary annotation (falling back to description) — see Alert webhooks. A signal grounded in a single engine finding may also show an inline value column. It comes in two shapes: a movement, e.g. 2.4% → 10.1% ↑, and a threshold breach's reading-against-its-bar, described below. A movement's two numbers are that finding's own — a baseline anomaly shows its typical baseline → its recent peak (or trough), a trend shows its early → late window average — never a separate reading of the series taken at render time, which is what used to let the column disagree with the text beside it (#309). Two consequences worth knowing. The pair mixes aggregates of different kinds on an anomaly row (a baseline mean on the left, a recent extreme on the right), so read it as "normally about X, recently as far as Y" rather than as two readings a day apart; the claim's own prose names both. And with a real LLM the model chooses which finding a claim cites, so the column is only as well-matched as that choice — the numbers themselves are always an engine finding's, never the model's. Under llm.provider: none the match is exact, one claim per finding.

A static threshold breach shows a column of a different shape — 95.4% (limit 90%): one reading against the bar it met, with no arrow, because a breach observes a single sample and drawing it as 90 → 95.4 ↑ would invent a movement over time that nothing measured (#320). The bar shown is always the threshold that actually fired — crit: when a value breaches both — so on a digest it may be the critical bar beside a row the model chose to label a warning (the severity is the model's assessment; the numbers are the engine's).

A signal renders prose-only — no column — when it cannot be tied to a single finding: it spans several metrics, cites none, or is about a series as a whole rather than one observation. Incident alerts can show the same columns as a digest (#323): the engine's findings are grounded evidence there too, so a breach that coincides with a firing incident arrives with its reading and its bar rather than with prose alone. An incident row about an alert rather than a finding should carry no column — an alert payload holds no engine-computed numbers — and the prompt asks the model to cite nothing for such a claim; as everywhere else, which finding a claim cites is the model's choice, so that is an instruction rather than a guarantee. Two paths are prose-only by construction, having nothing citable at all: the inbound-webhook alert summary, and a critical cycle whose source produced no metric series and therefore no engine findings.

The unit suffix comes from a query's unit: or a cloud source's curated metric definition (#237); values with no known unit render bare.

Shortened text

An ellipsis in a delivered message can mean two very different things: the sender wrote one, or InfraSigns removed text to fit. A † after a signal's text (or at the end of the summary block) means the second — this product shortened this row. On the plain-text floor and in PagerDuty's custom_details the same mark is written [*], matching that surface's ASCII [OK]/[WARN] vocabulary.

The mark carries no words; a legend does, rendered once per message below the signals: "† marks a row this report had to shorten; the original text was longer." When only the summary was shortened, a summary-worded legend renders instead. The mark is drawn by the renderer as chrome, not inserted into the text — so a sender who writes a literal † cannot forge one on the two channels where that guarantee is structural (Telegram and HTML email escape sender text), and on the others they still cannot forge the legend's own block or line.

One case where the legend does not survive: the plain-text floor — what Telegram falls back to when it refuses to parse the rich message, and the text/plain half of an email — places the legend above the footer rather than last, so a report wide enough to exceed the wire limit loses the footer and then the legend while the row marks above them remain. A [*] with nothing explaining it means what this section says it means.

What it covers: the caps this product applies to an alert name, an alert's label set, an annotation, a reported-severity label, an episode name, a check's source and query, a deploy's version pair, a claim's prose, a verdict rationale — and the extra cut a channel makes to fit its own wire limits (Slack sections and fields, Slack's notification-preview transcript as a whole, PagerDuty detail rows).

Two things it deliberately does not cover, so a missing mark is not read as a promise:

The report's identity line — the source, check name or service version (environment) beside the report title. It is capped at 80 runes, which a realistic identity does not reach, and it renders in three places (the subject line, Slack's plain_text header, the email pill) where no chrome is possible. A shortened identity still ends in a bare ....
The legacy prose body — the plain Body a message carries when a channel cannot render the structured report, and the deploy body kept in the archive. It is free-form multi-line text from three different producers, so a mark could only say "something in here was shortened" without letting you find it. The structured rows carry the mark instead. When there are more signals than fit the 4096-character limit, the least severe are dropped and a …and N more signals note is appended (never a mid-tag cut). At most 100 signals are considered for a single message even when more would fit — a bound on the fit search rather than on the report, since a webhook batch is sized by the sender; the same note carries the rest. A message the API cannot parse falls back to the plain-text floor. Other notifications (heartbeat, ad-hoc alerts) are plain text, truncated with ... if over the limit.

Threaded incidents

A resolution arrives as a reply to the message that announced the incident, so a busy chat reads as episodes rather than as a flat run of firing and resolved messages with no way to tell which resolution belongs to which alert. It needs no configuration and works in every chat type, including a private chat with the bot.

Threading is per episode, and an episode is the sender's own grouping — an Alertmanager alert group (its groupKey, or its group labels when the sender omits one), a CloudWatch alarm, or a single health check. It is not per individual alert: a group is delivered as one message per phase, so the alerts inside one group share a thread. A partial resolution — one alert of the group recovering while others still fire — replies into the thread without ending it, so the episode stays open until the whole group is clear. A sender that supplies no grouping at all degrades to one thread per payload format.

Digests, trend reports and deploy verdicts have no lifecycle and are always posted standalone.

A resolution ends the episode: the next incident on that group opens a fresh top-level message instead of being buried as a reply under the resolved one. A firing whose resolution is never delivered is reset after 24h idle.

The roots are held in memory, so a restart re-roots open episodes — and in the hosted service, so does saving the organization's configuration, a plan change, or an hour of inactivity. If the message being replied to was deleted, the resolution is still delivered; it simply arrives standalone. Durable thread tracking, and forum topics (a richer grouping that needs a supergroup with Topics enabled), are future work.

If the chat is upgraded to a supergroup

Telegram gives a group a new chat id when it is upgraded to a supergroup, and the old one stops working. InfraSigns does not silently follow the move — the id lives in your configuration, so rewriting it in memory would be lost on the next restart (and, in the hosted service, on the next configuration save) while quietly disagreeing with what you have on disk. Instead the failure names the fix: the delivery is recorded as failed with

telegram sendMessage: chat was upgraded to a supergroup — this message was not delivered; its new chat id is -1001234567890 (Telegram: "Bad Request: group chat was upgraded to a supergroup chat"); set notify.telegram.chat_id to -1001234567890

so the new id is in the error itself, on the Notifications page and in the log, rather than an opaque bad request. Copy it into notify.telegram.chat_id (and into bot.telegram.allowed_chat_ids if the Q&A bot uses that chat — see Q&A bot). The message that hit the migration is lost: nothing re-sends it, so if it was an incident firing, the next repeat delivers once the id is fixed.

Rate limits

Telegram throttles with 429 Too Many Requests and says how long to wait. A delivery channel does not wait it out — several channels share one delivery deadline, so sleeping here would spend the budget of whichever channel is sent next — so a throttled message is recorded as failed with the interval in the error, and the next notification for that alert delivers normally. The Q&A bot's poll loop is the one place that does honour the interval; see Q&A bot.

Readiness is deliberately more forgiving: a throttle on the health probe reports the channel healthy, because the token resolved and the endpoint answered. That means a 429, and also any response whose Bot API body says ok:false with a retry_after — an on-path proxy is free to rewrite the status, and a response that says "not now, come back in N seconds" is still an answer. Every other failure — a revoked token, an unreachable host, any other error status, including a 500 that says nothing about retrying — still fails it.

Tip

The same bot can also answer questions — see the Q&A bot, which reuses notify.telegram.token (Telegram) or notify.slack.token (Slack) by default.

Slack

Slack has two transports. Pick ONE with mode:

`mode`	Delivery	Threading	Needs
`webhook`	Incoming webhook	no	a webhook URL
`api`	Web API (`chat.postMessage`)	yes	a bot token (`chat:write`) + channel ID

mode is required whenever any Slack field is set; the fields for the unselected transport must be empty. Omit the whole slack block to disable it.

`webhook` — incoming webhook (simple)

Create an incoming webhook for your workspace: https://api.slack.com/messaging/webhooks (choose the channel that should receive digests and incident alerts).
Configure:

notify:
  slack:
    mode: webhook
    webhook_url: '${SLACK_WEBHOOK_URL}'

webhook_url must be an absolute http/https URL with a host that is not a wildcard bind address. A value with no scheme (hooks.slack.com/services/… — what you get pasting from a UI that shows the scheme in a separate chunk), one with no host (https://:443/services/…) and one naming a wildcard (https://0.0.0.0:443/services/…, https://[::]:443/…) are all refused. A wildcard is what a listener binds to, so the last two would reach the machine this process runs on rather than Slack — write the target host, or localhost if that is what you meant. No Slack channel is built from any of them, so there is nothing to probe and no notify entry in /readyz — self-hosted serve says so once at startup instead, and holds infrasigns_notify_channel_deliverable{channel="slack"} at 0 (see the metrics catalog). config validate marks the channel (incomplete — nothing is delivered) but does not name the field: on the file path this key is checked for presence and never parsed. In the hosted service the org-config save now refuses such a value, naming notify.slack.webhook_url; a section already stored keeps loading and keeps its card, so only the next save is blocked. The error never echoes the URL — an incoming-webhook URL is itself a credential, since anyone holding it can post to the channel.

`api` — Web API (threaded incidents)

The Web API transport posts an episode's follow-ups — later firing batches and the resolution — as thread replies to the first firing message, keeping one incident's activity in one thread instead of scattered across the channel. An episode is the sender's own grouping (an Alertmanager alert group, a CloudWatch alarm) or a single health check, not an individual alert — see Threaded incidents for what that means in practice. It needs a bot token, not a webhook:

Create a Slack app, add the chat:write bot scope, install it to your workspace, and copy the bot token (xoxb-…).
Invite the bot to the target channel and copy the channel ID (e.g. C0123ABCD).
Configure:

notify:
  slack:
    mode: api
    token: '${SLACK_TOKEN}'
    channel: 'C0123ABCD'

Threads are keyed per episode and held in memory, so a restart re-roots open episodes — and in the hosted service, so does saving the organization's configuration, a plan change, or an hour of inactivity. A resolution ends the thread, so the next incident on that episode opens a fresh top-level message; a firing whose resolution is never delivered is reset after 24h idle. Durable thread tracking is future work. Telegram threads the same way, through replies.

Email (SMTP)

Email is a fallback channel for teams that do not use Telegram or Slack. It sends an HTML message with a plain-text fallback over SMTP with STARTTLS.

notify:
  email:
    smtp_host: smtp.example.com
    smtp_port: 587                     # optional; defaults to 587 (STARTTLS submission)
    smtp_user: '${SMTP_USER}'          # optional; omit both user and password for an unauthenticated relay
    smtp_password: '${SMTP_PASSWORD}'
    from: "InfraSigns <[email protected]>"
    to:                                # at least 1 recipient
      - [email protected]
      - [email protected]

The channel is enabled when smtp_host is set; an empty smtp_host disables it.
smtp_user and smtp_password must be set together, or both left empty (an unauthenticated/localhost relay). Credentials are only sent after STARTTLS succeeds — InfraSigns never authenticates over a cleartext connection.
from and every to entry must be valid RFC 5322 addresses (a display name like InfraSigns <[email protected]> is fine); the config is rejected at load otherwise.
Keep to to 100 recipients or fewer. The bound is enforced when a configuration is SAVED in the hosted service — a longer list is refused there, naming the count and the maximum. A config file is deliberately not bounded: it loads whatever it lists, and the relay is what answers for it. The Helm chart is the exception — its values schema caps the list at 100, so a chart-installed deployment is refused at helm install/helm upgrade instead of at the first alert. That matters because each notification is delivered as a single SMTP transaction, and RFC 5321 §4.5.3.1.10 requires a receiver to accept at least 100 recipients in one, so 100 is the largest number every compliant relay must take — past that you are at your provider's limit rather than the standard's (Amazon SES stops at 50 destinations, Gmail's relay at 100), and a relay that refuses one recipient costs you the whole notification — the send stops at that point and nobody receives it, not even the addresses already accepted — while the config still looks fine. Point to at a mailing-list address if you need to reach more people; one entry there costs you nothing here.
Digests, incident alerts, trends reports and inbound-webhook alert summaries are rendered natively as a theme-aware HTML email (with a plain-text alternative), and their subject is the status-first summary line, e.g. ✅ production healthy — Infrastructure Digest or ⚠️ production: 1 warning — Infrastructure Digest. The count in it is the report's own verdict, so an inbound-webhook batch reads 🔴 alertmanager: 3 alerts firing — Incident Alert rather than a severity breakdown. The healthy phrasing is reserved for a report that actually asserts health: a resolved alert group reads ✅ alertmanager: Resolved — Incident Alert (the episode is over, so the glyph is the healthy one even though the phrasing is not).
At most 50 signal rows are drawn, with the same …and N more signals note the other channels use. This is the only rendering the message size depends on directly, and a webhook batch is sized by the sender.
Other notifications — health-check and deploy-verification transitions, heartbeat, ad-hoc alerts — keep the [InfraSigns] <title> subject, e.g. [InfraSigns] 🔴 Health check FAILED — api-latency.

PagerDuty

PagerDuty is an incident-only channel: it pages your on-call rotation when an incident fires and clears the page when it resolves, using the Events API v2. Unlike the digest channels, it never receives daily digests, trends reports or deploy verdicts — a pager is for incidents.

Two things are incidents: an inbound alert group from a webhook, and a health-check transition — a check that fails pages, and its recovery clears the page (health checks in alert mode). A check that errors does not page: "could not verify" is not "the thing is broken", and an LLM or fetch outage would otherwise open one incident per check at once. If you added PagerDuty for alert webhooks and also run checks, expect pages from them too — feeds: [incidents] is how you keep the pager for alert groups alone, and those two are the only values this channel takes (see Feed routing).

notify:
  pagerduty:
    routing_key: '${PAGERDUTY_ROUTING_KEY}'  # Events API v2 integration key
    severity: critical                        # optional; default for unspecified severity

In PagerDuty, open the service that should receive alerts → Integrations → add an Events API v2 integration. Copy its Integration Key (a 32-char routing key) into routing_key.
The channel is enabled when routing_key is set; empty disables it.
A firing incident sends a trigger event; its resolution sends a resolve event. The two are correlated by a dedup key — the episode: an alert group (Alertmanager's own grouping, a CloudWatch alarm) or a check (check|<name>). So a resolution clears exactly the incident its firing opened, re-firings within one episode reuse the same PagerDuty alert, and two unrelated alert groups page as two incidents rather than merging into one. The event's payload.summary — the incident title in the PagerDuty UI and the text of the push — is the first line of the message, because that field is single-line: the summarizer's opening sentence for an alert group, the pass → fail transition line for a check. payload.source, the "affected system" column, is the report's identity — the payload format for an alert group, the check's name for a transition — not the correlation key. When the message carries a structured report (the webhook receiver and health checks; deploy reports are structured too, but this channel never sees them), the facts behind that prose ride along in payload.custom_details: the verdict, the source, the period, the model's full analysis (which no longer fits in the one-line summary), and up to 20 alert rows with their severity badges, plus a signals_omitted count when the batch was larger. This is the one channel that cannot render a report, so it gets the structure as JSON instead.
severity sets the PagerDuty severity for messages whose own severity is info or unspecified (one of info, warning, error, critical; defaults to info). A message's own critical/warning severity always maps through directly, so an incident pages at its real urgency. Note this default escalates info-level incidents: with severity: critical, an all-info incident pages as critical — leave it at info (or unset) if you don't want that escalation.

Feed routing

Telegram, Slack, email and PagerDuty are optional and independent; configure any combination. By default every configured channel receives every notification this deployment produces — digests, incident alerts, trends reports, webhook-receiver summaries, deploy verdicts and health-check transitions. That is what you get by writing no extra key, and it is what every release before feed routing did.

feeds: narrows that, per channel. It answers "who wakes me up?" by naming the producers whose messages reach a channel:

feed	what it carries
`digest`	the scheduled per-source digest — the routine one and the critical one alike
`trends`	the scheduled cross-source trends report
`incidents`	inbound alert groups from the webhook receiver, firing and resolved
`deploys`	post-deploy verification verdicts
`checks`	health-check transitions, and the retirement notice for an orphaned check

notify:
  telegram:
    token: '${TELEGRAM_TOKEN}'
    chat_id: '${TELEGRAM_CHAT_ID}'
    feeds: [incidents, checks]      # wake me here
  email:
    smtp_host: smtp.example.com
    from: "InfraSigns <[email protected]>"
    to: [[email protected]]
    feeds: [digest, trends]         # and mail me the reading material

Routing is a delivery choice, not an on/off switch for a producer: a digest a channel is not routed to still runs on its schedule, is still archived, and still reaches whichever other channels asked for it. Which producers this deployment runs at all is a separate question, answered by the feeds list on the Notifications page — and the two are read together there: a card's Feeds row names the feeds it is routed to that this deployment actually produces, so routing a channel at deploys on a deployment with deploy verification off leaves it with nothing to name, and the card mutes with that reason instead of promising a delivery.

One thing worth reading twice: a critical digest is still the digest feed. incidents means exactly the inbound alert webhook, so a channel routed to incidents alone stays quiet on the day a digest goes red. A feed names the producer, not the mood.

The three states of `feeds:`

The key is a list, and all three of its states are different settings:

what you write	what the channel receives
nothing at all (the key is absent)	every feed — the default
`feeds: [digest, checks]`	exactly those, in any order
`feeds: []`	nothing: a channel that stays configured and never fires

The last row is a real, supported setting rather than a mistake — it is how you park a channel (credentials kept, delivery off) without deleting it — so nothing normalizes an empty list back into the default. Because the two are one character apart, every place that shows your channels says which one you wrote:

config validate marks it, so the line cannot read as "these will be notified":
```
  notify:     telegram (routed to no feed), slack
```
the Notifications page shows the card muted, with a note naming the reason and the remedy, instead of a Feeds row promising a delivery that will not happen.
serve says it at boot, which is the one of the three you get without asking for it. Some channels parked beside live ones is an INFO; every configured channel parked is a WARN, because at that point the deployment delivers nothing at all:
```
level=WARN msg="every configured notification channel is routed to no feed — nothing will be delivered anywhere" channels=telegram,slack
```

A typo is refused at load rather than left inert, and so is a repeat:

notify.slack.feeds[0] "digests" is not a known feed (want one of: digest, trends, incidents, deploys, checks)
notify.slack.feeds lists "digest" more than once; each feed may appear at most once

Like every other per-channel rule, this one is checked as part of the channel: a feeds: list under a channel that is switched off entirely — no telegram.token, no slack.mode, no email.smtp_host, no pagerduty.routing_key — is not validated. That gate is each channel's own presence test, and it can be looser than what it takes to actually deliver: a Telegram section with a token but no chat_id delivers nothing, yet its feeds: is still checked, so a typo there surfaces now rather than on the save that completes the pair. Nothing slips through, because enabling a channel is itself a load or a save and every one of those runs the check.

Routing is applied where a channel is built, so it is honoured identically in self-hosted mode and in the hosted service. Its one deliberate gap: a message carrying no feed at all is delivered everywhere, routing or not. Nothing produces one today — every message InfraSigns sends stamps its feed — and the direction is chosen so that a producer added in a future release cannot go silently undelivered before its routing lands.

PagerDuty accepts two feeds, not five

notify:
  pagerduty:
    routing_key: '${PAGERDUTY_ROUTING_KEY}'
    feeds: [incidents]        # pages on alert groups; health checks stay on chat

PagerDuty's vocabulary is incidents and checks only, and that is a capability, not a preference. The channel drops any message that carries no incident phase before it ever builds an event, and only two producers set one — the webhook receiver and the check runner. A digest, trends or deploy scope on this channel could therefore never match anything, so it is refused rather than silently accepted and ignored:

notify.pagerduty.feeds[1] "digest" cannot be delivered by PagerDuty: a pager carries incidents only, so a digest, trends or deploy message is suppressed before it reaches the Events API and this scope could never match — want one of: incidents, checks

The Helm chart refuses the same three at helm install/helm upgrade instead of at the first alert (its values schema carries the narrower enum for this channel). Omitting the key still means "every feed", which for PagerDuty is these two.

In the Helm chart

Every channel object under config.notify takes the same optional feeds list — config.notify.telegram.feeds, .slack.feeds, .email.feeds, and .pagerduty.feeds with its two-value enum. All four are left unset in values.yaml on purpose, because unset is the unchanged behaviour. See Kubernetes.

In the hosted service

Routing works the same way per organization, and an organization owner can edit it on the Notifications page. The Telegram, Slack and email edit forms each carry a Feeds this channel receives checkbox group, saved with the rest of that channel's fields; every box is ticked for a channel that has no feeds: key. Unticking all of them is the feeds: [] state above — the card then renders muted, with the reason and a pointer back to the control.

The controls offer only the feeds a tenant's own runtime honours, which is narrower than the vocabulary: digests, trends and health checks run per organization, while incident alerts and deploy verification are operator-level — the alert webhook receiver and the deploy verifier run process-wide — so they get no box. A box whose use could not change delivery would be worse than its absence. A per-organization PagerDuty channel therefore has exactly one feed to offer today: health-check transitions, the one feed that both pages a pager and runs per organization.

An operator can still seed one of the two operator-level feeds onto a tenant's channel by writing the stored configuration directly. That value is kept, not dropped: a save from these forms preserves any stored feed the control does not offer, and the form discloses it — "This channel is also routed to …, which this form cannot change. Saving keeps that." — so a tenant cannot silently narrow a routing they were never shown. It still delivers nothing to them until that feed becomes per-organization, and the card says so rather than naming it in a Feeds row.

PagerDuty is also the one channel a tenant cannot otherwise edit, and that has not changed: the routing key is seeded by the operator and is never shown, changed or removed here. What an organization owner now gets is a routing-only card, and nothing else. It appears only once a routing key has been seeded for that organization, because routing a channel that does not exist would store a choice nobody honours.