Natural-language health checks

Write a health assertion in plain language, put it on a cron schedule, and InfraSigns asks the LLM to verify it against the source's metrics:

checks:
  - name: disk-headroom
    schedule: "*/30 * * * *"
    source: prod            # which configured source's queries to consult
    query: "Is disk usage below 85% on all instances and not growing abnormally?"
    mode: monitor           # monitor (default) = record only; alert = notify
    model: ""               # optional per-check llm.model override
    severity: warning       # info|warning|critical, default warning — tier a FAILING check pages at
    for_runs: 1             # consecutive non-pass runs before the episode opens, default 1
    repeat_interval: "30m"  # re-assert an open failing episode this often; omit = never

Only the first four fields are required — the shortest working check is name/schedule/source/query, and everything below it is optional. The last three are the paging controls, and repeat_interval is the only one shown at something other than its default: see Tuning the paging.

Checks require a real LLM provider: a natural-language assertion needs reasoning, so llm.provider: none with checks: configured is rejected at config validate (a mechanical template verdict would mask real failures).

Each run fetches the source's configured queries over the last hour, summarizes them (latest/min/max per series), and asks for a verdict. Results land on the dashboard's Health checks card; in alert mode, transitions also notify your channels.

On the channels that render structured reports (Telegram, Slack, email, and the plain-text floor) a transition is the same layout a digest gets — PagerDuty gets it as a page instead, see below: the check's name and when it ran, a status-first verdict — Passed / Failed / Could not verify — the model's rationale (collapsed behind a tap on Telegram, shown in full on the other three), then the facts as rows: the state change (Change: pass → fail, or First run: fail when there is no previous result), the source that was read, and the assertion that was checked. If the previous result could not be read at all — the state write failed — the change row is omitted rather than claiming a first run. The time is when the check ran, which is a model round trip before the notification you are reading. A failed or unverifiable check draws a neutral accent on those rows rather than the healthy ✅, which would say the opposite of the verdict above them; a check that passed draws the ✅ without claiming your fleet is healthy — one check was verified, not a fleet.

A query that returns many series (one per instance, pod or node) is not sent in full: the prompt lists at most the first 30 series of each query name individually — fewer when the whole list is long, see below — and folds the rest into one aggregate line whose min and max span every folded series — so an extreme hiding in the remainder still reaches the model, even though the individual series producing it is not named. (The series listed individually carry their own numbers, so the group's overall extreme is the union of the two.) Whenever anything is folded, every query name the prompt lists states its total series count as well — a name dropped by the 40-name ceiling below states nothing, because it is not in the prompt at all. At most 40 distinct query names are listed, in configured order — so on a source that is already past 40, adding a query at the top of queries: pushes the last one out. See LLM providers for the exact shape.

Results are tri-state: `pass` / `fail` / `error`

error means the check could not run — the metrics fetch failed, no data came back, the LLM call failed, or its answer was unparseable. error notifies in alert mode: a monitoring system that cannot run its checks must say so, not go silent. It is deliberately distinct from fail (the LLM verified a problem).

The prompt applies a fail-closed rule: if the data is insufficient to verify the healthy condition, the verdict is fail with the rationale saying what was missing. Absent data entirely is error — the LLM is not even asked.

The series fold above is deliberately not "insufficient data" for an assertion that turns on an extreme — "is anything above X", "is everything up" — because the aggregate line's minimum and maximum span every folded series. A wide check does not start failing just because it is wide, and the prompt says so explicitly. What a fold does destroy is which series holds a value and how many series hold it, so an assertion about one specific labelled series, about a count ("are at least 18 replicas up?"), about a proportion, about an average or about a trend across the series is insufficient data when the series it needs were folded — the prompt routes that case back to the fail-closed rule, and the rationale says the group was folded. An assertion that mixes the two shapes is judged on the part that needs the folded series: the example at the top of this page, "Is disk usage below 85% on all instances and not growing abnormally?", is extremum-shaped in its first half and trend-shaped in its second, so on a folded query it fails closed on the growth half and the rationale says which half could not be checked. Writing check assertions in extremum form is therefore the reliable shape on very wide queries — and splitting a compound assertion into two checks is better still. A dropped query name is the clearest case of all — the metric genuinely is not in the prompt — so a check whose assertion depends on one fails closed and names it.

Alerting on transitions only

Notifications fire when the result changes — pass→fail, fail→pass (recovery), anything→error — never on repeats. A still-failing 30-minute check pages once, not 48 times a day, unless you opt back into a bounded reminder with repeat_interval. State is persisted in Postgres, so transitions survive restarts. On the very first run only a non-pass result notifies — and with for_runs above 1, not even that: nothing notifies until the streak is met.

What changes is the episode state, which by default is the same thing as the run's verdict and stops being the same thing the moment you set for_runs — see Tuning the paging. The dashboard's badge always shows the verdict of the last run either way; a non-passing row reads not notified beside it when no notification has carried that verdict yet.

A transition is an incident episode, keyed on the check itself: a failure opens it, a recovery closes it. That is what threads a check's failure and its recovery together on Telegram and on Slack's api transport — and, if you have configured PagerDuty, what triggers an incident on a failure and resolves it on recovery. A check in alert mode reaches the same pager your inbound alerts do; monitor mode never pages, which is why every new check should start there.

Three things are worth knowing before you promote a check to alert:

An error does not page. "Could not verify" is not "the thing is broken", and one LLM or fetch outage would otherwise open an incident for every alert-mode check at the same moment. Errors still notify your chat channels.
A check pages at warning severity unless you raise it. severity: sets the tier per check; see severity for how it interacts with notify.pagerduty.severity.
A rename or a delete strands an open failure. An episode is keyed on the check's name, so renaming or removing a check while it is failing leaves its incident open with nothing scheduled to resolve it — nothing runs for a check that is no longer configured. Resolve it by hand, or put the old name back until it recovers.

A send that fails is a different matter: a notification no channel took does not advance the stored episode state, so a lost send is re-sent on the check's next run, in both directions. Nothing is silently dropped because a channel was down for one run — the cost is that a send that succeeded but was not recorded (the process died in between) is delivered twice. PagerDuty folds the repeat into the same incident; on chat it is a duplicate line.

Modes

Start every new check in monitor: results are recorded and visible on the dashboard, but nothing pages you while you tune the wording. Promote to alert once the check has been quietly passing (or failing exactly when you expect) for a few days.

A monitor check never announces anything, so a failing one shows its fail badge with the not notified marker beside it for as long as it keeps failing. That is the marker telling the truth, not a fault: on a card that does not show the mode, it is the only thing on screen saying nobody is being told.

Promoting a check that is failing right now pages you on its next run — the episode was never announced, so opening it is a real transition and you get a normal failure notification, not a mid-incident update.

One upgrade note, for a check that was already in monitor mode when you moved to the release that added these fields. The upgrade seeds every check's announced state from its last result — right for alert checks, since it is what stops them all re-paging at once, and wrong for monitor ones, which never announced anything. A monitor check clears it on its first run after the upgrade; until that run a failing one shows no not notified marker. Nothing pages either way, because the mode does not send — but let a promoted check have that one monitor run before you switch it to alert. Promote it inside that window and the runtime sees a failure it believes was already announced: no transition, so no page, until the check recovers and you get a recovery for an incident you never received.

Tuning the paging

Three optional fields shape when and how loudly an alert-mode check reaches you. All three are notification policy and nothing else. The dashboard card still shows the verdict of the last run, infrasigns_checks_evaluated_total still counts every run, and the source timeline still journals every change of the raw verdict — so the flapping that made you reach for for_runs stays visible after for_runs has stopped it waking you.

`severity`

info, warning or critical; warning when unset. The spelling is case-exact — Critical is rejected at config validate rather than folded.

It sets the tier a failing check pages at, and only that. A recovery is always info — it is good news — and an error is always warning: "could not verify" is not "the thing is broken", and an error cannot page at all, so rendering an LLM outage in critical red would assert something the verdict does not say.

It interacts with notify.pagerduty.severity, which applies only to messages carrying no severity of their own:

a check at warning or critical — including one that just took the default — overrides the PagerDuty default and pages at its own tier;
a check at an explicit severity: info defers to it, which is how you route every check through one service-wide tier without repeating it per check.

It sets the tier the notification is, not the colour it looks. The verdict band at the top of a structured report is derived from the outcome — a failed check reads as a warning-tier verdict — so a severity: critical check pages at critical urgency under a verdict that is not painted critical red.

`for_runs`

How many consecutive non-pass runs must land before the episode opens. The default, 1, opens it on the first one — exactly how checks have always behaved. Raise it for a check that is right but twitchy: for_runs: 3 on a 5-minute check means a problem has to survive 15 minutes of it before anyone is woken.

It counts runs, not elapsed time, and the divergence from Alertmanager's for: is deliberate. Alertmanager evaluates on a uniform interval, so a duration there is a run count wearing different clothes; a check's schedule is a 5-field cron that can be as irregular as 0 9 * * 1-5, and a duration threshold could not be honoured without inventing a cadence the config does not have. Multiply by your own schedule to get the wall-clock delay.

An error is a non-pass run, so it counts toward the streak; a pass resets it to zero. With for_runs: 3:

runs	what happens
`fail`, `fail`, `fail`	the episode opens as `fail` on run 3, and pages
`fail`, `error`, `fail`	also opens as `fail` on run 3 — an unverifiable run is not a recovery, which is the same reason a `fail`→`error` never closes an open page
`error`, `error`, `error`	opens as `error` on run 3: your chat channels hear about the outage, PagerDuty does not, because an `error` never pages
`fail`, `pass`, `fail`	nothing opens — the `pass` reset the streak, and run 3 starts a fresh one

While a streak is still short of the threshold the dashboard shows the run's own badge with a not notified marker beside it: the run failed, and nobody has been told.

for_runs gates opening an episode and nothing else. A recovery is never damped: the first pass closes an open failure immediately, whatever the threshold is. Delaying good news would leave a resolved incident sitting on someone's pager for no reason.

There is deliberately no upper bound. A for_runs larger than the check will ever reach is a silent "never alert", and mode: monitor already says that out loud — so if that is what you want, say it that way.

`repeat_interval`

A Go duration (30m, 2h); unset means never. It re-asserts an episode that is already open and failing, so a failure you have not fixed keeps refreshing instead of going quiet after its one notification: on PagerDuty it re-triggers the same incident, and on chat it lands as an update in the same thread.

Two limits, both worth knowing before you rely on it:

The re-assert rides the check's own run — there is no second timer — so it happens on the first run at or after the interval has elapsed. On a regular schedule that means the effective period rounds up to a multiple of the schedule, which is rarely the interval you wrote: repeat_interval: 30m on a check scheduled 0 3 * * * re-asserts once a day, and on */20 * * * * it re-asserts every 40 minutes, because at +20m the interval has not elapsed yet. Only when the interval is an exact multiple of the schedule do you get the number you asked for. On an irregular schedule — including */25, whose runs wrap at the top of the hour — there is no single period at all; the rule still holds run by run. The knob can only make the reminder rarer than the check's own schedule, never more frequent — to hear about a failure more often, run the check more often.
Only a fail is re-asserted. Both the open episode and the run doing the re-asserting have to be fail, so a check whose latest run came back error is not re-asserted while it stays that way — an error cannot page, and repeating it would be chat noise with no pager value behind it. A check stuck on error is a fetch or LLM problem, and the error verdict on the dashboard is where that belongs.

Writing good checks

Phrase the assertion either way — "Is X healthy?" or "Has X exceeded Y?" — the verdict is always about whether there IS a problem, not a literal yes/no.
The LLM sees only the source's configured queries (latest/min/max over the last hour). If your assertion needs a metric, add the query to the source first.
One assertion per check. Transitions are per check; a compound check that is half-failing flaps.

Cost

Every check run makes one billable LLM call, counted against llm.max_calls_per_day. A 30-minute schedule is 48 calls/day — and the budget is shared with digests, trends, and webhook summaries: a misconfigured * * * * * check burns 1440 calls/day, exhausts the default cap of 200 by ~03:20 UTC, and every other feature falls back to template output for the rest of the day. Size schedules accordingly and use model to point frequent checks at a cheaper model.

When the budget is exhausted, a check does not fall back to a template — a template cannot reason, and letting it vote would flip a genuinely failing check to a spurious pass/RECOVERED. Instead the run is an error ("llm daily budget exhausted"), which notifies once on the transition and shows on the dashboard until the budget resets at midnight UTC.

Metrics

infrasigns_checks_evaluated_total{check,result} counts runs by outcome; infrasigns_check_duration_seconds times the full fetch→verdict→notify run.