Configuration

InfraSigns is configured via a single YAML file. Pass the path with --config (default: config/config.yaml).

Full example

locale: en                      # LLM report language (ISO code): en | ru. Empty defaults to en.

severity_aliases:               # optional: map your own severity labels onto info|warning|critical
  p1: critical                  # without it, any other label reads as a warning

server:
  port: 8080                    # HTTP server: health/metrics/API/webhook (+ /app UI)

database:
  # The shipped config injects this from the environment: dsn: '${DATABASE_DSN}'.
  # A literal value works too; the bundled compose Postgres uses
  # postgres://infrasigns:infrasigns@postgres:5432/infrasigns?sslmode=disable
  dsn: '${DATABASE_DSN}'

sources:
  - name: production
    url: http://prometheus:9090
  - name: staging
    url: http://staging-prometheus:9090

llm:
  provider: openai              # openai | anthropic | none (deterministic engine, no key)
  api_key: sk-...
  model: gpt-4o-mini

notify:
  telegram:
    token: "123456:ABC-..."
    chat_id: "-100123456789"
  slack:                        # optional; omit the block to disable. Pick ONE transport:
    mode: webhook               # "webhook" (incoming webhook) or "api" (Web API, threads per episode)
    webhook_url: "https://hooks.slack.com/services/T00/B00/xxxx"  # required when mode is "webhook"
    # mode: api
    # token: "xoxb-…"           # bot token (chat:write scope); required when mode is "api"
    # channel: "C0123ABCD"      # channel ID; required when mode is "api"
  email:                        # optional; empty smtp_host disables the channel
    smtp_host: ""
    from: "InfraSigns <[email protected]>"
    to: [[email protected]]
  pagerduty:                    # optional; INCIDENTS ONLY (digests are not paged)
    routing_key: '${PAGERDUTY_ROUTING_KEY}'  # Events API v2 integration key; empty disables the channel
    severity: critical          # default for unspecified severity: info|warning|error|critical (default info)

reports:
  digest:
    schedule: "0 8 * * *"      # cron expression, UTC
  trends:                       # optional weekly-style trends report
    enabled: true
    schedule: "0 9 * * 1"
    window: "168h"
    step: "1h"
  retention_days: 90            # prune archived reports after N days; 0/absent = keep forever

webhook:                        # inbound alert receiver — see Webhook receiver page
  token: '${WEBHOOK_TOKEN}'     # min 16 chars; empty disables the endpoint
  dedupe_window: "5m"

incidents:                      # incident history (webhook episodes)
  retention_days: 90            # prune RESOLVED episodes after N days; 0/absent = keep forever

timeline:                       # source timeline journal (web UI per-source Timeline)
  retention_days: 90            # prune events after N days; 0/absent = keep forever

source_health:                  # frozen per-source rows: health, and severity readings (#358)
  retention_days: 90            # prune removed sources' rows after N days; 0/absent = keep forever

api:
  token: '${API_TOKEN}'         # guards POST /api/digest/trigger and /mcp; empty disables both

ui:
  enabled: false                # experimental web UI under /app
  charts:
    enabled: true               # source-detail live metric charts; false disables them + the /series endpoint

heartbeat:
  url: '${HEARTBEAT_URL}'       # dead-man's-switch ping; empty disables

checks:                         # natural-language health checks — see Health checks page
  - name: disk-headroom
    schedule: "*/30 * * * *"
    source: production
    query: "Is disk usage below 85%?"
    mode: alert                 # alert = notify on transitions; monitor (the default) records only
    severity: warning           # info|warning|critical — tier a FAILING check pages at
    for_runs: 1                 # consecutive non-pass runs before the episode opens
    repeat_interval: "30m"      # re-assert an open failing episode; omit = never
                                # the three above are notification policy — inert in monitor mode

deploys:                        # post-deploy verification — see its page
  enabled: true
  source: production
  delay: "5m"

bot:                            # Q&A bot — see its page
  telegram:
    allowed_chat_ids: [123456789]  # non-empty enables this transport (fail-closed allowlist)
    # token: ""                 # falls back to notify.telegram.token when empty
  slack:                        # Socket Mode; operators @-mention the bot
    allowed_channel_ids: [C0123ABCD]  # non-empty enables this transport
    app_token: "xapp-..."       # required (Socket Mode app-level token)
    # bot_token: ""             # falls back to notify.slack.token when empty

Language (locale)

locale: en   # ISO language code: en | ru. Empty (or omitted) defaults to en.

locale is a top-level setting (per-org in the hosted service, a deployment default when self-hosted). It sets the language of the LLM-generated report prose — the digest, incident, trends, and inbound-alert summaries: the model is instructed to write the summary and observation text in the configured language, while severity levels and other machine-readable fields stay canonical.

The LLM summary providers (openai, anthropic) honor it for the report prose, and the deterministic providers (llm.provider: none and the mock/budget-exhausted fallbacks) localize their fixed chrome — headers, verdict labels, recommendations, alert counts — through the built-in catalog, while leaving in English the interpolated values (metric names, trend words, engine finding details) and the change-tracking and recent-log-line sections (shared verbatim with the English LLM prompt).

The notification report chrome is also localized across every channel (Telegram, Slack, email, plain-text): the report title, the verdict line (counts, severity words and recommended-action clause, with CLDR-correct plurals), the section labels, the value-column direction word, and the email subject/<html lang>. Locale-neutral symbols stay fixed — the status glyphs, the value arrows (↑/↓/→), the ASCII badges ([OK]/[WARN]/[CRIT]) and the INFRASIGNS wordmark — and the claim prose and metric values remain canonical. One known residual: the report period line (the analysis-window label and timestamp, e.g. 24h to 17 Jul, 09:00 UTC) uses English date formatting and connector regardless of locale.

The web UI localizes its chrome progressively. Localized so far: the app shell (sidebar and topbar navigation, the account and organization menus, ARIA labels), the Dashboard, the sources, reports, incidents, incident-detail, and settings pages (headings, filters, empty/error states), the add-source wizard (shared by the first-run onboarding page), and the source-detail and notifications pages (status/endpoint/queries/timeline chrome, the channel cards and every channel edit form), and the interleaved-<code>/<a>/<strong> form hints and channel intros across those pages (the settings intro link, the wizard and edit-form hints, the notifications channel intros, and the dashboard run-digest CTA), and the Go view-model prose (the Settings and Notifications read-only view models — channel field labels, feed names and status lines, settings rows with CLDR plurals — the source-card state labels/detail, the browser-tab page titles, and the sign-in landing). The app-shell, onboarding, and sign-in pages set <html lang> to the configured locale.

These surfaces still render English pending the final web slices (#201): the Alpine runtime labels that travel via data-* to JavaScript (the two reports toggle labels, the wizard and healthcheck-editor runtime labels — Slice 3e-3); the health-check and deploy-verification LLM rationales (a separate verdict prompt, not localized yet even on the real providers); and a follow-up Go-handler-prose slice for the members/billing/organization standalone pages and the dashboard check-card / incident-row timeline prefixes ("since …"). A locale you set there validates and is accepted, but those surfaces stay English until then.

Severity aliases (severity_aliases)

InfraSigns understands three severity labels — info, warning and critical. Anything else your alert rules write reads as a warning, which is conservative but costly if your fleet is on a p1/sev1/page scheme: a collection cycle can never become an incident summary, and a pushed alert is delivered and paged at warning urgency instead of critical (see the alert webhook). Teach it your vocabulary instead of rewriting every alert rule:

severity_aliases:
  p1: critical
  sev1: critical
  page: critical
  p2: warning
  p3: info

• The label is rewritten once, as the alert arrives — on collection and on the inbound webhook alike — so everything downstream sees the canonical label: the incident decision, the notification tier and PagerDuty urgency, the ordering the LLM prompt cuts by, and the get_active_alerts MCP tool.
• Your original label is not destroyed. Only the alert's severity reading changes; the alert's own severity label keeps whatever your rule wrote. That is also why turning the table on, editing it, or removing it cannot re-open or re-key an incident episode: episode identity is built from the labels, which the rewrite never touches.
• Names are matched the way the label is read — case-insensitively and whitespace-trimmed, so one p1: entry covers P1 and a padded " p1 ". Two entries that differ only in case or padding are rejected at startup rather than letting one of them win at random.
• Targets are info, warning or critical, spelled exactly. A typo is a startup error, not a label that quietly keeps reading as a warning at 3am.
• The canonical labels themselves cannot be aliased. critical: warning is rejected: every notification tier and every claim in these docs rests on those three words meaning what they say. warn — which InfraSigns already reads as a warning — is aliasable, since it is not one of the three.
• An aliased label stops being unrecognized, which is the point: the WARN line and infrasigns_alerts_unrecognized_severity_total go quiet for it, because you answered the question they were asking. Labels you have not aliased still report exactly as before.
• On the hosted service the table is bounded: at most 200 mappings, and a name of at most 500 runes (#366). Both are far above any real vocabulary — the schemes this feature exists for name a handful of labels — and the name limit is the same one applied to a sender's severity label when it is recorded, with every surface that shows a label truncating far shorter still. The name is measured as you wrote it, padding included, not as the trimmed form the matching uses. Targets need no limit of their own: they must be one of the three words above, spelled exactly.

These are limits on saving, not on loading, and they do not apply to this file: a self-hosted config is the operator's own input and is not size-checked at all. On the hosted service they are checked wherever an organization's configuration is written, so no save can leave behind a table its own editor cannot then edit.

The editor sends the whole table on every save, so that request is size-capped too. Since #370 the cap is sized to carry any table the store will accept, so a valid table is never too large to submit; a genuinely runaway paste still meets a legible limit instead of a generic bad request.

If a hosted organization is upgrading with a stored table already over one of the two limits, nothing about its monitoring changes — collection, reports and every page carry on. What stops is saving: any configuration change is refused until the table is smaller. The editor still lists the whole table, and a Clear all button (alongside each row's own delete) shrinks it; a table over the count limit now posts and comes back naming that limit, so deleting rows until what remains will save converges either way. - A table that matches nothing validates cleanly. The names are checked for shape, not against your fleet — p-1: where your rules write p1 passes validation and does nothing. The startup log names the entry count and the mapping it loaded — the first 50 entries, with aliases_omitted counting any beyond that — but reading back what you loaded rules out a typo you can see, not one you can't. - What the table is measured against is on Settings (#358). An Unmapped severity labels your sources send card lists, per source, the labels that source's most recent collection cycle carried and no alias maps, each with how many alerts carried it. An inert entry is then visible by contrast: the label you mapped is absent from the list, and the label your fleet actually writes is still on it. Three states are kept apart on purpose, because collapsing them is the failure this card exists to prevent — a source reads as recognized only when its last cycle actually assessed something (at least one alert carrying a severity: label, all of them mapped), a source whose last cycle carried no alert with a severity label gets its own neutral line covering both a quiet cycle and a fleet that omits severity: entirely (neither concludes anything about your vocabulary, and the second is read as warning by the very fallback this card exposes), and a source with no reading at all to show is simply absent — it has never scraped successfully, you removed it from config, or retention reaped its last reading. A source that scrapes fine but has no alert firing is not absent; it lands in the neutral line. Dropping a removed source is deliberate: a stale reading must not pad the all-clear. The counts are that cycle's reading, not a running total, and the same WARN line and infrasigns_alerts_unrecognized_severity_total above remain the surface to alert on. Scraped sources only. An alert pushed to the alert webhook is not attributed to a source, so its labels cannot appear on this card — an empty card is not evidence that a push-only fleet's vocabulary is understood (#350 tracks keying that signal on the sender). - Editable in the hosted UI, per organization (#355). On the hosted service the table lives on Settings: an Alert severity section reports how many labels are mapped, and a Severity aliases card below it lists every mapping — read-only for every member, edited a row at a time by an owner (#363). The card appears once the organization has saved a configuration at all — in practice, once it has added a source, since the add-source wizard is the only step that creates one. Only an owner can change the table; every other member reads it. The severity for a new row starts unset rather than pre-picked: this table escalates, so choosing a tier is always deliberate. A save takes effect on the organization's next collection cycle, and it never rewrites history — rows already recorded keep the tier they were recorded with. Self-hosted, the file above stays the only way to change the table, and Settings reports the loaded entry count there too, for the same reason the rest of that page exists: to see what the process actually loaded without shelling into the container. - Editing through the UI leaves the rest of your configuration untouched. A save replaces the whole table and rewrites nothing else; clearing every row removes the table rather than storing an empty one. The reverse holds as well and always has: a severity_aliases: block an operator seeded by hand survives every other UI edit, because each form merges into the stored document rather than replacing it. - Every path that reads an alert's severity label honors the table. There are two: the collection cycle behind digests and incidents, and the get_active_alerts MCP tool. Trends read metric series and never see an alert, and a health check derives its own severity from whether the probe passed, so neither has a label to alias. On the hosted service only the collection cycle applies, because MCP is not offered there — so an organization's own table covers everything it can see. Pushed alerts are not an exception so much as outside the question: the alert webhook is process-global, so an inbound alert is the deployment's — recorded and delivered at the operator level and read with the operator's table. That is the same reason incidents.retention_days is not applied per organization; see the Settings page.

Escalation is the intended effect: mapping p1: critical means a p1 alert now opens an incident summary and pages at critical urgency. That is a deliberate inversion of the conservative default, and it is why the table is validated strictly — you are declaring what your fleet already meant.

Database

database:
  dsn: '${DATABASE_DSN}'         # postgres:// or postgresql:// URL (key=value form is rejected)

PostgreSQL 13+ is required. The DSN role must own the schema and hold CREATEROLE (migration 017 creates the infrasigns_app group role; managed Postgres master users — RDS, Cloud SQL — have it, a hand-provisioned owner may need ALTER ROLE ... CREATEROLE). The connection pool is capped at 10 connections (30 m max lifetime) — no tuning knobs; open an issue if a deployment actually saturates it.

Row-level security (defense-in-depth). Tenant tables carry PostgreSQL RLS policies in addition to the application's own scoping; every statement runs in a transaction pinned to an organization. With the default single-role DSN this is transparent. Two operational notes:

• Backups: pg_dump of a database with forced RLS must run as a superuser or a BYPASSRLS role — as a plain table-owner role it fails closed on the tenant tables. Do not work around a failing dump with --enable-row-security: it would silently dump zero tenant rows.
• Least-privilege role (groundwork): migrations create a NOLOGIN group role infrasigns_app holding exactly the DML grants the server needs; a deployment can mint a login member of it (CREATE ROLE ... LOGIN PASSWORD '...' IN ROLE infrasigns_app;). Today serve runs migrations at startup on its one DSN and migrations need the owner, so database.dsn must stay an owner-role DSN — the group role exists for the hosted multi-tenant deployment (where migrations and serving separate) and is what the integration tests run under. Self-hosted loses no protection: FORCE ROW LEVEL SECURITY subjects even the table owner to the policies (only superusers bypass, e.g. the bundled compose's bootstrap user).

Sources

sources:
  - name: production             # display name in digests and logs
    url: http://prometheus:9090  # Prometheus HTTP API base URL (bundled compose hostname)

• Source names may use only letters, digits, dots, underscores, and dashes (^[A-Za-z0-9._-]+$) — they double as metric label values, deep-link IDs, and the source's web detail-page URL. new and grid are reserved (they collide with UI routes). The same rule applies to check names.
• Every URL key must name a host, checked on the host NAME rather than on the authority. https://:8080 — a port and nothing else — parses, looks well-formed and used to load; but a hostless authority resolves to localhost, so such a value silently points InfraSigns at its own machine. config validate rejects it for a source url, a healthcheck endpoints[].url, heartbeat.url, bot.mcp_servers[].url, auth.base_url and auth.github.enterprise_url alike; the add-source wizard, infrasigns digest trigger --addr and the Slack webhook channel refuse it where they meet it. A wildcard address (https://0.0.0.0:8080, https://[::]:8080) reaches localhost the same way and is not rejected (#398).
• type selects the collector: prometheus (the default when omitted), cloudwatch (see AWS CloudWatch below), digitalocean (see DigitalOcean below), hetzner (see Hetzner Cloud below), healthcheck (see Healthcheck URLs below), or loki (see Loki (LogQL) below). A Prometheus source needs url (and takes an optional bearer token, see Authenticated Prometheus-compatible backends below); a CloudWatch source needs region and no url; a DigitalOcean or Hetzner source needs a token and one or more resources; a healthcheck source needs one or more endpoints; a loki source needs a url and one or more metric queries (with optional log_queries for log context, and an optional bearer token).
• Multiple sources are supported — each produces an independent digest, and Prometheus, CloudWatch, DigitalOcean, Hetzner, healthcheck and loki sources can be mixed freely
• The severity: label on your alerts decides how a collection cycle is reported: a firing critical alert makes it an incident summary, delivered at critical severity, instead of the regular digest. On a Prometheus source the label is whatever your alert rules wrote, so it is read case-insensitively and whitespace-trimmed — Critical and CRITICAL count. An unrecognized scheme (p1, sev1, page) and an alert carrying no severity: label at all are read as a warning: they ride the next digest and never turn a cycle into an incident summary. That is the default, not the only outcome: it is deliberate — escalating every label InfraSigns does not recognize would turn a typo into an incident — so a fleet on a non-standard scheme gets digests only until it declares the scheme with severity_aliases, which rewrites such a label as it arrives and lets it open an incident like any other critical. Everything below describes the labels no alias covers. On a collection cycle — scheduled, triggered manually through the digest API, or requested through the MCP tool — you do not have to read this page to find out: a cycle carrying an unrecognized label logs a WARN line containing the phrase unrecognized severity labels rank as warning — each path then spells out its own consequence, so grep the phrase, not a whole line — with the source and the labels themselves, and increments infrasigns_alerts_unrecognized_severity_total{path="scheduler",source} on every such cycle — the un-throttled surface to alert or graph on. The log line is throttled to at most once per source per 24h, so a label that first appears inside that window waits it out; each line names up to five labels plus the count of the ones it cut, so the next warn describes the remaining vocabulary rather than only what changed — with the residual that a label past the cap is counted but never named, so a source with more than five unrecognized spellings shows the five carried by the most alerts and a count (#358 — before it, the five that sorted first alphabetically). The throttle is in memory on the running scheduler, so it is not a promise about a source for all time: a process restart re-warns on the next cycle, and in cloud a rebuilt org runtime — a config save, a plan change, or an org idle long enough to age out of the runtime cache — does the same. What it bounds is the repetition that made the line unreadable: a handful of lines a day for a misconfigured source instead of one on every cycle. Two labels that rank as warning are deliberately not reported, because neither is something to fix: warn, which InfraSigns already understands as a warning, and no severity: label at all, which is just an alert rule that never set one. A healthcheck source's down alerts take the same path, but their severity: is configuration rather than a free-form label and is validated case-exactly at startup (see Healthcheck URLs).

  The inbound alert webhook raises the same signal, with a worse consequence and a coarser key (#345). Everything above is about collection cycles. A pushed alert's severity: goes through the very same reading and falls back to warning the same way, and the webhook receiver now logs a WARN line prefixed webhook: and carrying the same unrecognized severity labels rank as warning phrase, and increments infrasigns_alerts_unrecognized_severity_total{path="webhook",source} on every such request. Two things differ. What goes wrong is worse: a pushed alert is not merely capped at digests, it is delivered — every channel renders it as a warning and PagerDuty pages at warning urgency instead of critical, since only critical maps to a PagerDuty critical and this is the only path that carries a sender-supplied severity there (a health-check transition also pages, but at the tier its own severity: sets — your config, not a label a sender can distort). And source means something else there: the detected payload format, not the fleet that pushed — in practice always alertmanager, since a CloudWatch payload carries no severity: label for the receiver to fail to recognize. So the 24h log throttle is shared across senders: two AlertManager fleets pushing to one endpoint let whichever warns first silence the naming of the other's labels until the cooldown passes, and only a process restart clears it. The counter moves for both, which is why it, not the line, is the surface to alert on. (#350 tracks keying the signal on the sending fleet instead of the format.) The remedy is the same one the collection path uses: severity_aliases applies to pushed alerts too, and it is the only way to make this path page at critical urgency for a label InfraSigns does not know on its own.

• log_source (optional) — the name of another configured source whose logs ground an incident on this one. See Grounding an incident in another source's logs below.
• InfraSigns is read-only — a Prometheus source calls only GET /api/v1/alerts and GET /api/v1/query; a CloudWatch source calls only ListMetrics / GetMetricData; a DigitalOcean source calls only the read-only Monitoring and resource-listing endpoints; a Hetzner source calls only the read-only Metrics and resource-listing endpoints; a healthcheck source only issues GET/HEAD requests to the URLs you configure; a loki source calls only GET /loki/api/v1/query_range
• No changes to your Prometheus configuration are required

Authenticated Prometheus-compatible backends (token)

Many OpenTelemetry-native and hosted backends expose an authenticated Prometheus-compatible query API — Grafana Cloud, Dash0, Grafana Mimir, VictoriaMetrics Cloud, and Thanos behind an auth proxy. Point a Prometheus (or Loki) source at their query endpoint and add an optional bearer token:

sources:
  - name: grafana-cloud
    url: https://prometheus-prod-01.grafana.net/api/prom   # the query API base
    token: '${GRAFANA_TOKEN}'                               # Authorization: Bearer
    queries:
      - name: http_error_rate
        query: 'sum(rate(http_requests_total{status=~"5.."}[5m]))'
        signal: errors

• token is sent as Authorization: Bearer <token> on every query. Leave it unset for an unauthenticated Prometheus — the field is optional. When set it must be at least 16 characters, and the url must be https (a bearer is refused over cleartext http to a non-loopback host, so it can't be sniffed on the wire — a loopback host is exempt for local dev: localhost (also LOCALHOST and localhost., the same name), or an IP literal that parses as loopback, including an IPv6 zone such as [::1%25eth0]). Short address literals like 127.1, 0177.0.0.1 and 2130706433 are not loopback here — Go's parsers do not implement that syntax, so the value goes to DNS as a name, and a resolver search domain could expand 127.1 to 127.1.corp.example.com and carry the bearer token to a remote host in cleartext. The same goes for a Unicode look-alike of the name (localhoſt).
• The token is a secret: inject it via an environment variable (token: '${GRAFANA_TOKEN}') exactly like every other credential (see Kubernetes). Self-hosted YAML and the cloud add-source wizard both collect the token (#251); in the wizard, entering a token skips the live probe (it verifies on first collection, like the cloud sources) and the source-detail page lets an owner rotate it. A cloud organization stores the token in its config; self-hosted copy-paste emits a commented ${PROM_TOKEN} placeholder to uncomment and wire from the environment. Never put a bearer in the url query string — a ?token=… (or a bare trailing ?) is rejected at config load, because a query parameter is a credential carrier that can't be reliably redacted from error logs. Put it in token: the bearer rides an Authorization header, so it never appears in the source URL, in metric labels, in the web UI, or in sanitized error logs.
• The header is scoped to the configured host: if the endpoint issues an HTTP redirect to a different host, the token is not forwarded (it stays on your backend, never a redirect target).
• Digests, incidents, trends, and the source-detail page's live-charts widget all read the authenticated source normally — the charts dial sends the same bearer header (#252). (In cloud the live-charts endpoint is not yet served per organization, so those charts show no data there regardless of authentication.)
• This is the deliberate shape of OpenTelemetry support — InfraSigns is a view + analysis layer over your existing backend, not a TSDB. Apps instrumented with OTel reach InfraSigns through a backend that ingests OTLP (Prometheus ≥3.x native OTLP endpoint, Dash0, Grafana Cloud, …). See OpenTelemetry for end-to-end recipes and metric-name normalization notes.

Incident attribution (alert_labels)

Inbound webhook alerts can be attributed to a configured source, so its detail-page timeline shows them under the Incidents chip:

sources:
  - name: production
    url: http://prometheus:9090
    alert_labels:          # attribute alerts whose labels contain ALL pairs (AND)
      cluster: production

• With exactly one source and no alert_labels anywhere, every alert is attributed to it automatically — zero config for the common single-source install. Adding a second source (or any alert_labels) switches to explicit label matching.
• An alert matching no predicate stays unattributed (it still appears on the Incidents page, and its "Attributed to" filter has an explicit Unattributed option); an alert matching more than one source is left unattributed and logged, never guessed. Predicates that make ambiguity inevitable (one a subset of, or equal to, another) are rejected at config load.
• CloudWatch alarms carry only a region label, so alert_labels: {region: eu-west-1} is the only usable predicate for them; AlertManager and Grafana alerts pass their full label set through.
• Attribution happens at receive time — there is no bulk backfill for incidents recorded before it was configured. An episode still delivering events catches up on its next delivery that reaches the store (a repeat firing past webhook.dedupe_window refreshes the open episode — dedupe-suppressed repeats don't; a re-sent resolution fills an unattributed history row), but fully quiet history stays unattributed.

Query hints for the analysis engine

Each custom query optionally carries hints for the deterministic analysis engine (used in every mode; with llm.provider: none they drive the report directly):

sources:
  - name: production
    url: http://prometheus:9090
    queries:
      - name: cpu_usage_percent
        query: '(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100'
        signal: utilization   # latency | traffic | errors | saturation | utilization
        warn: 70              # optional static thresholds; higher-is-worse —
        crit: 90              # a finding fires at value >= threshold
        unit: "%"             # optional display suffix for signal value columns

• signal overrides the name-based golden-signal classification (unhinted metrics are classified by substring heuristics: "latency", "error", "queue", "cpu", "request", …).
• unit is an optional display suffix appended verbatim to every number of this query's value column, in a digest, a trends report or an incident alert alike (23.7 → 11.2 ↓ becomes 23.7% → 11.2% ↓; a threshold breach's 95.4 (limit 90) becomes 95.4% (limit 90%)). Use "%" for a percentage, " GB" (leading space) for a word unit; omit it to render bare numbers. It is cosmetic only — it never affects a threshold or the analysis (a metric value carries no unit on its own, and a name is not a reliable unit). DigitalOcean, CloudWatch, and the node fallback set carry a unit from their curated metric definitions where the API guarantees one (cpu/memory %, storage/network bytes, latencies s/ms); metrics with no reliably-known unit render bare (#237). Hetzner attaches none — its cpu is summed across vCPUs (a % there would misstate the range) and its other metrics are rates.
• warn / crit produce threshold-breach findings in digests and trends reports. Thresholds are higher-is-worse; for lower-is-worse metrics (free space, availability) invert the query expression instead. warn must be <= crit when both are set.
• Baseline anomaly detection (recent window vs the rest of the fetch window, z-score >= 3) and trend direction need no hints; anomalies need at least 16 data points in the analysis window (for trends: window / step >= 16 — below that only trend and threshold findings fire; the digest's derived window always yields enough points). Escalation to critical additionally requires the deviation to be large relative to the metric's own level (≈30% of the baseline mean), so a rock-stable gauge whose tiny variance turns a benign excursion into a many-σ outlier stays a warning instead of escalating to critical. This is range-free by design; for a bounded metric where danger is proximity to a ceiling (a disk filling to 95%), configure a crit threshold — that fires independently of the anomaly detector.

AWS CloudWatch

A cloudwatch source pulls metrics natively from AWS, for teams without a self-hosted Prometheus:

sources:
  - name: aws-prod
    type: cloudwatch
    region: eu-west-1
    # uses the standard AWS credential chain: env vars, ~/.aws, or an IAM role
    namespaces: [AWS/EC2, AWS/RDS]   # optional; defaults to all supported

• Credentials come from the standard AWS chain (env vars, shared ~/.aws/config/credentials, or an IAM role / IRSA on EKS) — never from this file. The minimal read-only IAM policy:

{
  "Effect": "Allow",
  "Action": ["cloudwatch:GetMetricData", "cloudwatch:ListMetrics"],
  "Resource": "*"
}

• role_arn (optional) makes the source assume a specific IAM role via sts:AssumeRole instead of using the ambient chain above:

sources:
  - name: aws-prod
    type: cloudwatch
    region: eu-west-1
    role_arn: arn:aws:iam::123456789012:role/infrasigns-readonly

The assumed role carries the read-only policy above; the process's own identity needs only sts:AssumeRole on that role. In self-hosted this is optional (leave it out to use the ambient chain). In the hosted multi-tenant service it is required — see the cross-account note below.

• namespaces scopes which curated metric sets are fetched. Supported: AWS/EC2, AWS/RDS, AWS/Lambda, AWS/ApplicationELB, AWS/ECS. Omitting it fetches all of them. For each namespace InfraSigns discovers the running resources via ListMetrics and fetches the standard golden-signal metrics (CPU, latency, errors, saturation) in one batched GetMetricData call. To bound cost, at most 500 series are fetched per cycle (excess is dropped with a log warning).
• metrics optionally replaces the curated set with an explicit list, e.g. a custom or application namespace:

sources:
  - name: aws-prod
    type: cloudwatch
    region: eu-west-1
    metrics:
      - name: queue_depth
        namespace: AWS/SQS
        metric_name: ApproximateNumberOfMessagesVisible
        stat: Average            # Average | Sum | Minimum | Maximum | SampleCount
        dimensions: { QueueName: jobs }   # optional; omit to fetch every instance
        signal: saturation       # same engine hints as Prometheus queries

• CloudWatch alarms are ingested on the push side, not here: point an SNS topic at an InfraSigns webhook subscription (X-Infrasigns-Source: cloudwatch). Metrics (this section) and alarms (the webhook) are independent — configure whichever you need.
• Cross-account (hosted service). In the multi-tenant hosted service a CloudWatch source must set role_arn: without it the source would resolve against the worker's own AWS identity, reading the operator's metrics rather than yours (a confused-deputy), so a role-less cloudwatch source is skipped for hosted organizations. To grant access, create a role in your AWS account that trusts the InfraSigns hosting account and pins your organization id as the STS external ID:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::<infrasigns-hosting-account>:root" },
  "Action": "sts:AssumeRole",
  "Condition": { "StringEquals": { "sts:ExternalId": "<your-organization-id>" } }
}

InfraSigns always sends your organization id as the external ID (it is server-supplied, never taken from your config), so a role scoped to your org id can be assumed only for your organization — no other tenant can name a role under it. Self-hosted deployments need no external ID (a single AWS account); a role_arn there is assumed directly. The add-source wizard collects the region, role_arn, and namespaces and shows your organization id as the external ID to pin here (with a ready-made trust-policy snippet).

Operator hardening (hosted-service deployments). role_arn is tenant-supplied and can name any account — including the hosting account. The per-tenant isolation above holds only if the worker's own IAM principal cannot assume a role that ignores the external ID. Scope the worker's sts:AssumeRole grant to external customer roles (not Resource: "*"), and ensure no role in the hosting account trusts the worker/account root without an sts:ExternalId condition — otherwise a tenant naming such a role would be assumed despite the external-ID mismatch, re-opening the confused-deputy one level up. This is a deployment-side obligation InfraSigns cannot enforce in code.

DigitalOcean

A digitalocean source pulls metrics natively from the DigitalOcean Monitoring API, for teams on Droplets, managed databases, and load balancers without a self-hosted Prometheus:

sources:
  - name: my-do-account
    type: digitalocean
    token: ${DO_TOKEN}          # personal access token, read scope
    resources:
      - type: droplet
        tags: [production]      # optional: only droplets carrying ALL listed tags
      - type: database          # managed MySQL clusters
        ids: [db-abc123]        # optional: these exact ids (skips listing)
      - type: load_balancer     # optional: omit ids/tags for all of the type

• token is a DigitalOcean personal access token with read scope. It is expanded from the environment like every secret (${DO_TOKEN}) and rides an Authorization header — it never appears in logs or error messages. No write scope is needed; InfraSigns only reads metrics and lists resources.
• resources selects what to monitor — at least one selector is required. Each names a type (droplet, database, or load_balancer) and optionally narrows it:
• ids: [...] — monitor exactly these resource ids (no listing performed).
• tags: [...] — list resources of the type and keep only those carrying every listed tag (AND). DigitalOcean load balancers and databases also support tags.
• neither — monitor all resources of the type in the account.
• Curated metrics per type (the analysis engine finds anomalies and trends on all of them; only the percentage metrics carry static thresholds):
• droplet — CPU, available memory, free filesystem, 1-minute load, public inbound/outbound bandwidth.
• database — CPU %, memory %, disk % (with 80/95 warn/crit thresholds). DigitalOcean exposes monitoring metrics only for MySQL clusters, so non-MySQL databases are skipped during discovery (logged once per cycle). Per-service metrics (connections, query latency) need a per-cluster service identifier and are not wired yet.
• load_balancer — requests/sec, current connections, average response time, and HTTP responses (for error-rate analysis).
• No alarms. DigitalOcean has no AlertManager, so InfraSigns anomaly detection is the only alert path for these sources — there is no push/webhook side.
• Cost bound. At most 200 (resource × metric) Monitoring calls are issued per cycle (excess is dropped with a log warning), fanned out with bounded concurrency. Each cycle additionally makes one reachability probe plus the resource-discovery listing calls (paged), so the DO API call count is ~200 metric calls plus discovery — a tags filter over a large account pages the fleet to find matches.
• Self-hosted only (for now). A DigitalOcean source uses the single configured token, with no per-tenant credential — run it in a deployment you control, scoped to one DigitalOcean account. (Unlike CloudWatch, it has no per-source cross-account mechanism for the hosted service yet.)

Hetzner Cloud

A hetzner source pulls metrics natively from the Hetzner Cloud Metrics API — the cost-conscious choice for EU/Go teams on Hetzner without a self-hosted Prometheus:

sources:
  - name: my-hetzner
    type: hetzner
    token: ${HETZNER_TOKEN}       # API token, read scope
    resources:
      - type: server
        labels: {env: production} # optional: only servers with ALL these labels
      - type: load_balancer
        ids: ["4711"]             # optional: these exact numeric ids (skips listing)

• token is a Hetzner Cloud API token with read permission. It is expanded from the environment like every secret (${HETZNER_TOKEN}) and rides an Authorization header — it never appears in logs or error messages. No write permission is needed; InfraSigns only reads metrics and lists resources.
• resources selects what to monitor — at least one selector is required. Each names a type (server or load_balancer) and optionally narrows it:
• ids: [...] — monitor exactly these numeric resource ids (no listing).
• labels: {k: v, ...} — list resources of the type and keep only those whose labels contain every listed key=value pair (AND), applied server-side as a Hetzner label selector.
• neither — monitor all resources of the type in the project.
• Curated metrics per type (the analysis engine finds anomalies and trends on all of them; none carries a static warn/crit threshold — Hetzner's server CPU is summed across vCPUs rather than a normalized 0-100%, and every other metric is a raw rate or count, so a fixed threshold would misfire):
• server — CPU (summed across vCPUs), disk read/write IOPS and bandwidth, network in/out packets-per-second and bandwidth. Multi-disk / multi-NIC servers produce one series per device (a device label disambiguates).
• load_balancer — open connections, connections/sec, requests/sec, and inbound/outbound bandwidth.
• 5-minute resolution. The Hetzner Metrics API returns 5-minute averages for anything but very short ranges, not raw samples — so anomaly detection over Hetzner series is coarser than over a fast-scrape Prometheus. This is a Hetzner API limitation, not a configuration knob.
• No alarms. Hetzner has no AlertManager, so InfraSigns anomaly detection is the only alert path for these sources — there is no push/webhook side.
• Cost bound. At most 200 resources are monitored per cycle (excess is dropped with a log warning). Hetzner returns every requested metric type for a resource in one Metrics call, so a cycle makes at most ~200 metric calls (fanned out with bounded concurrency) plus one reachability probe and the paged resource-discovery listings.
• Self-hosted only (for now). Like DigitalOcean, a Hetzner source uses the single configured token, with no per-tenant credential — run it in a deployment you control, scoped to one Hetzner Cloud project.

Healthcheck URLs

A healthcheck source periodically probes a list of HTTP endpoints and reports each one's availability and response latency — synthetic monitoring for a URL with no metrics backend, or for an operator with no Prometheus at all.

sources:
  - name: public-endpoints
    type: healthcheck
    endpoints:
      - url: https://api.example.com/healthz   # required; http or https
      - url: https://example.com/status
        method: HEAD           # GET (default) or HEAD
        expect_status: 204     # exact status; omit for "any 2xx"
        timeout: 5s            # default 10s, max 30s
        severity: warning      # down-alert severity: critical (default) | warning | info

• endpoints — one or more probes. Each needs a url; method, expect_status, timeout and severity are optional (defaults above) — but the UI editors require an explicit severity (#357): a form row has a select the user simply never touched, and the tier is what pages, so the add-source wizard and the source-detail edit form refuse an unset one while this file keeps its default. A stored source that omits it keeps working and opens its edit form showing critical, the tier in force. A probe is up when the endpoint answers with the expected status (any 2xx when expect_status is unset), down on any other status, a transport error, or a timeout. Redirects are followed and the final status is checked, so an expect_status in the 3xx range is rejected at config load (it can never be the final status).
• No secrets in the URL. The url is shown in labels, the web UI, and every digest/incident delivered to your channels and the LLM. Userinfo (user:pass@) and the entire query string are stripped from that displayed/labelled form, so a ?token=… would silently vanish there — put credentials in the endpoint's own auth, not the URL. Because two endpoints that differ only in userinfo or query then look identical, they are rejected as duplicates at config load: distinguish endpoints by scheme/host/path, not by a query string.
• Metrics. Each endpoint emits up (1/0) and latency_ms series, labelled by that stripped url — so the digest shows current availability and response time.
• Down = alert. A down endpoint raises an alert at the endpoint's severity; a critical one (the default) makes the cycle a paged incident, a warning or info one rides the next regular digest.
• No history / limited trends. A healthcheck has no metrics backend — each cycle probes live and records a single point, so trend analysis (which needs several points) is effectively flat. The value of a healthcheck source is the down alert and the current state in each digest, not a trend. Each endpoint is probed twice per digest cycle (once for the alert, once for the metric series) and once more per trends cycle, so size your endpoints' own rate limits accordingly.
• Readiness. A down endpoint never marks the pod NotReady: /readyz reflects the prober itself (always ready), and endpoint health is reported through the digest and the up metric instead — so one unreachable URL cannot stop the other sources' digests, the UI, or metrics.
• Cloud-safe. Unlike the token-based cloud collectors, a healthcheck source carries no ambient credential, so a hosted deployment routes its probes through the same egress (SSRF) gate as a tenant Prometheus URL — a probe that resolves to a private or metadata address is refused at dial time.

Loki (LogQL)

A loki source runs your LogQL metric queries against a Grafana Loki instance and feeds the resulting numeric series into the same digest / incident / trends engine as Prometheus — so logs-derived signals (error rate, per-level counts, anomaly rates) sit next to your metrics. It is the near-twin of a Prometheus source: same url + queries shape, only the query language and the endpoint differ.

sources:
  - name: app-logs
    type: loki
    url: http://loki.monitoring.svc:3100    # Loki HTTP API base URL
    tenant_id: team-a                       # optional; X-Scope-OrgID for multi-tenant Loki
    queries:                                # at least one; each is a LogQL METRIC query
      - name: error_rate
        query: sum(rate({app="api"} |= "error" [5m]))
        signal: errors                      # optional engine hint
        unit: /s                            # optional display suffix in the digest
      - name: warn_logs
        query: sum by (level) (count_over_time({app="api"} | logfmt | level="warn" [5m]))
    log_queries:                            # optional; raw log lines for LLM context
      - name: api_errors
        query: '{app="api"} |= "error"'     # a raw LogQL LOG selector (returns lines)
        limit: 30                           # optional per-query line cap (default 20, max 100)

• url — the Loki HTTP API base. Each query is issued as GET {url}/loki/api/v1/query_range.
• queries — LogQL metric queries — rate, count_over_time, sum, bytes_rate, quantile_over_time, etc. — whose result is a matrix of numeric series, exactly like a PromQL range query. name, signal, warn, crit and unit behave exactly as they do for a Prometheus query (see Query hints for the analysis engine). A raw LogQL log query (a bare stream selector, which returns log lines) is rejected here — wrap it in a metric aggregation, or put it in log_queries.
• log_queries (optional add-on; needs a metric query) — raw LogQL log selectors (e.g. {app="api"} |= "error") whose recent lines are captured as LLM context on a critical cycle (#246). Because a loki source emits no alerts, a "critical cycle" for it means a critical finding — e.g. an error-rate metric query breaching its crit: threshold. When that fires, the most recent lines (per-query limit, default 20, max 100; capped in total; max 10 log_queries) are attached to the digest / incident summary so the model can reason over the actual logs. On a normal cycle no log query runs. log_queries carry no signal/warn/crit/unit (a line is not a numeric series) and are not citable claim evidence — they are context only. They are an add-on, not a substitute: a loki source still needs at least one metric queries, because the metric queries are the numeric digest/trends signal — a source with none has nothing to summarize. That holds for a loki source configured purely as another source's log_source target too, whose log_queries fire on that source's critical cycle.

  Privacy: captured log lines are sent to your configured LLM provider (for provider: openai/anthropic, a third-party API) and may be quoted into the stored report. Application logs often contain secrets or PII (tokens, emails, request bodies) — scope your log_queries selectors to streams safe to share, especially on a hosted LLM. Control characters and embedded newlines are neutralized, but secrets are not redacted.

• tenant_id (optional) — the Grafana Loki tenant to query. A multi-tenant Loki (the common self-hosted setup) requires an X-Scope-OrgID header to select the tenant and rejects a tenant-agnostic query with no org id; set tenant_id and InfraSigns sends it on every query. InfraSigns does not treat it as a credential (it names a tenant), so it may be a literal or a ${VAR} — put real authentication in front of Loki (a reverse proxy, or the bearer token below) rather than relying on X-Scope-OrgID as access control. It is sent only on the query endpoint, never on Loki's tenant-agnostic /ready readiness check. The value must be free of control characters (it rides an HTTP header) and at most 150 bytes — a longer or CR/LF-bearing value is rejected at config load.
• Authentication. For an authenticated Loki endpoint (e.g. Grafana Cloud Logs) set an optional bearer token — sent as Authorization: Bearer <token>, exactly like a Prometheus source. It is a secret; inject it via ${VAR} and keep it out of the URL.
• No secrets in the URL. A query string (?token=…, or a bare trailing ?) is rejected at config load, exactly like a Prometheus source — put the bearer in token, or use Loki's own auth (e.g. a reverse proxy). Userinfo (user:pass@) is allowed and still sent on the wire to connect, but it is stripped from the displayed URL in the web UI and logs, so a credential there never renders.
• Readiness. Healthy / /readyz probe Loki's GET /ready — a loki source that cannot reach Loki reports unhealthy, but the probe is cheap and unmetered.
• Cloud-safe. Like a healthcheck and Prometheus URL, a tenant's loki url is dialed through the egress (SSRF) gate in a hosted deployment.

Grounding an incident in another source's logs (log_source)

In a real deployment the alerts live in Prometheus and the logs live in Loki — two different sources. Log capture (log_queries, above) is per-source, so on its own it can only ground a loki source's own critical finding: an alert-driven incident, on a prometheus or healthcheck source, has no logs to attach because neither of those source types can fetch any.

log_source closes that gap. It names another configured source whose recent log lines are attached to this source's incident summary:

sources:
  - name: prod
    url: http://prometheus:9090
    log_source: app-logs        # a critical alert on prod is summarized WITH app-logs' logs

  - name: app-logs
    type: loki
    url: http://loki.monitoring.svc:3100
    queries:
      - name: error_rate
        query: sum(rate({app="api"} |= "error" [5m]))
    log_queries:                # required for a log_source target
      - name: api_errors
        query: '{app="api"} |= "error"'

• The target must exist, be a loki source, and carry at least one log_queries — all three are checked at config load, so a target that would silently produce nothing fails loudly instead of shipping an empty log section.
• Naming yourself is rejected. Leaving log_source unset already means "my own logs", so a self-reference is a misunderstanding rather than a shorthand.
• Any source type may set it (including another loki source), and it changes nothing until a cycle turns critical — a firing critical alert, or a critical finding from the analysis engine. On a normal cycle no log query runs.
• Unset behaves exactly as before: a loki source with log_queries still grounds its own critical findings in its own logs.
• It REPLACES this source's own logs, it does not add to them. Setting log_source on a source that can fetch logs itself — a loki source with its own log_queries — means its critical cycles are grounded in the named source's lines and no longer in its own. There is one origin per cycle, which is what keeps the "Recent log lines from source …" line in the prompt honest. Set it on a log-capable source only when the other source's logs are the ones worth reading.
• Self-hosted YAML configuration only, today. There is no hosted-UI write path for log_source: the add-source wizard and the source edit form do not offer it, so an organization on the hosted service cannot set it. The runtime honors the key wherever it is present. Hosted-UI support is tracked in #346.
• The prompt (and the provider: none report) names the origin — "Recent log lines from source app-logs" — so the model is not told another source's logs are this one's.
• A missing target degrades, it does not fail. If the named source is not running (it failed to initialize, or a hosted plan's source cap truncated it), the cycle logs a warning and is summarized without log context.

Limitation — the lines are not correlated to the firing alert. What is captured is the log source's standing log_queries over the cycle's window. They are not filtered by the firing alert's instance / job / pod labels — loki queries are not templated at all today — so on a large, busy log source the context is "what this source was logging while the alert fired", not "what the alerting instance was logging". Scope the log_queries selectors to the streams that matter for the alerts you expect. Per-alert label templating is tracked in #347.

The privacy note under log_queries applies unchanged: the captured lines are sent to your configured LLM provider and may be quoted into the stored report.

LLM

llm:
  provider: openai   # openai | anthropic | none
  api_key: sk-...
  model: gpt-4o-mini

provider: none runs the deterministic analysis engine with no API key and no external calls; checks: and deploys.enabled are rejected in that mode. See LLM Providers for the standalone mode details, model recommendations, cost estimates, and the daily cost guard (llm.max_calls_per_day).

Notifications

notify:
  telegram:
    token: "..."     # bot token from @BotFather
    chat_id: "..."   # group, channel, or user chat ID
  slack:             # omit to disable; see docs/notifications.md for the api (threaded) transport
    mode: webhook    # "webhook" | "api"
    webhook_url: "https://hooks.slack.com/services/T00/B00/xxxx"  # required when mode is "webhook"
  email:
    smtp_host: ""    # SMTP host; empty disables the channel
    smtp_port: 587   # optional; defaults to 587 (STARTTLS)
    smtp_user: ""    # optional; set with smtp_password for authenticated relays
    smtp_password: ""
    from: "InfraSigns <[email protected]>"
    to: [[email protected]]
  pagerduty:         # incidents only — see note below
    routing_key: '${PAGERDUTY_ROUTING_KEY}'  # Events API v2 integration key; empty disables the channel
    severity: critical  # default for unspecified severity: info|warning|error|critical (default info)

Channels are optional and independent — configure any combination. Telegram, Slack, and email receive every notification (digests, trends, and incident alerts). PagerDuty is incident-only: a firing incident — an inbound alert group or a failing health check — triggers a PagerDuty alert and its resolution resolves it (correlated per episode), while digests, trends and deploy verdicts are not paged. See Notifications for setup instructions.

Scheduled reports

Both the daily digest and the trends report run on a cron schedule under reports:

reports:
  digest:
    schedule: "0 8 * * *"    # daily health digest
    model: ""                # optional llm.model override for this cycle (digest + incident summaries)
    stale_episode_after: "24h"  # an open incident episode nothing has re-asserted for this
                                # long stops standing the verdict up; defaults to 24h
  trends:                    # optional resource-utilization + trend report
    enabled: true
    schedule: "0 9 * * 1"    # e.g. Mondays at 09:00
    window: "168h"           # analysis look-back (Go duration); defaults to 7d
    step: "1h"               # range-query resolution; defaults to 1h
    timezone: "UTC"          # IANA timezone for the schedule; defaults to UTC
    model: ""                # optional llm.model override for this report
  retention_days: 90         # prune archived reports after N days; 0/absent = keep forever

The digest analyzes a range window derived from its own schedule: the gap between two consecutive runs, clamped to [1 hour, 7 days], at a resolution of about 120 points per series (step >= 30s). A daily digest therefore looks back 24 hours — anomalies and trends that fired and resolved between cycles still show up. There is no knob: the window follows reports.digest.schedule. On sub-hourly schedules consecutive windows overlap (the 1h minimum keeps a useful anomaly baseline), so a short-lived anomaly may appear in more than one digest.

Each digest (and incident summary) also carries a "What changed" section derived from the same window (no configuration, no extra state):

• New / removed scrape targets — the target set is up evaluated at the window's start and end; a target present at the end but not the start is new, one present at the start but not the end was removed. A target merely down (up=0) still emits samples, so it is not reported — only a change in service discovery membership counts. Membership is evaluated over a 15-minute lookback at each boundary, so targets scraped as slowly as every 15 minutes (and brief scrape gaps at a boundary) do not read as spurious changes. Detection needs a populated baseline, so a freshly started Prometheus with less than a window of history reports no new targets rather than flagging the whole fleet.
• Newly firing alerts — alerts whose start time falls inside the window, as opposed to recurring ones already active before it. (An alert that both fired and resolved within the window is not shown — the digest sees the currently firing set.)
• Threshold crossings — metrics that went from below a configured warn/crit threshold at the window's start to at/above it at the end. Only queries with a warn/crit hint are eligible; the comparison is between the window's first and last samples.

It is fed to the summary alongside the metric snapshot so the analysis can correlate a change — a new exporter, a freshly crossed threshold — with an anomaly. The section is omitted when nothing changed. Because the analysis window follows the schedule, a sub-hourly cadence (whose windows overlap the 1h minimum) may report the same change in several consecutive digests, and a cadence longer than 7 days can miss a change older than the 7d window cap.

Open episodes on the digest

Everything above is derived from the analysis window, which means it can only describe what the window contains. An incident that opened last Tuesday and was never resolved is older than every window a daily digest looks at, so the digest used to say nothing about it — and could legitimately render "All healthy" over a fire that had been burning for three days.

Each digest now carries one row per open episode, with the alert's name, when it started, and how long ago that was:

Still firing: DiskFull — since 3 Aug 2026, 14:20 UTC (2d)

An open episode is one the inbound alert receiver opened and nothing has resolved. (An episode nothing has re-asserted recently renders differently and does not stand the verdict up — see the last bullet below.) The rows are facts read from the incident archive, not something the model is asked to mention — but the model is told about them, so its prose can correlate a long-running fire with what the metrics are doing and will not write "no issues observed" above a row that says otherwise.

Details worth knowing:

• The verdict answers for them. The report you are sent — Telegram, Slack, email, the plain-text floor, PagerDuty — can never read "All healthy" while it carries an episode a sender is still re-asserting, and no open-episode row ever draws the green healthy accent. Such an episode at critical or warning tier is counted like any other offending signal ("1 critical"); one whose sender graded it severity: info — the case a count cannot see — renders "N still firing" at the warning tier. The tier the sender chose is never overruled; the digest only declines to call it health. An episode nothing has re-asserted lately is the exception, and it is deliberate — see An episode nothing re-asserts is demoted, not hidden below.
• The archive card is not covered yet. The same digest's entry in the web archive (and the dashboard card, and the MCP latest_digest tool) can still show its green HEALTHY badge: that badge is derived from the live alert count stored alongside the report, and a stored report carries neither the still-firing rows nor a count of them. Closing it needs a new column, so it is filed separately — read the delivered report, not the archive badge, for this.
• Which episodes appear on which source's digest. A digest is per configured source, while episodes arrive by webhook and are matched to a source only best-effort (see Incident attribution). A source's digest therefore carries the episodes attributed to it plus every episode nothing could be attributed to — so no episode is invisible, which is the whole point, while an episode attributed to another source stays on that source's digest. With no attribution rules configured, nothing is attributed and every digest carries every open episode.
• Long lists are cut, oldest kept. Episodes are ranked worst-severity first and then longest-running, and every cut keeps that prefix — the read itself asks the database for the 200 longest-running open episodes, the report renders at most 10 of them, and the prompt sees at most 10. The rendered list is closed by an "…and N more still open" note ("open", not "firing", because the cut takes the tail of the list — which is exactly where the demoted rows sort); that note is a count, not another incident, so it never adds to the verdict's own count. Past 200 the counts become a floor — a fleet in that state should be reading the incidents page, not a digest row. Two consequences of ranking by age at the read: at the 200 boundary an old info episode is kept ahead of a newer critical one, and since a demoted episode is an old one by construction, a digest whose read filled that 200 keeps its verdict amber even when every row it managed to read is demoted — the rows it did not reach are the newer ones, and it will not report an open set it could not finish reading as healthy.
• It degrades quietly. If the archive cannot be read, the digest ships without the section rather than failing; the failure is logged as a warning.
• An episode nothing re-asserts is demoted, not hidden. A deleted Alertmanager rule, a sender configured with send_resolved: false, a curl'd test alert or a decommissioned host leaves an episode open permanently — nothing ever resolves it, and retention prunes only resolved rows. Once nothing has re-asserted such an episode for longer than reports.digest.stale_episode_after (default 24h), the digest stops standing its verdict on it: the row still appears, reading

Open, not re-asserted: DiskFull — since 3 Aug 2026, 14:20 UTC (2d), last seen 5d ago

but at the info tier, outside the "N still firing" count, and sorted below every episode that is still being re-asserted. A digest whose only open episodes are stale can read "All healthy" again — which is the point: a permanently amber verdict is noise that hides the next real episode. The row never disappears, because nothing resolved it and the report must not claim otherwise.

The clock is the episode's last re-assert, not its start: a repeat firing from the sender refreshes it, so an alert Alertmanager is still repeating never goes stale however long it has been open. Two things have to be true for that refresh to happen, and InfraSigns enforces both rather than assuming them:

• The sender has to repeat at all. Alertmanager and Grafana re-notify an unchanged group every repeat_interval. CloudWatch does not — an SNS alarm notification is a state change, so an alarm sitting in ALARM sends nothing more. Its updated_at would freeze at the transition, so CloudWatch episodes are never demoted on this clock, however long they have been open. Nothing can tell an abandoned CloudWatch episode from a burning one, so the remedy for those is the incident page's Mark resolved, not a timer.
• The repeat has to survive the deduper. A repeat inside webhook.dedupe_window is suppressed and never reaches the archive, so the real refresh period is at least the longer of dedupe_window and the sender's repeat_interval. stale_episode_after must therefore be longer than webhook.dedupe_window; a config where it is not is rejected at load, with both values in the message.

The 24h default is six missed re-asserts at Alertmanager's own default repeat_interval of 4h (with the 5m default dedupe window, which is nowhere near binding), so a receiver outage or a paused sender cannot demote a live fire. There is deliberately no "off" value — 0s and negatives are rejected at config load, since a zero would be indistinguishable from an absent key; to keep the pre-stale_episode_after behaviour, set something very large ("87600h").

To make the row go away entirely rather than be demoted, close the episode: either make the sender send its resolution, or use the incident page's Mark resolved action. The two halves are deliberately independent — the demotion handles the sender that went quiet without anyone noticing, the manual close handles the alert an operator knows will never resolve. - Cloud: incidents are received process-wide today, so a hosted organization has no incident rows of its own and the section stays empty for it. The read is organization-scoped, so it starts working the moment per-tenant reception does.

Trend detection needs at least 4 data points: for trends, window / step >= 4 is validated at config load (e.g. window: 1h at the default 1h step is rejected with a clear error instead of silently reporting every trend as "stable").

Each report — and the webhook receiver via webhook.model — can override the global llm.model; see LLM Providers.

Every delivered report lands in the web archive (the reports table). reports.retention_days prunes rows older than N whole days — the prune runs at startup and after each persisted report, and deletes the report's feedback with it. 0 (the default) keeps everything forever; retention is opt-in so an upgrade never silently deletes history.

Incident episodes from the webhook receiver accumulate in the incidents table the same way; the sibling incidents.retention_days (top-level) prunes RESOLVED episodes older than N days, at startup and after each persisted batch of incidents. Open (still-firing) episodes are never pruned, however old.

The source timeline journal (source_events: fetch failures/recoveries and check-verdict changes, shown on the web UI's per-source Timeline) has its own sibling knob, timeline.retention_days — pruned at startup and after each journaled transition. Only transitions are journaled, so growth is slow; the default 0 keeps them forever.

The source_health table holds one last-known health row per source. When a source is removed from config its collector stops, so that row freezes and its web UI card becomes a neutral "removed" card. source_health.retention_days prunes those frozen rows once they are older than N days — pruned at startup and after each journaled transition, the same path as timeline. A live source's row keeps updating every collection cycle, so only genuinely-removed sources are reaped; the default 0 keeps them forever. Independently of this knob, the web UI hides a removed source's card after 30 days so the sources grid stays bounded even with retention off.

Since #358 the same knob also governs source_unrecognized_severities, the per-source reading behind the Unmapped severity labels your sources send card on Settings — same shape and same prune path, because a live source rewrites its reading every cycle too. The freeze behaviour is nearly the same, with one difference worth knowing: a health row advances on every collection attempt, while a severity reading advances only on a successful one. So a source that keeps failing for longer than the retention window has its reading reaped while its health row stays fresh — the card then reports it as never having reported, which its health card contradicts, and correctly: there is no reading because there was no successful cycle to take one from. Two more consequences: lowering this knob deletes the evidence behind that card as well as the health rows, and — as with the sources grid — the card applies its own view-side bound, so a source you removed from config stops being listed and stops counting toward its all-clear regardless of whether you ever set retention.

In the hosted (cloud) service these three per-org knobs — reports, timeline, and source_health — are additionally capped by your subscription tier (Free 7 days / Solo 90 / Team 365). The effective retention is min(configured retention_days, tier cap), and the 0=keep-forever default is likewise capped to the tier, so history is never kept longer than your plan allows. A value below the cap is honored as-is; raising retention_days above the cap has no effect (the Billing and Settings pages show the effective, tier-clamped value). Self-hosted has no plan, so the configured value is used verbatim. incidents.retention_days is not capped per-org because incidents arrive through the process-global webhook receiver.

Each schedule is a standard 5-field cron expression. Examples:

Schedule	Meaning
0 8 * * *	Every day at 08:00
0 8 * * 1-5	Weekdays at 08:00
0 8,20 * * *	Twice a day at 08:00 and 20:00

Digest schedules are evaluated in UTC. The trends report uses reports.trends.timezone (defaults to UTC).

Web UI (experimental)

ui:
  enabled: true   # serves the embedded web UI under /app; off by default
  charts:
    enabled: true # source-detail live metric charts (default on when the UI is on)

An embedded dashboard over the same store that backs your digests, incidents, and reports, plus an add-source wizard that probes Prometheus live and generates config for you. The wizard also generates config blocks for CloudWatch, DigitalOcean, Hetzner, healthcheck, and Loki sources (region / resource types / endpoints / LogQL queries / Loki tenant_id + log_queries; a cloud token is emitted as a ${…_TOKEN} env placeholder) — but it does not live-verify these: that would require a token (or arbitrary-URL probing) on this unauthenticated endpoint, so they are added-and-applied and verify on their first collection. It shares server.port and is unauthenticated — restrict /app at the network layer until self-hosted auth ships. See the Web UI page for the full tour, the wizard flow, and the security posture.

Each source's detail page draws live metric charts for its configured queries, fetched on view from that source's Prometheus. They add per-view range queries to Prometheus (bounded: at most 24 charts × 8 series × ~120 points per view, 4 concurrent, one 12s budget) — but because /app is unauthenticated, the /series endpoint is a live-query path anyone who can reach the port can trigger, so it is one more reason to restrict /app at the network layer. Concurrent identical requests (same source and window — a scripted loop or many viewers on the same page) are coalesced onto a single Prometheus fan-out, so the cross-request amplification is bounded to one in-flight fetch per source/window rather than one per caller; there is no rate limit yet, and adding one is deferred until /app grows real auth (which changes the threat model). Set ui.charts.enabled: false to turn the charts and their /series endpoint off while keeping the rest of the UI; absent, the charts are on whenever the UI is.

Sign-in (hosted)

auth:
  base_url: https://app.example.com          # public origin; fixed paths are appended (no query/fragment/userinfo)
  github:
    client_id: "Iv1.abc123"                  # setting EITHER provider turns ON cloud sign-in
    client_secret: ${GITHUB_OAUTH_SECRET}    # from the environment, never committed
    # enterprise_url: https://ghe.example.com # optional; a GitHub Enterprise Server instance
  google:                                     # optional; enable one or both providers
    client_id: "1234.apps.googleusercontent.com"
    client_secret: ${GOOGLE_OAUTH_SECRET}
  # session_ttl: 720h                         # login lifetime (default 30 days)

Setting either auth.github.client_id or auth.google.client_id switches the deployment into cloud mode: the web UI stops resolving the single implicit organization and instead resolves the acting organization per request from a login session, and mounts the sign-in routes (/login, /auth/{provider}/login, /auth/{provider}/callback, /auth/logout). The login page shows a button per configured provider. Leaving auth empty (the default) keeps InfraSigns self-hosted single-tenant — no login, every request scoped to the default organization.

• Disabled until a provider's client_id is set. Cloud mode requires ui.enabled: true (the login flow is served by the web UI) and auth.base_url; each configured provider also requires its client_secret (an unconfigured provider is simply omitted, not validated).
• auth.base_url must be https for any non-localhost host: the session cookie is __Host--prefixed and therefore Secure-only, so a plain-http origin would mint a cookie the browser refuses. http://localhost (in any case, and the root-qualified localhost.) or a loopback IP literal (including an IPv6 zone such as [::1%25eth0]) is accepted for local development. The name match is ASCII-case-insensitive only, so a look-alike such as localhoſt (U+017F) is a different DNS name and is refused; short address literals like 127.1 are refused too — Go resolves those as DNS names, where a search domain can send the traffic to a remote host.
• auth.base_url must be an origin, optionally with a path prefix. Every consumer appends a fixed path to it — each provider's registered callback is <base_url>/auth/github/callback / <base_url>/auth/google/callback, the members page shows the owner <base_url>/invite/<token> to copy and send to the invited teammate, and Stripe returns to <base_url>/app/billing — so credentials (@), a query string (including a bare trailing ?) and a fragment (including a bare trailing #) are rejected by config validate rather than silently rewritten. A trailing slash is fine. "Origin" includes a host, and https://:8080 — a port and nothing else — is rejected for the reason Sources gives for every URL key: every URL built from such a base would have been hostless, so the registered OAuth callback could never match and the invite and Stripe return links named no origin.
• auth.github.enterprise_url points GitHub sign-in at a GitHub Enterprise Server instance instead of github.com. Leave it unset for github.com. A GHE instance serves the whole flow from one origin, so one base URL is enough: InfraSigns derives the endpoints it requests — <base>/login/oauth/authorize, <base>/login/oauth/access_token and the REST API base <base>/api/v3 — from it (a trailing slash is fine). Give it the instance origin, not the API base — a value whose path resolves to an /api/v3 base is rejected (so /api/v3, /api/v3/ and /api/v3/. alike), because /api/v3 is appended to derive the REST base. It must be https unless the host is loopback — the same set auth.base_url accepts, described above — and it must carry no credentials (@), query string (including a bare trailing ?) or fragment (including a bare trailing #) — config validate rejects those rather than silently rewriting the value. When sign-in is enabled, setting it without auth.github.client_id is an error: the GitHub provider would never be registered and the key would have no effect. It does not by itself turn cloud mode on — so if no provider client id is set anywhere, the whole auth block is off, this key is validated against nothing and is simply unused. What the https rule buys is transit protection and nothing more — the authorize redirect carries the CSRF state, the callback carries the authorization code, and the token exchange carries the client secret and returns the access token, all readable and replayable over plain http on a routed network. It is not a phishing or endpoint-pinning defence: this is an instance-level operator key at the same trust level as auth.github.client_secret, api.token and database.dsn — it is read once at process start, is never part of an organization's config, and cannot be set by a tenant, so anyone able to point it at a hostile host could already hand over the client secret. Endpoints are configurable for GitHub Enterprise Server; InfraSigns is not tested against a live GHE instance. There is deliberately no Google equivalent: Google has no on-prem analogue — Workspace, Cloud Identity and GCP all authenticate against the same endpoints regardless of tenant. To confirm the key took effect, check the startup log: the cloud sign-in enabled line reports github_origin, which is github.com by default and the resolved instance origin otherwise (a misspelled or mis-nested key is only warned about, so this line is the way to tell it apart from a key that was ignored).
• A GHE instance behind a private CA needs its bundle mounted. The token exchange and both profile requests use Go's default HTTP transport against the system root pool, and the published image is gcr.io/distroless/static-debian12 — the public Mozilla bundle only. If your instance presents a certificate from an internal CA, mount the CA bundle into the container and point Go at it with SSL_CERT_FILE=/path/to/ca-bundle.crt (or SSL_CERT_DIR); otherwise the first symptom is x509: certificate signed by unknown authority on the OAuth callback, after the browser round-trip has already succeeded.
• GitHub sign-in requires a verified email, on GHE too. InfraSigns takes the account's primary verified address (falling back to any verified one) and refuses the login if there is none — an unverified address cannot establish identity. On github.com that is rarely an obstacle; on a GHE instance it depends on the instance having outbound email configured, and accounts provisioned through LDAP or SAML commonly report verified: false. If every user is refused while the endpoints are demonstrably right, this is the thing to check first.
• Sessions are opaque and server-revocable (a row in the database keyed by a random cookie value), not JWTs — logout and ban take effect immediately. Expired sessions are reaped in the background; correctness never depends on the reaper (an expired session is never honored). session_ttl bounds a login before re-auth (default 30 days).
• On a brand-new first sign-in a user is bootstrapped with a personal organization; a user who belongs to several organizations picks the active one on an org-picker, and a returning user who belongs to none (e.g. removed from their only organization) lands on that picker with a Create organization action rather than a silently re-minted org. Deauthorizing a member takes effect on their next request (the acting organization is re-checked against live membership every time).
• Accounts are linked by verified email. Signing in with GitHub and later with Google (or vice versa) using the same verified email resolves to one account, not two — the second provider is attached to the existing identity.

The requested scopes are minimal: GitHub read:user + user:email, Google openid + email + profile — the verified email (used as the identity and billing address) plus display name and avatar, no repository, Drive, or Gmail access. Helm support for the hosted deployment lands with the cloud rollout; the published chart remains self-hosted single-tenant.

Billing (hosted)

Stripe billing turns the plan limits (Free / Solo / Team) into a paid product on the hosted service. It is cloud-only and requires sign-in (above) — every checkout and webhook is scoped to an organization.

billing:
  secret_key: ${STRIPE_SECRET_KEY}       # Stripe secret API key (sk_…); its presence turns billing ON
  webhook_secret: ${STRIPE_WEBHOOK_SECRET} # verifies the /billing/webhook signature (whsec_…)
  price_solo: ${STRIPE_PRICE_SOLO}       # Stripe Price ID (price_…) the Solo plan maps to
  price_team: ${STRIPE_PRICE_TEAM}       # Stripe Price ID the Team plan maps to

Setting billing.secret_key switches on the paid boundary: the /app/billing page gains Subscribe / Manage actions (owner-only), the Stripe webhook mounts at POST /billing/webhook, and the source/member/Slack quota gates fail closed on a plan read error (a read failure must never hand out a higher tier's limits once money is involved). Leaving billing empty keeps the read-only billing page and the fail-open gates — every org stays on its 14-day Team trial or Free.

• Requires sign-in and at least one price. Billing validation fails closed unless auth is enabled (it needs the session/org spine), billing.webhook_secret is set (an unverified webhook could forge plan state), and at least one of price_solo / price_team is configured.
• Secrets come from the environment, never the config file: secret_key and webhook_secret are ${ENV} references. The price IDs are not secret but are referenced the same way for parity.
• The webhook is unauthenticated by design — Stripe calls it, and the only trust boundary is the HMAC signature over the raw body (verified with webhook_secret). Point your Stripe webhook endpoint at <base_url>/billing/webhook and subscribe it to checkout.session.completed, customer.subscription.updated, customer.subscription.deleted, and invoice.payment_failed.
• Downgrades keep the config, freeze premium capabilities. A cancelled subscription drops the plan to Free and a failed payment marks it past-due (access kept through Stripe's dunning). Your configuration is never rewritten, but on the lower plan the worker enforces the plan live: it stops delivering Slack (Free has no Slack) and stops monitoring any sources beyond the plan's source limit (the first N by config order are kept; the rest show as over-limit on the health page, uncollected). Over-cap members are retained (a downgrade never removes a teammate). History retention is also clamped to the lower tier's cap — the one data-destructive downgrade effect: existing report/timeline/source-health rows beyond the new tier's window are reaped on the next prune (re-upgrading widens the window again but cannot restore already-deleted rows). None of this rewrites the blob — re-upgrading resumes delivery and monitoring with no reconfiguration. The same rule fires when a Team trial lapses to Free (there is no Stripe event for that — it is resolved live at delivery time), so a trial that quietly ends stops delivering Slack too.

Like sign-in, billing is not wired into the published Helm chart — it lands with the hosted cloud rollout; the self-hosted chart never sets it.

Q&A bot

bot:
  telegram:                        # getUpdates long-poll
    allowed_chat_ids: [123456789]  # non-empty enables this transport (fail-closed allowlist)
    # token: ""                    # falls back to notify.telegram.token when empty
  slack:                           # Socket Mode; operators @-mention the bot
    allowed_channel_ids: [C0123ABCD]
    app_token: "xapp-..."          # required
    # bot_token: ""                # falls back to notify.slack.token when empty
  mcp_servers:                     # external MCP tool servers the agent consumes (self-hosted only)
    - name: runbooks               # [A-Za-z0-9_-], ≤40; prefixes every tool as <name>__<tool>
      url: https://runbooks.internal/mcp    # MCP HTTP (streamable) endpoint
      # token: "${RUNBOOKS_TOKEN}" # optional bearer; https required off localhost
      tools: [list_runbooks, get_runbook]   # per-server allowlist of tool names to expose

An LLM agent that answers infrastructure questions in Telegram and/or Slack using the same read-only tools as the MCP server. Each transport is enabled by its own fail-closed allowlist. Requires a reasoning llm.provider (openai/anthropic); one question costs up to 9 calls against the shared llm.max_calls_per_day. See the Q&A bot page for the security model and operational notes.

bot.mcp_servers lets the agent consume read-only tools from external MCP servers (a runbook server, a cloud provider's endpoint), namespaced as <name>__<tool> and gated by a per-server tools allowlist. Validated whenever present, but only consumed when a Q&A transport is enabled — and, like the whole AI surface, suppressed in the hosted service. See External MCP tool servers.

API authentication

POST /api/digest/trigger runs a digest cycle on demand — a billable LLM call plus user-visible notifications — so it requires a bearer token and is disabled until one is configured:

api:
  token: '${API_TOKEN}'   # min 16 characters; empty disables the trigger endpoint and /mcp

Requests must send Authorization: Bearer <token>. The bundled .env.example ships a dev-only token so the compose quickstart works out of the box — replace it for anything reachable beyond localhost. The infrasigns digest trigger CLI picks the token up from $API_TOKEN (or --token).

Both surfaces this token gates — the digest trigger and the MCP /mcp endpoint — run against the process configuration and the default organization, so in the hosted (cloud sign-in) mode they are suppressed regardless of api.token: they would act on the operator's data rather than a signed-in organization's, and each organization's digests already run automatically per organization. The token therefore only enables these surfaces in self-hosted mode.

The read-only endpoints — /healthz, /readyz, /metrics, /sources — are deliberately unauthenticated: they expose operational health, not data, and platform probes (Kubernetes, load balancers) need them open. Restrict them at the network layer if required.

Upgrading from a version without api.token: the trigger endpoint was previously unauthenticated and is now disabled until a token is configured. Add API_TOKEN=<16+ chars> to your existing .env (compose) or secret (Kubernetes) and, for automation calling the endpoint directly, send Authorization: Bearer <token>.

HTTP endpoints

Method	Path	Auth	Purpose
GET	/healthz	none	liveness: status + uptime, always 200
GET	/readyz	none	readiness: sources/notify are live probes (dial the source / notifier); subsystems (scheduler, bot-telegram, bot-slack, webhook, mcp) report whether each enabled component started and its worker is running. The bot transports additionally report disconnected — advisory, non-gating — when their live chat connection is lost (see bot readiness; mirrored by the infrasigns_bot_connected gauge). 503 when degraded
GET	/metrics	none	Prometheus scrape (metrics catalog)
GET	/sources	none	latest persisted per-source health
POST	/api/digest/trigger	Bearer api.token	manual digest run (?source=, ?wait=true); disabled without a token, and route absent entirely in cloud sign-in mode (suppressed regardless of token)
POST	/webhook/alerts	Bearer webhook.token	inbound alert receiver (details); route absent without a token
POST	/webhook/deploys	Bearer webhook.token	deploy events for post-deploy verification (details); route absent unless deploys.enabled
GET/POST/DELETE	/mcp	Bearer api.token	MCP server (details); route absent without a token, and absent entirely in cloud sign-in mode (suppressed regardless of token)
GET	/app*	none	embedded web UI (project-slugged: /app/default/...); present only when ui.enabled: true
POST	/app/sources/verify	none	wizard probe: two fixed read-only GETs against the entered URL; aggregate reply only
POST	/app/sources/preview	none	wizard preview analysis: bounded 24h range fetch + deterministic engine findings; no LLM
POST	/app/sources/config	none	wizard config generation (pure computation, no network)
POST	/app/{project}/reports/{id}/feedback	none	records digest feedback (vote + note) from the web UI — an unauthenticated write; restrict /app at the network layer like the other UI routes
POST	/app/{project}/incidents/{id}/resolve	none (cloud: any member of the organization)	closes one still-firing episode by hand (details) — the same unauthenticated write in self-hosted; a sender that is still repeating the alert re-opens the episode on its next send
GET	/assets/*	none	embedded UI static assets; present only when ui.enabled: true

The unauthenticated endpoints expose operational health, not data — restrict them at the network layer if required (see API authentication).

Metrics collected

InfraSigns queries these node_exporter metrics — the same golden-signal set the add-source wizard suggests for node jobs, so preview findings match what the runtime produces. Each carries warn/crit thresholds for the deterministic analysis engine:

# cpu_usage_percent — warn 80, crit 95
(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100
# memory_usage_percent (per instance) — warn 85, crit 95
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
# disk_usage_percent (worst non-tmpfs/overlay mount per instance) — warn 80, crit 90
max by (instance) ((1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100)

If node_exporter is not present, metric queries return empty results (logged as warnings) and the digest is generated from active alerts only.