Observability

InfraSigns exposes its own operational surface on the same HTTP server as the rest of the app (server.port, default 8080): liveness and readiness probes, and a Prometheus metrics endpoint. All three are unauthenticated — they expose operational health, not data, and platform probes (Kubernetes, load balancers, uptime monitors) need them open. Restrict them at the network layer if required.

Endpoint	Purpose	Status codes
`GET /healthz`	Liveness — is the process up?	always `200`
`GET /readyz`	Readiness — is the process up, and is everything it depends on reachable?	always `200` while the process answers; the diagnosis is the body's top-level `status` (`ok` / `warning`)
`GET /metrics`	Prometheus scrape	`200`

Liveness — `/healthz`

/healthz checks nothing external. It answers 200 OK as long as the process is running and its HTTP server can serve a request — exactly the signal a liveness probe wants (restart the pod only when the process itself is wedged, never because a downstream dependency is briefly unavailable).

{ "status": "ok", "uptime": "2h0m0s" }

uptime is the time since the process started, truncated to whole seconds.

Use /healthz for a Kubernetes livenessProbe. Never point a liveness probe at /readyz: it answers 200 for every process that can answer at all, so it would restart nothing — and it dials most configured sources and notification channels on each call, which is a real cost to pay every few seconds for a signal that cannot move. (Most, not all: a healthcheck source and a channel routed to no feed both answer ok without dialing, for the reasons given under /readyz below.)

Readiness — `/readyz`

/readyz reports whether the components InfraSigns depends on are alive. It runs every registered check concurrently under a 5-second budget. A failing check never cancels its siblings, so the response always reflects the true state of every component.

Nothing here gates the pod, and the HTTP code carries no fault information (#507). Every group — sources, notify and subsystems alike — is advisory: a failing check is named in the body and moves the top-level status word, and it does not take the pod out of its Service endpoints. So read this endpoint for what it is, a start-marker with a diagnostic body rather than a gate:

A 200 from /readyz is not evidence that any dependency is reachable. It says the process is up and answered. The diagnosis is the top-level status word and the body under it; the machine signal is infrasigns_readiness, which is what replaced the status code.

top-level `status`	code	means
`ok`	`200`	every entry in the body reads `ok`
`warning`	`200`	something in the body does not — an unreachable source, a channel that will not dial, a database that will not answer a `Ping`, a chat transport that lost its connection. The process serves; a dependency it uses is down
`degraded`	`503`	unreachable in the shipped binary, on any deployment and with or without the chart. No group is registered as gating, so nothing can produce this value or this code. It is kept as the seam a multi-replica deployment would flip back — one argument in `health.runHealthChecks` — and it is written down here so its absence is not read as health

Why nothing gates. A NotReady verdict can only remove this pod from its Service, and the chart caps replicaCount at 0 or 1 — values.schema.json says "maximum": 1 and deployment.yaml refuses anything else outright — so there is never a healthy sibling for traffic to move to. What eviction actually removes is the last endpoint, and with it the web UI that would explain the fault, /metrics (which is what a ServiceMonitor scrapes) and the webhook receiver that ingests your alerts. About thirty seconds of Postgres unavailability would have been enough to trigger it — the chart sets periodSeconds: 10 and leaves Kubernetes' default failureThreshold: 3 — which a managed-database failover, a pooler restart or a pg_upgrade all produce. For a monitoring product that is backwards: it is needed most exactly when something it watches is down.

First install is still gated, elsewhere and harder. serve pings and migrates the database before it binds the HTTP listener, and exits if it cannot, so a process that cannot reach its database never answers this endpoint at all; and on the chart's default arm the pre-install hook Job runs migrate up first, so helm install fails there rather than on a probe. What stopped is the STEADY-STATE eviction, which at one replica bought nothing.

curl -s /readyz | jq -r .status answers warning, not ok, while anything in the body is broken — that word is the only fault signal in the response, and it is what a shell check should compare. Automation that reads the HTTP code learns nothing beyond "the process answered"; point it at infrasigns_readiness == 0 instead.

The checks are grouped into three maps:

{
  "status": "ok",
  "sources": {
    "production": "ok",
    "staging": "error: dial tcp 10.0.0.5:9090: connect: connection refused"
  },
  "notify": {
    "telegram": "ok",
    "email": "ok"
  },
  "subsystems": {
    "scheduler": "ok",
    "bot-telegram": "ok"
  }
}

sources — one entry per source the deployment's config FILE declares. These are live probes: the source is reachable and its API answers. A source that failed to initialize (bad URL, unreachable at startup) surfaces its error here. They are advisory: a failing source turns the top-level status to warning, leaves the code at 200, and does not remove the pod from its Service. A source added through the web UI is deliberately NOT here — that used to be because readiness gated the whole pod, so a source added from a browser must not be able to take that browser's own UI off the network, and since #507 that reason no longer holds. What survives is narrower and still decides it: these are the OPERATOR's own declared sources, and a process-wide probe body is not the place for a tenant-added one. Widening it is a candidate, not a defect. Its health is on its own page and in GET /sources, both of which read the health records rather than this probe. GET /sources takes the api.token bearer — unlike this endpoint, it is an inventory rather than a probe — so a script reading it needs that credential. That credential is not a read key. The same api.token authorizes POST /api/digest/trigger, which spends a billable model call and delivers notifications to your channels, and the mutating /api/maintenance endpoints, which can mute an organization's alerting; there is no read-scoped token to issue instead. So gating /sources (#558) means a dashboard scraper that polls it has to hold operator WRITE authority — for some deployments a worse exposure than the inventory the gate closes. That trade was taken deliberately rather than adding a second token to issue, rotate and document, and it is stated here so you can decide it for your own deployment. If a scraper only wants up/down per source, take it from infrasigns_source_up on /metrics, which stays open and needs no token — the source NAMES are on that endpoint either way; what /sources adds is the last collector error text and the scrape timestamps.
notify — one entry per notification channel this deployment actually built. A section whose enable condition is unmet builds no channel and so has no entry here even though the section is present and valid — a telegram: block carrying a token but no chat_id, for example. A channel routed to no feed at all (feeds: []) reports ok without dialing anything: it cannot deliver, so its reachability is not a readiness question, and probing it would spend a dial on a channel that sends nothing. (Until #507 the argument was stronger than that — a parked channel that failed its probe would have evicted the pod, taking the UI, the webhook receiver and MCP with it. No channel can do that any more; this group is advisory too, and a failing channel shows as error: … beside a top-level warning and a 200.) Every other channel — including one routed to some feeds but not others — is a live probe of reachability and transport, not of the send action itself: the email channel dials the SMTP server and negotiates STARTTLS (so an unreachable host, missing STARTTLS, or expired cert shows up here), and Slack performs a lightweight reachability check. They deliberately stop short of the rate-limited credential operation — email does not authenticate on every probe — because doing so on a probe that runs every few seconds would trip a provider's per-connection auth throttle (SES, Gmail), which would then report a channel as broken while Send would in fact deliver. (Before #507 that also flapped the pod NotReady; the throttle argument stands on its own without it.) A wrong password instead surfaces at the first real Send (logged, with a delivery row and a failure metric).
subsystems — the deployment's own moving parts. One entry per enabled long-lived worker, plus two entries that are not workers. It gated the pod until #507 and no longer does; it is advisory like the other two. The set is drawn from whichever of these are turned on: database (always present), scheduler (always present), bot-telegram, bot-slack. Disabled components are simply absent from the map (and the whole subsystems key is omitted if none are registered).

database is the first entry in THIS GROUP that is not a worker, and the only one of them that can fail for an ordinary runtime reason — the sources and notify groups are full of checks that can, which is what they are for. It is a Ping against the pool under a 2-second budget — not a query: there is no RLS interaction to get wrong, no lock to contend for, and no credential re-exercised on a probe that runs every ten seconds. A Ping does wait for a pool slot, so a pool pinned for longer than two seconds reads as unhealthy, and that is the right answer rather than a residual: a process that cannot borrow a connection cannot serve a request either. It does not gate: every other surface this pod offers writes to the database, which is exactly why evicting the only pod over it removes the UI that would tell you so. Alert on infrasigns_readiness{kind="subsystem",name="database"} == 0.

The MCP mount carried an mcp entry until #552 and no longer does. It was not a worker: the mount is a stateless wrapper, so its check returned ready unconditionally and could not fail while /readyz was answering at all — it gated nothing. What it did do is appear only when api.token was set, so an unauthenticated poller of this endpoint could read whether that key was configured. Nothing replaces it, because there was nothing to replace: /mcp is mounted on every deployment and answers 401 until the token is set, and the start-up log says which of the two you are in.

The alert receiver carried a webhook entry until #558 and no longer does, for the same two reasons. Its check could only ever answer ok while the endpoint was reachable — the receiver stops in Shutdown, a process-exit path on which /readyz has already gone — so it gated nothing; and the entry appeared if and only if webhook.token was configured, which made this unauthenticated body a read of whether you had wired up a receiver. That is the bit the same release removed from the mux and from /metrics, so leaving it here would have been the third place to read it. Nothing replaces it: unlike the two chat transports the receiver has no live connection to lose — it is a request-driven HTTP handler — so there is no honest non-gating signal for it to report. infrasigns_webhook_requests_total is where receiver traffic is visible.

provisioning is the one entry that is not a worker, and it obeys the opposite rule: it is registered only when the start-up provisioning pass left this deployment running something other than what its stored document says, for a reason no config-file edit can repair. Four states do that: the stored document is refused by a rule that tightened since it was written, or it will not load at all, or an optimistic-lock race was lost on every attempt, or the write failed outright. Its PRESENCE is the signal; it reports disconnected and never ok, so there is no state in which it is a check that passes.

It does not gate the pod. For the two that are about the document, the reason is the one the sources probes follow — a source somebody added through the web UI must not be able to take the UI that would remove it off the network, and the remedy this state prints is a page this pod serves, which a gating verdict would put behind a kubectl port-forward. For the other two, gating buys nothing anyway: a lost race is settled by another process and a failed write by the database. /readyz still answers 200; its top-level status reads warning rather than ok, because since #507 that word reports whether anything in the body is not ok, and this entry is never ok.

The reason is not in the body. The advisory form carries a fixed word and no message, so /readyz says "provisioning": "disconnected" and nothing else. What to alert on is the boot ERROR — it names the cause, the remedy where there is one to name, and the offending source where the cause is a source it can name — and the the default organization's sources line beside it, which says which document the process is actually running. Its state is fixed at boot, so it does not flap: it clears on the restart that succeeds. It never appears on a deployment with no sign-in configured, where no provisioning pass runs at all.

What the subsystem checks mean (and what they don't)

Everything in this section is about the subsystems group; a failure in sources or notify reads warning and 200 and is covered above. So does a failure here — no group gates — but what the values MEAN differs, and that is what this section is for.

For the worker subsystems, the error state is a structural marker, not a connectivity probe: a component turns its entry to error only when it never started or its background worker exited, never on a transient runtime error. Both of those are boot and shutdown facts, so in a pod that is serving requests these entries read ok. That is deliberate, and it was deliberate when this group still gated: a Telegram hiccup or a momentary Slack disconnect must not be reported as though the process were broken.

database is the exception in the other direction: its error is a connectivity probe, and it is meant to be. It is the only entry in this group whose failure is a live runtime condition rather than a start-up fact — which used to be the argument for this group gating the pod, and is now the argument for watching infrasigns_readiness{kind="subsystem",name="database"}.

provisioning is neither: it is not a worker and reports no worker's liveness, and it is registered only in the failed state. It uses the same advisory overlay the bots use for a lost connection, and reads disconnected rather than error.

The two chat bots additionally carry a live-connection signal — the only subsystems with a real, losable upstream (a Telegram long-poll / Slack Socket Mode connection). When that connection is persistently lost while the worker keeps spinning (the "zombie bot": a sustained Telegram 409 from a second consumer, or a Slack dial that failed or had its token rejected), the entry reads disconnected. This value is advisory and non-gating — the HTTP code stays 200 and the pod keeps serving — because a dead chat listener must not evict the pod from serving webhooks, UI, and metrics. The top-level status word does move — it reads warning, because this entry is not ok — and that is what #507 decided: the word reports whether anything in the body is not ok, and this entry is not. Nothing about the pod's membership of its Service changed with it. It is mirrored by the infrasigns_bot_connected gauge (the machine alerting surface — alert on == 0 for 5m so a brief reconnect can't page).

provisioning borrows the same advisory value for a different reason: it has no upstream and nothing to lose, and reads disconnected because that is this endpoint's spelling for "present, and not ok, and not your pod's problem to be evicted over".

Subsystem	`error` when…	`disconnected` when…	`ok` when…
`database`	the pool will not answer a `Ping` within 2s — unreachable, out of connections, credentials rejected	—	the pool answers
`scheduler`	never started	—	the lease-scheduler poller is running
`bot-telegram`	never started, or any of its three goroutines — the poll loop, the command worker, the question worker — has exited	~15s of committed poll backoff: three consecutive `getUpdates` failures at the 5s spacing (e.g. a sustained 409) — the third flips it as it begins waiting, so ~10s of silence has actually elapsed — or one `429` asking for a longer wait	all three are alive and the transport is connected
`bot-slack`	never started, or any of its three goroutines — the event pump, the command worker, the question worker — has exited	≥3 consecutive failed connection dials, or immediately on a rejected token (`invalid_auth`) / the connection giving up	all three are alive and the transport is connected
`provisioning`	(never — it reports no worker)	always, whenever the entry exists at all	(never — absent instead)

The practical consequence: ok on a worker subsystem means "started and its worker is alive", not "connected and answering". The bots go one step further — disconnected distinguishes a lost-connection zombie from a healthy ok — and no value in this body gates the pod. provisioning is not on that scale at all: it has no ok, so read its presence, not its value. Watch the infrasigns_bot_connected gauge (and logs) for the machine signal. Source and notifier connectivity is reflected in the sources and notify groups, which are live probes and, since #507, advisory — they move the top-level word to warning and the HTTP code not at all.

The scheduler subsystem carries the same caveat with a specific edge: the lease scheduler claims every digest/trends/health-check cycle from the database, so if Postgres is unreachable no scheduled cycle runs at all — yet scheduler stays ok (the poller loop is alive, it just cannot claim). The database entry beside it does see that particular cause now — it reads error: …, the top-level word reads warning, and infrasigns_readiness{kind="subsystem",name="database"} goes to 0; the HTTP code stays 200. What it does not see is a claim scan that fails while the pool is perfectly healthy — a permission error on the schedule table, a statement timeout, a migration half-applied — because database pings the pool and runs no query. infrasigns_schedule_claim_errors_total is the signal for that; alert on it (rate() > 0).

The bots' connected/disconnected reflects the connection lifecycle, not whether the bot is answering — it does not catch every zombie. Specifically it can't see (a) a connection that fails intermittently and recovers before the debounce window closes (one success resets the streak on either transport), nor (b) a Slack socket that is connected but has silently stopped delivering events, or whose ack writes fail (Socket Mode's keepalive eventually recycles such a socket, self-healing the gauge without ever surfacing the gap). Treat connected as "the transport link is up"; pair the gauge with the infrasigns_bot_questions_total rate to catch a bot that is linked but not answering.

Metrics — `/metrics`

/metrics serves the Prometheus text exposition format. Metrics are collected into a dedicated registry (not the global default) alongside the standard Go runtime (go_*) and process (process_*) collectors, so a scrape yields both the application metrics below and the usual runtime/GC/FD stats.

Point any Prometheus at it with a plain scrape config:

scrape_configs:
  - job_name: infrasigns
    static_configs:
      - targets: ["infrasigns:8080"]

On Kubernetes, the Helm chart ships an optional ServiceMonitor for prometheus-operator — see Kubernetes → Enabling ServiceMonitor.

Both of those are things you point at your own deployment. This repository carries no scrape configuration for the hosted service itself: docker/prometheus.yml scrapes only Prometheus and the node exporter — not InfraSigns' own /metrics — the chart ships serviceMonitor.enabled: false, and the chart cannot express a hosted deployment at all (charts/infrasigns/values.yaml says so where it declines to offer llm.max_fleet_calls_per_day as an example). A hosted-only series is therefore scraped by whatever the operator runs beside the hosted process, and what that is lives outside this tree.

Metric naming

Every application metric is exported under the infrasigns_ prefix. Names follow the standard OpenTelemetry → Prometheus translation:

dotted instrument names become underscores (infrasigns.webhook.requests → infrasigns_webhook_requests);
monotonic counters gain a _total suffix (infrasigns_webhook_requests_total);
duration histograms are in seconds and gain a _seconds suffix plus the usual _bucket / _sum / _count families (infrasigns_scrape_duration_seconds);
gauges keep their bare name (infrasigns_source_up).

Metrics catalog

Metric	Type	Labels	Meaning
`infrasigns_source_up`	gauge	`org`, `source`	`1` if the last collection from this source succeeded, else `0`. `org` is the owning organization (the default org for self-hosted; the tenant org in cloud, so same-named sources across tenants stay distinct series)
`infrasigns_scrape_duration_seconds`	histogram	`source`	duration of one data-source collection (alert fetch + range query)
`infrasigns_scrape_errors_total`	counter	`source`	failed data-source collections
`infrasigns_digests_generated_total`	counter	`source`, `status`, `produced`	digest cycles after a successful collect. `status`: `success` / `failure` (LLM or notify failed) / `skipped` (no series or alerts, so summarization never ran). `produced` (#348) is WHICH SUMMARY the cycle asked for — `incident` when it escalated (a recognized `critical` alert was firing, so the incident prompt was the one that ran), `digest` for an ordinary cycle, `none` when it ended before it could choose either. It says nothing about the outcome: read it with `status`, because a cycle whose LLM call or notification failed is still recorded under the `produced` it escalated to (`{produced="incident",status="failure"}` is a real, reachable series). That is deliberate — the question `produced` exists to answer is "did this source ever escalate", and a failed escalation is still an escalation. Note this is a different question from `infrasigns_schedule_cycles_total`'s `kind`, which names WHICH SCHEDULE fired (digest / trends / checks) — same-looking values, unrelated meaning. Not all nine `(status, produced)` combinations occur: `skipped` pairs only with `none` (that path runs before the cycle knows whether its alerts are critical), and `success` / `failure` never pair with `none`, so five combinations are reachable. Adding this label was a breaking change to an existing series — an unaggregated selector now matches up to two series where it matched one, and `sum without(produced)(…)` restores the old number
`infrasigns_digest_duration_seconds`	histogram	`source`, `status`, `produced`	end-to-end collect → summarize → notify duration. Same attribute set as `infrasigns_digests_generated_total`, from one call site, so the two are joinable
`infrasigns_alerts_unrecognized_severity_total`	counter	`path`, `source`	units of work carrying at least one alert whose `severity:` label InfraSigns does not recognize (`p1`, `sev1`, `page`, …). Such a label ranks as warning on both paths, but the consequence differs, which is what `path` separates. A label you have declared in `severity_aliases` is rewritten to a canonical one as the alert arrives, so it never reaches this counter: what climbs here is the vocabulary InfraSigns has not been taught, not every vocabulary that differs from its default — which is what makes a non-zero value actionable rather than merely informational. `source` names the FLEET on both paths (#350) — this row is the operator-facing home of that meaning; the code's is `webhook.Handler.signalKey`. `path="scheduler"`: a digest collection cycle, `source` = the configured source name; the label cannot itself open an incident summary, so the fleet is capped at digests. `path="webhook"`: an inbound alert-webhook request, `source` = the configured source name the alerts' labels match — the same `alert_labels` predicates that attribute an incident, but MATCHING ONLY: incident attribution additionally has a shortcut where a single configured source with no `alert_labels` anywhere claims every alert, and this signal deliberately does not inherit it, because naming a fleet no label identified is a guess and on that shape it would fold two pushing fleets onto one key. So an install that configures no `alert_labels` keeps reporting `alertmanager` here, unchanged by #350. Because attribution is per alert and a request can carry several fleets, one request increments once per distinct `source` in it, not once in total: summing over `source` counts request-slices rather than requests, so this path's rate is not comparable with `infrasigns_webhook_requests_total`. An alert whose labels match no source's `alert_labels`, or match several, falls back to the detected payload format; of the three formats only `alertmanager` is reachable, since a CloudWatch payload carries no `severity:` label for the receiver to fail to recognize (the tier is derived from the alarm state) and an unrecognized payload yields no alerts at all. So a fleet that pushes without `alert_labels` still reports as `alertmanager`, and a source you have named `alertmanager` is indistinguishable from that fallback — the one collision this attribute has, accepted rather than namespaced so the value set stays what your dashboards already query. On this path the alerts are delivered — every channel routed to the `incidents` feed (by default all of them, see feed routing) renders them as a warning and PagerDuty pages at warning urgency instead of critical, for as long as the vocabulary goes unfixed. The signal is deliberately not gated on the outcome: a cycle that also carries a recognized `critical` alert counts here and becomes an incident summary (a fleet mid-migration, where the signal matters most). So a climbing counter alone does not prove the source only ever gets digests — alert on it being non-zero for a source at all. Since #348 the sharper rule — climbing here while that source never escalates — IS writable, because `infrasigns_digests_generated_total` now says which summary each cycle asked for: `rate(infrasigns_alerts_unrecognized_severity_total{path="scheduler",source="X"}[1d]) > 0 unless on(source) (sum by(source)(rate(infrasigns_digests_generated_total{source="X",produced="incident"}[1d])) > 0)`. Two things in that shape are load-bearing. `on(source)` rather than a bare `unless` is about label matching: this counter carries `path`+`source` while `infrasigns_digests_generated_total` carries `source`+`status`+`produced`, and `unless` matches on the FULL label set — so an unqualified `unless` finds no counterpart on the right, removes nothing, and fires for exactly the source that escalates every day (`sum by(source)` collapses `status` so the right side carries only the label being matched on). And `unless` rather than `and … == 0` fails for two independent reasons, of which only the first is label matching. `and` is a set operator too, so a bare `and` also matches on the full label set and finds no counterpart — the same defect as a bare `unless`. Adding `on(source)` fixes that half and not the other: a source that has never escalated has no `produced="incident"` series at all, so `== 0` yields an empty vector, and an `and` against an empty vector matches nothing however it is matched — so `and on(source) (sum by(source)(rate(…)) == 0)` silently never fires either, on exactly the fleet it is for. No `severity_label` attribute: the label is the operator's own free-form vocabulary and its value set is unbounded — the labels themselves ride the paired `unrecognized severity labels rank as warning` log line, which repeats at most once per `source` per 24h in the running process, so a restart re-warns on the next cycle or request. On `path="scheduler"` only, those labels are additionally persisted per source and shown to the tenant on Settings (#358) — that surface is not throttled (it is rewritten every cycle, so a corrected alias table clears it on the next one — which on the default daily schedule is a day later anyway; only a sub-daily cadence beats this counter's 24h log twin) and it is per-organization, where this counter is not. `path="webhook"` still has no such surface: #350 gave the signal the fleet's NAME, not a persisted per-source reading, and the receiver is process-global rather than part of any organization's runtime — so for a push-only fleet the counter and the WARN line remain the way to find out. The two paths also reset differently: on `path="scheduler"` a rebuilt cloud org runtime (a config save, a plan change, or an org idle long enough to age out of the runtime cache) also re-warns, while the receiver is built once per process, so only a restart clears its throttle. That throttle is now per fleet on both paths — before #350 the receiver keyed it on the payload format, so two AlertManager fleets pushing to one endpoint shared a slot and the first to warn silenced the naming of the second's labels for the cooldown
`infrasigns_notifications_sent_total`	counter	`channel`, `status`	notification delivery attempts. `channel`: `telegram` / `slack` / `email` / `pagerduty`; `status`: `success` / `failure`
`infrasigns_notify_channel_deliverable`	gauge	`channel`	`1` if this configured notification channel can deliver, `0` if the section is present but incomplete so no channel was built. `channel`: `telegram` / `slack` / `email` / `pagerduty`. A channel you never configured has NO series — absent and `0` are different claims. "Configured" means any of the section's TRANSPORT fields is set, not the one field that enables the channel (`feeds`, and PagerDuty's `severity`, are excluded: they only scope what an existing channel sends). Set once at startup from the process config, so it never moves while the process runs, and a hosted organization's own channels are not covered — the per-org runtime builds its channels without passing through this disclosure, so Notifications is where a tenant's incomplete section is reported. It gates nothing: the failure it describes used to be reported by `/readyz` at a time when a `notify` failure took the pod out of its Service, so one incomplete channel took the whole process down. No `/readyz` entry gates any more (#507) — a channel that cannot deliver reads `error: …` beside a top-level `warning` and a `200`. This series and `infrasigns_readiness` answer different questions: this one is for a section that built no channel at all, which therefore has no `/readyz` entry to report through; `infrasigns_readiness{kind="channel"}` is for a channel that WAS built and cannot reach its endpoint. Four values are reachable at `0`: a Telegram section with a token and no `chat_id`; one with a `chat_id` and no token; a `mode: webhook` Slack section whose `webhook_url` is not an http/https URL naming a remote host (`mode: api` cannot reach `0` — config validation requires the same token and channel the build gate asks for); and an email section with any transport field set while `smtp_host` is empty, which an unset `${SMTP_HOST}` produces silently. PagerDuty cannot reach `0` at all — its section's one transport field is the routing key, so being configured and being able to deliver are the same question; it carries the attribute so a future divergence has somewhere to land. Alert on `== 0`, with one qualification: three of those four are always a fault, but a Telegram token with no `chat_id` is also the legitimate shape for running the Q&A bot on that token with no delivery channel — moving it to `bot.telegram.token` removes the `notify.telegram` section and with it this series
`infrasigns_readiness`	gauge	`kind`, `name`	`1` if this `/readyz` entry read `ok` on the last probe, `0` for anything else. One series per entry of the readiness body. `kind`: `source` / `channel` / `subsystem` — the three groups the handler registers, a closed set. `name`: a source name from the config file; one of `telegram` / `slack` / `email` / `pagerduty`; or one of `database` / `scheduler` / `bot-telegram` / `bot-slack` / `provisioning`. This is the machine signal that replaced the HTTP status code: since #507 nothing on `/readyz` gates, so a `200` is not evidence about any of these entries. Alert on `== 0`. It is written BY THE PROBE, so it moves only while something polls `/readyz` — the kubelet `readinessProbe`, the compose healthcheck, and nothing at all on a bare process nobody curls. And it LATCHES rather than going stale: the exporter is a pull reader that re-emits the last recorded point on every scrape, with no explicit timestamp in the exposition, so a series nothing is refreshing is indistinguishable from a live one. `up` does not cover that — in every case below the process is up and `/metrics` is being scraped, and only the `/readyz` polling stopped. Polling that stops while an entry reads `ok` holds the series at `1` and hides a fault that develops afterwards; polling that stops while it reads `error` holds it at `0`, so a fired alert never clears; and a process nothing has ever polled has NO series here at all, so `== 0` cannot fire — that last case is the only one PromQL can see, and Suggested alerts carries the rule for it. The remedy for the other two is the polling: keep a `readinessProbe`, a compose healthcheck or an uptime monitor pointed at `/readyz`. Which values reach `0`, by producer — not every entry can, and a permanently-`1` series is not the same claim as a healthy one. `kind="source"`: a collector whose `Healthy` dials — `prometheus` (also `0` for a Prometheus that answers but has no active scrape targets), `loki`, `cloudwatch`, `digitalocean`, `hetzner` — when that dial fails; and a source whose construction failed at start-up, which returns its error on every probe and so reads `0` from boot. A `healthcheck` source cannot reach `0`: its `Healthy` is an unconditional start-marker, and a down endpoint of it is reported through its own alerts and `infrasigns_source_up` instead. Only the config file's sources are in this group, so a source added through the web UI has no series here. `kind="channel"`: any of the four notifiers whose probe refuses — an SMTP host that will not resolve or will not STARTTLS, a Telegram token the API rejects, a Slack endpoint that will not answer, a PagerDuty host that will not connect. A channel routed to no feed (`feeds: []`) cannot reach `0`: it reports `ok` without dialing, so its `1` is not a reachability claim either. `kind="subsystem"`: `database`, when the pool will not answer a `Ping` inside 2s — the one entry in this whole body whose `0` is an ordinary runtime event, and the one worth paging on; `provisioning`, which is registered only for a start-up pass that already degraded and reads `0` for as long as it exists, so its `0` is a boot-fixed fact and its ABSENCE is the healthy state; and `bot-telegram` / `bot-slack` through the lost-connection overlay (`infrasigns_bot_connected` is the dedicated series for that half). `scheduler` cannot reach `0` today, and neither bot can through its own `Healthy`: those fail only on a worker that never started or has already exited, and every worker starts before the HTTP server mounts and stops after it is gone. They carry a series so a future failure mode has somewhere to land, and so a reader can see which workers this deployment runs. Cardinality is the readiness body's own: one series per configured file source, per built channel, per enabled subsystem
`infrasigns_maintenance_suppressed_total`	counter	`source`	notifications withheld because a maintenance window was active. `source` is the configured source the notification was about, or empty when it is not scoped to one source (today only the weekly trends report, which spans every source) and an org-wide window suppressed it. Counts MESSAGES, not channels — a suppressed message never reaches a channel, so it appears in no `infrasigns_notifications_sent_total` series under any status. Read the two together: a source going quiet here while `notifications_sent` stays flat is planned silence; both flat is not
`infrasigns_reports_generated_total`	counter	`status`	trends reports by outcome (`success` / `failure`)
`infrasigns_report_duration_seconds`	histogram	`status`	trends report fetch → summarize → notify duration
`infrasigns_checks_evaluated_total`	counter	`check`, `result`	health check runs. `result`: `pass` / `fail` / `error`
`infrasigns_check_duration_seconds`	histogram	`check`, `result`	one health check fetch → verdict → notify run
`infrasigns_checks_retired_total`	counter	`status`	stored check episodes the retirement sweep acted on — a check whose name is no longer in your declared set (#416). `status`: `resolved` (an announced failing or errored episode was closed by a `Health check RETIRED` message a channel took, and the stored row deleted), `silent` (nothing had been announced for that check, so the row was deleted with no message), `deferred` (the retirement message reached no channel — every notifier failed — so the row is kept and retried), `undeleted` (the sweep settled the row — a channel took the retirement message, or there was nothing to announce for that check — and then deleting the stored row failed, so it survives and the next sweep repeats the same decision, re-announcing the retirement if there was a message; a database fault, not a delivery one, and one value for both cases because the remedy is the same). All four are reachable. A maintenance window never produces `deferred`, whatever its scope: a retirement is a resolution, and resolutions are never suppressed. `deferred` is the one to alert on: it means an incident is still open with nothing closing it. Alert on it being non-zero rather than on `rate() > 0` — how soon it retries depends on what you still have configured. With another check still scheduled it retries on the next checks cycle, so `deferred` climbs steadily; but the case this sweep exists for is deleting your last check, and with none configured there is no checks schedule to run — the retry is then the next process start (self-hosted only), so you see one increment and a flat line while the incident stays open. No `check` label: check names are unbounded free text, and a log line carries the name for each case worth chasing (a `WARN` for `deferred`, an `ERROR` for `undeleted`). Retirements are rare by nature, so a flat counter is the normal reading — it says nothing about health
`infrasigns_checks_skipped_total`	counter	`reason`	health check runs abandoned before a verdict was reached — no result persisted, no notification sent, no stored row advanced (#472). It is deliberately not a fourth value on `infrasigns_checks_evaluated_total`'s `result`: that string is the same value persisted into `check_state.result` and read by every dashboard summing evaluated runs, so a skip there would enter the pass/fail/error vocabulary and be counted as an outcome. `reason`: `fleet_llm_ceiling` (the operator's fleet-wide daily LLM ceiling refused the verdict call — see `infrasigns_llm_fleet_exhausted_total`). That is the only producible value today, so the series carries one dimension with one member; a second skip cause would join this enumeration rather than earn its own counter. Reachable only on a hosted deployment with `llm.max_fleet_calls_per_day` set — self-hosted cannot configure that ceiling, so its series is structurally absent, not zero. A skipped run is lost, not deferred: the check's next fire is its ordinary next cadence, because a check is a probe of now
`infrasigns_schedule_cycles_total`	counter	`kind`, `status`	lease-scheduler cycle claims. `kind` is WHICH SCHEDULE fired — not to be confused with `infrasigns_digests_generated_total`'s `produced`, which says which summary a digest cycle asked for. `kind`: `digest` / `trends` / `checks`; `status`: `success` / `failure` (the runtime cap fired, the cycle returned an unrecognised error, or it panicked and the poller recovered — the process survives and the claim is released) / `skipped` (nothing ran and nothing is wrong: no data to summarize, a manual run held the engine's lock, or another build held that organization's build slot for the whole wait) / `no_runtime` (this cycle had no runnable runtime, which is wider than a missing or invalid org config — a store read that failed, a build that errored, and a build that panicked all land here too; a cycle cut short by a SHUTDOWN records no outcome at all rather than this one, so a rolling deploy does not show up here) / `lease_lost` (we stopped holding the claim mid-cycle — it was re-claimed by another worker or the row was deleted, which the store cannot tell apart, and since the renewal heartbeat also runs while the runtime is being resolved it can happen before any work was done. It does not mean a duplicate run happened). The canonical definition is the vocabulary block in `internal/lease/poller.go`; this row paraphrases it and should be re-read against it. No `org` label — tenant cardinality is unbounded
`infrasigns_schedule_claim_errors_total`	counter	—	failed lease-scheduler claim-scan queries — the scheduler cannot reach its schedule table (a DB outage). While this climbs, no scheduled cycle runs, and `/readyz` still reports `scheduler: ok` (the poller is a start-marker, not a liveness probe) — the `database` entry reads `error: …` and `infrasigns_readiness{kind="subsystem",name="database"}` drops to `0` when the cause is the pool itself, and nothing but this counter sees a claim scan failing against a healthy pool. Alert on `rate() > 0`
`infrasigns_deploys_verified_total`	counter	`result`	post-deploy verifications. `result`: `pass` / `fail` / `error`
`infrasigns_deploy_duration_seconds`	histogram	`result`	one deploy verification fetch → verdict → notify run
`infrasigns_verdict_input_truncated_total`	counter	`path`, `loss`	verdict prompts whose metric data the prompt caps reduced before the model saw it (#331). `path`: `checks` / `deploys`. `loss`: `names_omitted` (a whole configured query name was dropped by the 40-name ceiling — that metric is not in the prompt at all, and a check whose assertion depends on it fails closed), `series_folded` (series were moved into their query name's aggregate line, which keeps their min/max but not which series holds a value), `labels_cut` (a listed series' identity was rendered partial, so the prompt forbids pairing on it). One increment per loss KIND per prompt, never per series, so summing over `loss` counts loss-slices rather than prompts — and the three are not interchangeable, since each has its own remedy. All three are reachable, but not equally: `names_omitted` needs a source configured with more than 40 query names, which the curated collectors cannot reach on their own (a CloudWatch source across all five namespaces is 20 names), so on a fleet of curated sources only a Prometheus or Loki source can produce it and a flat series there means the ceiling never fired — not that nothing was lost. Counted where the prompt input is assembled, so it counts prompts built: a call the daily budget then refuses is still counted, and so is one built under `llm.provider: mock`, where the facts are true of the request even though no prompt is rendered. `llm.provider: none` still reaches neither path, but since #471 the two are quiet for different reasons. `deploys`: config rejects `deploys.enabled` under `none` in both scopes — the process config and a stored organization configuration — because deploy verification has no per-organization runtime, so the write gate is the only place that question can be asked. `checks`: the process config still rejects them under `none`, but a stored organization configuration may now carry checks under it, because a tenant's `llm.provider` is not the one that would run them — the hosted worker substitutes the operator's provider and builds no checks runner at all when that effective provider is `none`, so nothing is assembled to count. When reading this counter on a hosted deployment, the `llm.provider` that matters is the operator's, never a tenant's. The same three numbers ride the paired `verdict metric data truncated` log line and a warning row on the delivered report, so this counter answers "how often, and where", not "which query". That log line is a `WARN` only when `names_omitted` fired; a run that merely folded series or shortened an identity logs it at `DEBUG`, since both follow your queries' cardinality and would otherwise warn on every run forever — which is why the counter, not the log, is the fleet-level reading for those two
`infrasigns_bot_questions_total`	counter	`transport`, `status`	Q&A bot questions. `transport`: `telegram` / `slack`; `status`: `answered` / `refused_budget` / `error` / `unauthorized` / `dropped`. `dropped` is a question whose answer path never ran: a full worker queue threw it away, or a shutdown stopped the worker while it was still queued (the queue is kept only eight deep — see docs/bot.md — so this is a reachable, not theoretical, value); `error` covers only a question whose answer path ran and failed
`infrasigns_bot_commands_total`	counter	`transport`, `command`, `status`	Deterministic bot commands (parsed before the model). `transport`: `telegram` / `slack`; `command`: `help` / `incidents` / `resolve` / `windows` / `mute` / `muteall` / `unmute` / `unknown`; `status`: `ok` / `invalid` / `unknown` / `error` / `unverified`. A command from a non-allowlisted chat is not counted here — it never becomes a command, and lands on `infrasigns_bot_questions_total{status="unauthorized"}` instead. A command thrown away without running — a full worker queue, or a shutdown that stopped the worker with it still queued — is counted `error`, since dispatch never runs for it and the request itself was fine; the one exception is a Telegram command whose addressee could not be verified, which stays `unverified` however it was lost. `unverified` is Telegram-only, and structurally so: it fires when a group-chat `/cmd@Name` names an addressee this bot cannot confirm is itself — no successful `getMe` since boot, or none within two refresh intervals, which expires a cached name the hourly refresh has stopped renewing (see docs/bot.md) — and it is recorded the same way whether the command was refused or lost to a full queue, and Slack commands never carry that ambiguity — a Slack command only ever reaches this bot already wrapped in an `app_mention` addressed to it, so there is no "which bot" question for Slack to ask, and the value can never be produced on that transport
`infrasigns_bot_connected`	gauge	`transport`	`1` if the Q&A bot transport's live connection is up, `0` once lost (non-gating; a zombie-bot signal — alert on `== 0 for 5m`). `transport`: `telegram` / `slack`
`infrasigns_webhook_requests_total`	counter	`status`	inbound alert-webhook requests. `status`: `accepted` / `unauthorized` / `invalid` / `empty` / `rejected`. Since #558 `unauthorized` is reachable on every deployment, not only one with `webhook.token` set: the route is mounted unconditionally, and the refusal that stands in for an unbuilt receiver records the same value the receiver records on its own `401` — so a POST followed by a scrape of `/metrics` (which is unauthenticated, on the same mux) no longer reports whether you configured one. The other four values need a built receiver. On a deployment with no token, treat a climbing `unauthorized` as internet scanning rather than as a misconfigured sender
`infrasigns_webhook_alerts_total`	counter	`outcome`	individual alerts inside accepted webhook requests. `outcome`: `firing` / `resolved` / `deduped`
`infrasigns_mcp_requests_total`	counter	`status`	MCP HTTP requests. `status`: `served` / `unauthorized`. Both values are reachable on every deployment since #552: `/mcp` is mounted whether or not `api.token` is set, so a deployment without one counts an `unauthorized` per request rather than serving no route — which is what makes a rising `unauthorized` on such a deployment ordinary internet scanning rather than a signal
`infrasigns_access_request_submissions_total`	counter	`status`	submissions to the public access-request form. Hosted only, and the one write route in the product that demands no credential of any kind. Say credential, not session: `/webhook/alerts`, `/webhook/deploys` and `/billing/webhook` are mounted sessionless on the same hosted mux, but each demands a bearer token or a Stripe signature, so only the credential phrasing picks this route out. Self-hosted mounts no landing page and no route, so this series is structurally absent there rather than zero. `status`: `accepted` (a row was written; a repeat of an address already on the list is accepted too, since the store upserts to one row) / `invalid` (the ADDRESS was refused — empty, over the 254-octet bound, or not a bare address) / `too_large` (the request body exceeded the 4 KiB cap, so no field was parsed at all) / `unreadable` (the body failed to parse for any other reason — reachable from a script, not from the form) / `throttled` (the rate limiter refused) / `error` (the store write failed). All six are reachable. What `throttled` sees: an increment is a submission the bucket TURNED AWAY, so the series counts victims and not the cause. The limiter is one bucket per process (60/hour, burst 10, a constant with no config key), so a poster submitting one well-formed address a minute matches the refill rate exactly, their own submissions are ACCEPTED, this counter stays flat, and the intake path can be held shut for as long as they keep it up with nothing here moving. Alert on `infrasigns_access_request_tokens_available` instead, and read this counter beside it for how many genuine visitors the condition has cost. Refusals are deliberately not logged — an anonymous caller decides how often one happens, so a per-request log line is disk they control. NOT counted here: a cross-origin POST, which the cross-origin check refuses outermost with a `403` before the handler runs (it writes a `WARN` log line instead — the one unbounded log lever this route has), and a request to the route in a deployment where it is not mounted, which the mux answers. No address and no client-identifier attribute: this application reads no client address anywhere, and the `status` vocabulary is a closed set of six owned by `internal/web`, so the counter's cardinality is six series and a caller cannot move it
`infrasigns_access_request_tokens_available`	gauge	—	tokens left in the public access-request form's rate-limit bucket, sampled at collection time. It is an observable gauge read from the limiter itself on every scrape rather than a value recorded when a request arrives, and that is the whole design: the condition it exists for is a poster submitting at exactly the refill rate, whose own submissions are accepted and who therefore generates no refusal to hang a synchronous recording on. Near zero for longer than the bucket takes to refill means the intake path is being HELD SHUT and every genuine visitor is refused; a real burst of interest drains it once and climbs back at one token a minute. No attributes: there is one bucket per process, its size and refill are compile-time constants with no config key (burst 10, 60/hour), and this application reads no client identifier, so there is nothing to key a dimension on. Hosted only, like the counter above — the route is mounted only where sign-in is configured, and where it is not this instrument observes nothing at all, so the series is structurally absent rather than zero. The remedy for a sustained refusal is at the edge in front of the process (a rate limit or a challenge at the proxy): the rate is a constant, a restart refills only the burst, and retiring rows does not touch the bucket
`infrasigns_llm_requests_total`	counter	`path`, `provider`, `status`	LLM requests by outcome. A budget-exhausted call counts as `success` on the paths that fall back to a template summary (`scheduler`, `trends`, `webhook` — they still delivered a report) and as `failure` on the verdict-class ones, which refuse rather than synthesize (`checks`, `deploys`, `bot`). `path` is which caller made the call: `scheduler` (a digest collection cycle), `trends` (a trends report), `checks` (a health check verdict), `deploys` (a post-deploy verification), `webhook` (an inbound alert-webhook request) or `bot` (one Q&A conversation turn — a question that takes N tool-calling steps records N, which is what it spends from the daily budget). Every call site that goes through the daily LLM budget reports here, with ONE exception (#472): a health check refused by the operator's fleet-wide ceiling records no sample at all — neither a `failure` nor a duration observation — because the ceiling refuses before a request leaves the process, and counting a request that was never made would also drag this histogram's neighbour toward zero. `infrasigns_checks_skipped_total{reason="fleet_llm_ceiling"}` counts that run instead. `deploys` and `bot` have no fleet arm and still record `failure` for it; the three report-class paths still record `success`, falling back to a template exactly as they do for an organization's own cap
`infrasigns_llm_request_duration_seconds`	histogram	`path`, `provider`, `status`	LLM API call duration. Split by `path` before reading a quantile: the paths run under very different budgets — `scheduler` and `trends` under the 5m cycle budget they share with every fetch in the cycle and with resolving the organization's runtime (`lease.CycleTimeout` covers `Resolve` as well as `Run`, so a cycle claimed just after a config change has less than five minutes), `checks` and `deploys` under a 2m run budget, `bot` under a 3m per-question one, `webhook` under a slice of its 28s request budget (up to 22s for a firing group; a resolved group gets only what the firing one left, which can be nothing) — so one quantile over all of them describes none of them
`infrasigns_llm_budget_used`	gauge	`org`, `provider`, `cap`	billable LLM calls consumed in the current UTC day; `cap` is the effective daily cap — the configured `llm.max_calls_per_day` self-hosted, or `min(configured, subscription-tier ceiling)` in cloud (#262). `org` is the owning organization (the default org for self-hosted; the tenant org in cloud) — the gauge is last-writer-wins state, so `org` keeps per-tenant budgets on distinct series rather than overwriting each other
`infrasigns_llm_budget_exhausted_total`	counter	`org`, `provider`	times the daily LLM budget ran out (the first call a UTC day's CAP denies); a hosted verdict call refused by the report reserve is NOT one — the day still has calls left, held for the reports (llm.md); `org` distinguishes the owning organization
`infrasigns_llm_fleet_used`	gauge	`provider`, `cap`	billable LLM calls consumed fleet-wide in the current UTC day, across every organization the operator's key serves, with the configured ceiling in `cap` (`llm.max_fleet_calls_per_day`, #472). No `org` label, deliberately: the counter is one row per day for the whole service, so an `org` dimension could never distinguish anything and a flat per-tenant series would read as "this tenant spent nothing" rather than "this dimension does not exist here". Reachable only where a fleet ceiling is configured, which config permits only in hosted mode — a self-hosted process runs one summarizer under `llm.max_calls_per_day` and its series is structurally absent, not zero. Recorded on every take that consulted the ceiling, allowed or refused, so this is the series to alert on before the ceiling binds. `infrasigns_llm_budget_used` is its per-organization sibling and the two are not summable — an organization's call is charged to both
`infrasigns_llm_fleet_exhausted_total`	counter	`provider`	times the operator's fleet-wide daily LLM ceiling ran out — the first call a UTC day's ceiling refuses, not one per refused call (#472). Distinct from `infrasigns_llm_budget_exhausted_total`, which is one organization's own cap running out: this one refuses every organization at once, reports and verdicts alike, until midnight UTC — and no tenant can act on it, which is why the tenant-facing surfaces name the cause and offer no remedy. No `org` label, for the reason `infrasigns_llm_fleet_used` states. Reachable only in hosted mode with a ceiling configured
`infrasigns_llm_budget_reserved_denied_total`	counter	`org`, `provider`	verdict-class calls (a health-check or deploy verdict, a Q&A turn) refused by the reduced ceiling the report reserve carves out (llm.md). Every refusal counts, not just the day's first, so this is how much verdict work the reservation actually turned away — an organization whose checks have all stopped for the day shows a rising count here while `..._exhausted_total` stays at zero and `infrasigns_llm_budget_used` stops moving at cap minus the reserve — a refusal accounts nothing, so nothing advances that gauge again until a report call spends from the reserve. Reachable only where a reserve is configured, which is the hosted per-organization runtime: a self-hosted process sets none, so its series stays flat and a verdict call it refuses increments `..._exhausted_total` instead. `org` distinguishes the owning organization

Duration histograms use custom bucket boundaries tuned to each operation's expected range (sub-second for a scrape, tens of seconds for an LLM-backed report), so latency percentiles are meaningful without hand-tuning le values.

Some labels are intentionally omitted to keep cardinality bounded: the webhook receiver's own counters (infrasigns_webhook_requests_total, infrasigns_webhook_alerts_total) carry no per-source label, deploy metrics no per-service label, and bot metrics no per-chat label. Slice those dimensions from the incident/report archive in the web UI, not from Prometheus. The one source label the receiver does emit — infrasigns_alerts_unrecognized_severity_total{path="webhook",source} — is the configured source name an alert's labels attribute to, falling back to the detected payload format when nothing does. It is still bounded by construction rather than by omission, by a different construction: a sender's labels only select among the sources you configured, they cannot mint one, so the value set is your own source names plus at most the three payload formats. The other inbound surface, the public access-request form's infrasigns_access_request_submissions_total, carries no client-identifier label at all: this application reads no client address anywhere, so its six status values are the whole of it.

No counter for shortened delivered text, deliberately. infrasigns_verdict_input_truncated_total above covers one half of #331 — what the prompt caps kept away from the model. The other half, text this product shortened before delivering it to a reader, gets a log line and no metric, and the asymmetry is a judgement rather than an omission: the prompt half's loss is invisible in the product's output, while the render half's is now visible in the delivered message itself (the † mark and its legend — see Notifications), which is a stronger surfacing than a counter. Grep for it instead:

report: delivered text shortened

One WARN line per delivered report, never per row, carrying report (digest / incident / trends / check / deploy), the subject (source, check, service, or sources for a trends report, which has no single subject), rows_cut, rows and summary_cut. The residual is what a counter would have bought: there is no time series, so "how often is this happening across the fleet" is a grep rather than a graph. Adding it later is one instrument plus one recorder.

The counts are the whole report's, not one channel's: a channel that drops rows to fit its own limit draws fewer marks than rows_cut names.

Suggested alerts

A few starting points — adapt the thresholds to your cadence:

groups:
  - name: infrasigns
    rules:
      # The process stopped exposing metrics at all.
      - alert: InfraSignsDown
        expr: up{job="infrasigns"} == 0
        for: 5m

      # A configured Prometheus source has been unreachable for a while.
      - alert: InfraSignsSourceDown
        expr: infrasigns_source_up == 0
        for: 15m

      # A /readyz entry has been reporting a fault. This is the alert that
      # replaced the HTTP status code: nothing on /readyz gates the pod any
      # more, so `kubectl get pods` shows 1/1 and this gauge is the only
      # machine-readable signal that a dependency is down (#507).
      #
      # The database entry is the one to page on — it is the only readiness
      # entry whose 0 is an ordinary runtime condition. The rest are worth a
      # ticket, not a page: an unreachable source or channel is usually
      # somebody else's outage, and infrasigns_source_up already covers the
      # collection half of the source case at its own cadence.
      #
      # A `for:` window works here because the value TOGGLES between probes
      # rather than ramping.
      #
      # THE GAUGE LATCHES, AND THAT IS WHAT THESE RULES HAVE TO BE READ
      # AGAINST. It is written by the /readyz handler, and the OTel Prometheus
      # exporter is a pull reader: it re-emits the last recorded point on every
      # scrape, with no explicit timestamp in the exposition, so a series nothing
      # is refreshing looks exactly like a fresh one. Three consequences, and
      # `up` covers none of them — in all three the process is up and /metrics is
      # being scraped; what stopped is the /readyz polling:
      #
      #   - polling stops while the entry reads ok   -> the series holds 1
      #     forever and a fault that develops later is invisible here;
      #   - polling stops while the entry reads error -> the series holds 0
      #     forever, so a fired alert never clears, even after the dependency
      #     recovers;
      #   - nothing ever polls (a bare binary nobody curls) -> the series is
      #     ABSENT, not 0, so `== 0` cannot fire at all. That is what
      #     InfraSignsReadinessUnreported below is for.
      #
      # So keep something polling /readyz. The chart's readinessProbe does
      # (periodSeconds: 10) and the compose healthcheck does (interval: 10s);
      # a bare process has nothing, and needs a cron or an uptime monitor
      # pointed at /readyz before any rule below means anything.
      - alert: InfraSignsDatabaseUnreachable
        expr: infrasigns_readiness{kind="subsystem",name="database"} == 0
        for: 5m

      - alert: InfraSignsDependencyUnreachable
        expr: infrasigns_readiness{name!="database"} == 0
        for: 15m

      # The process is up and scraped, and no /readyz entry has ever been
      # recorded — so nothing is polling /readyz and every rule above is
      # silent rather than satisfied. This is the one staleness case PromQL can
      # see: the others latch at a real value and are indistinguishable from a
      # live one, which is why the remedy for them is the polling itself.
      #
      # `database` is the selector because it is the entry the rule above pages
      # on, and serve registers it unconditionally — so its absence means
      # nothing has polled /readyz, never that this deployment has no such
      # dependency.
      - alert: InfraSignsReadinessUnreported
        expr: up{job="infrasigns"} == 1
          unless on(instance, job) infrasigns_readiness{kind="subsystem",name="database"}
        for: 15m

      # Notifications are failing to deliver — you may be missing digests/incidents.
      - alert: InfraSignsNotifyFailing
        expr: increase(infrasigns_notifications_sent_total{status="failure"}[1h]) > 0

      # A notification section you configured builds no channel, so nothing is
      # delivered there. The metrics catalog lists the shapes the metric covers;
      # The telegram selector is what this quiet default costs: it also mutes a
      # chat_id with no token, which is always a fault. Drop the selector — and
      # alert on the bare `== 0` — once no notify.telegram section carries a
      # token without a chat_id, since that shape is also how the Q&A bot runs.
      - alert: InfraSignsNotifyChannelIncomplete
        expr: infrasigns_notify_channel_deliverable{channel!="telegram"} == 0

      # The daily LLM budget ran out — reports fell back to template summaries.
      - alert: InfraSignsLLMBudgetExhausted
        expr: increase(infrasigns_llm_budget_exhausted_total[1d]) > 0

      # Hosted only: the FLEET-wide daily ceiling is nearly spent. Replace 800
      # with 80% of your own llm.max_fleet_calls_per_day — the ceiling rides the
      # gauge as the `cap` LABEL, and PromQL cannot do arithmetic on a label
      # value, so the threshold is a literal here rather than derived.
      #
      # Alert on this gauge, not on infrasigns_llm_fleet_exhausted_total: by the
      # time that counter moves, every organization is already refused and only
      # you can raise the ceiling. The series is ABSENT (not zero) unless
      # llm.max_fleet_calls_per_day is set, so this rule is silent on every
      # self-hosted deployment rather than firing on one.
      - alert: InfraSignsLLMFleetCeilingNear
        expr: infrasigns_llm_fleet_used > 800
        for: 5m

      # Hosted only: the public access-request form's bucket is being held empty,
      # i.e. every genuine visitor asking for access is being refused.
      #
      # Alert on this gauge, not on
      # infrasigns_access_request_submissions_total{status="throttled"}: that
      # counter moves when a VISITOR is turned away, and the poster who emptied
      # the bucket is being accepted, so it is flat in exactly this scenario.
      # The series is ABSENT (not zero) on a self-hosted deployment, which
      # mounts no such route, so this rule is silent there rather than firing.
      #
      # TWO causes satisfy it and they want OPPOSITE actions, so read that
      # counter beside the gauge to tell them apart: a poster holding the bucket
      # empty leaves rate(...{status="throttled"}[15m]) near zero, because only
      # the occasional genuine visitor is refused, while real interest arriving
      # faster than one a minute makes it large, because everybody above the
      # refill rate is turned away. Under the first the remedy is at your EDGE
      # (a rate limit or a challenge in front of the origin) — the rate is a
      # compile-time constant, a restart refills only the burst of 10, and
      # retiring rows does not touch the bucket. Under the second the limit is
      # simply too low for your traffic, and raising it is a code change today
      # (internal/web/ratelimit.go), which is where that constant's own comment
      # says it becomes a config key.
      #
      # avg_over_time rather than a bare comparison under `for: 30m`: the bucket
      # RAMPS back toward 1 between takes, so one scrape landing near the top of
      # a ramp resets a `for:` window and delays the page. A mean cannot be moved
      # by a single sample — it sits near 0.5 while the path is held shut and
      # near the burst of 10 when it is not — and a genuine burst that drains the
      # bucket once and climbs back at a token a minute stays far above 1.
      - alert: InfraSignsAccessRequestIntakeHeldShut
        expr: avg_over_time(infrasigns_access_request_tokens_available[30m]) < 1

If you configure no rule at all you are not blind: the process logs one WARN the first time a UTC day's fleet usage crosses 80% of the ceiling, and a second the first time that day's ceiling actually refuses a call. Both re-arm at midnight UTC, and the two are separate flags on purpose — a day that crosses 80% and then exhausts is two events, and sharing one flag would silence whichever landed second. Treat them as the fallback, not the primary signal: a log line is read only by whoever is already looking.

infrasigns_source_up and infrasigns_notifications_sent_total{status="failure"} are the two signals worth alerting on first: together they tell you whether InfraSigns can still see your infrastructure and reach you about it.

Dead man's switch

Alerting on the absence of a metric requires your Prometheus to still be scraping InfraSigns — which is exactly what breaks during some outages. For an independent liveness signal, enable the heartbeat ping (heartbeat.url): InfraSigns pings an external uptime monitor (healthchecks.io, Better Uptime, Dead Man's Snitch) after each successful cycle, and the monitor alerts you if the ping stops.