Observability

InfraSigns exposes its own operational surface on the same HTTP server as the rest of the app (server.port, default 8080): liveness and readiness probes, and a Prometheus metrics endpoint. All three are unauthenticated — they expose operational health, not data, and platform probes (Kubernetes, load balancers, uptime monitors) need them open. Restrict them at the network layer if required.

Endpoint	Purpose	Status codes
GET /healthz	Liveness — is the process up?	always 200
GET /readyz	Readiness — are the subsystems that matter alive?	200 ready / 503 degraded
GET /metrics	Prometheus scrape	200

Liveness — /healthz

/healthz checks nothing external. It answers 200 OK as long as the process is running and its HTTP server can serve a request — exactly the signal a liveness probe wants (restart the pod only when the process itself is wedged, never because a downstream dependency is briefly unavailable).

{ "status": "ok", "uptime": "2h0m0s" }

uptime is the time since the process started, truncated to whole seconds.

Use /healthz for a Kubernetes livenessProbe. Never point a liveness probe at /readyz — a readiness failure is meant to pull the pod out of rotation, not to kill and restart it.

Readiness — /readyz

/readyz reports whether the components InfraSigns depends on are alive. It runs every registered check concurrently under a 5-second budget and returns 200 with "status": "ok" when they all pass, or 503 with "status": "degraded" when any check fails. A failing check never cancels its siblings, so the response always reflects the true state of every component.

The checks are grouped into three maps:

{
  "status": "ok",
  "sources": {
    "production": "ok",
    "staging": "error: dial tcp 10.0.0.5:9090: connect: connection refused"
  },
  "notify": {
    "telegram": "ok",
    "email": "ok"
  },
  "subsystems": {
    "scheduler": "ok",
    "webhook": "ok",
    "mcp": "ok"
  }
}

• sources — one entry per configured Prometheus source. These are live probes: the source is reachable and its API answers. A source that failed to initialize (bad URL, unreachable at startup) surfaces its error here.
• notify — one entry per configured notification channel. These are live probes of reachability and transport, not of the send action itself: the email channel dials the SMTP server and negotiates STARTTLS (so an unreachable host, missing STARTTLS, or expired cert shows up here), and Slack performs a lightweight reachability check. They deliberately stop short of the rate-limited credential operation — email does not authenticate on every probe — because doing so on a probe that runs every few seconds would trip a provider's per-connection auth throttle (SES, Gmail) and flap the pod NotReady while Send would in fact deliver. A wrong password instead surfaces at the first real Send (logged, with a delivery row and a failure metric).
• subsystems — one entry per enabled long-lived worker. The set is drawn from whichever of these are turned on: scheduler (always present), webhook, bot-telegram, bot-slack, mcp. Disabled components are simply absent from the map (and the whole subsystems key is omitted if none are registered).

What the subsystem checks mean (and what they don't)

The subsystem error state is a structural marker, not a connectivity probe: a component turns its entry to error (which gates the pod) only when it never started or its background worker exited — never on a transient runtime error. This is deliberate: /readyz gates the entire pod out of its Service endpoints, so a Telegram hiccup or a momentary Slack disconnect must not take the webhook receiver, web UI, and /metrics offline with it.

The two chat bots additionally carry a live-connection signal — the one subsystem with a real, losable upstream (a Telegram long-poll / Slack Socket Mode connection). When that connection is persistently lost while the worker keeps spinning (the "zombie bot": a sustained Telegram 409 from a second consumer, or a Slack dial that failed or had its token rejected), the entry reads disconnected. This value is advisory and non-gating — the top-level status stays ok and the pod keeps serving — because a dead chat listener must not evict the pod from serving webhooks, UI, and metrics. It is mirrored by the infrasigns_bot_connected gauge (the machine alerting surface — alert on == 0 for 5m so a brief reconnect can't page).

Subsystem	error (gating) when…	disconnected (advisory) when…	ok when…
scheduler	never started	—	the lease-scheduler poller is running
webhook	shutting down	—	the receiver is mounted and live
bot-telegram	never started, or its worker goroutine has exited	≥3 consecutive getUpdates failures (e.g. a sustained 409)	the poll worker is alive and connected
bot-slack	never started, or its worker goroutine has exited	≥3 consecutive failed connection dials, or immediately on a rejected token (invalid_auth) / the connection giving up	the Socket Mode worker is alive and connected
mcp	(never — a mounted handler is always ready)	—	mounted

The practical consequence: ok on a non-bot subsystem means "started and its worker is alive", not "connected and answering". The bots go one step further — disconnected distinguishes a lost-connection zombie from a healthy ok — but neither value gates the pod. Watch the infrasigns_bot_connected gauge (and logs) for the machine signal. (Source and notifier connectivity is reflected in the sources and notify groups, which are live gating probes.)

The scheduler subsystem carries the same caveat with a specific edge: the lease scheduler now claims every digest/trends/health-check cycle from the database, so if Postgres is unreachable no scheduled cycle runs at all — yet scheduler stays ok (the poller loop is alive, it just cannot claim). That failure is surfaced by the infrasigns_schedule_claim_errors_total counter, not by /readyz; alert on it (rate() > 0) to catch a DB-starved scheduler.

The bots' connected/disconnected reflects the connection lifecycle, not whether the bot is answering — it does not catch every zombie. Specifically it can't see (a) a connection that fails intermittently but never three times in a row (one success resets the streak — both transports debounce on consecutive failures), nor (b) a Slack socket that is connected but has silently stopped delivering events, or whose ack writes fail (Socket Mode's keepalive eventually recycles such a socket, self-healing the gauge without ever surfacing the gap). Treat connected as "the transport link is up"; pair the gauge with the infrasigns_bot_questions_total rate to catch a bot that is linked but not answering.

Metrics — /metrics

/metrics serves the Prometheus text exposition format. Metrics are collected into a dedicated registry (not the global default) alongside the standard Go runtime (go_*) and process (process_*) collectors, so a scrape yields both the application metrics below and the usual runtime/GC/FD stats.

Point any Prometheus at it with a plain scrape config:

scrape_configs:
  - job_name: infrasigns
    static_configs:
      - targets: ["infrasigns:8080"]

On Kubernetes, the Helm chart ships an optional ServiceMonitor for prometheus-operator — see Kubernetes → Enabling ServiceMonitor.

Metric naming

Every application metric is exported under the infrasigns_ prefix. Names follow the standard OpenTelemetry → Prometheus translation:

• dotted instrument names become underscores (infrasigns.webhook.requests → infrasigns_webhook_requests);
• monotonic counters gain a _total suffix (infrasigns_webhook_requests_total);
• duration histograms are in seconds and gain a _seconds suffix plus the usual _bucket / _sum / _count families (infrasigns_scrape_duration_seconds);
• gauges keep their bare name (infrasigns_source_up).

Metrics catalog

Metric	Type	Labels	Meaning
infrasigns_source_up	gauge	org, source	1 if the last collection from this source succeeded, else 0. org is the owning organization (the default org for self-hosted; the tenant org in cloud, so same-named sources across tenants stay distinct series)
infrasigns_scrape_duration_seconds	histogram	source	duration of one data-source collection (alert fetch + range query)
infrasigns_scrape_errors_total	counter	source	failed data-source collections
infrasigns_digests_generated_total	counter	source, status	digest cycles after a successful collect. status: success / failure (LLM or notify failed) / skipped (no series or alerts, so summarization never ran)
infrasigns_digest_duration_seconds	histogram	source, status	end-to-end collect → summarize → notify duration
infrasigns_alerts_unrecognized_severity_total	counter	path, source	units of work carrying at least one alert whose severity: label InfraSigns does not recognize (p1, sev1, page, …). Such a label ranks as warning on both paths, but the consequence differs, which is what path separates. A label you have declared in severity_aliases is rewritten to a canonical one as the alert arrives, so it never reaches this counter: what climbs here is the vocabulary InfraSigns has not been taught, not every vocabulary that differs from its default — which is what makes a non-zero value actionable rather than merely informational. path="scheduler": a digest collection cycle, source = the configured source name; the label cannot itself open an incident summary, so the fleet is capped at digests. path="webhook": an inbound alert-webhook request, source = the detected payload format, not the fleet that sent it — and in practice always alertmanager, since a CloudWatch payload carries no severity: label for the receiver to fail to recognize (the tier is derived from the alarm state) and an unrecognized payload yields no alerts at all; here the alerts are delivered — every channel renders them as a warning and PagerDuty pages at warning urgency instead of critical, for as long as the vocabulary goes unfixed. The signal is deliberately not gated on the outcome: a cycle that also carries a recognized critical alert counts here and becomes an incident summary (a fleet mid-migration, where the signal matters most). So a climbing counter alone does not prove the source only ever gets digests — alert on it being non-zero for a source at all. No exported instrument separates an incident cycle from a digest one (infrasigns_digests_generated_total counts both, keyed by success/failure/skipped), so the sharper rule — climbing here while that source never escalates — cannot be written until #348 lands. No severity_label attribute: the label is the operator's own free-form vocabulary and its value set is unbounded — the labels themselves ride the paired unrecognized severity labels rank as warning log line, which repeats at most once per source per 24h in the running process, so a restart re-warns on the next cycle or request. On path="scheduler" only, those labels are additionally persisted per source and shown to the tenant on Settings (#358) — that surface is not throttled (it is rewritten every cycle, so a corrected alias table clears it on the next one — which on the default daily schedule is a day later anyway; only a sub-daily cadence beats this counter's 24h log twin) and it is per-organization, where this counter is not; path="webhook" has no such surface, because the signal is not keyed on the fleet (#350). The two paths reset differently beyond that: on path="scheduler" a rebuilt cloud org runtime (a config save, a plan change, or an org idle long enough to age out of the runtime cache) also re-warns, while the receiver is built once per process and is not part of the per-org runtime, so only a restart clears its throttle. That throttle is additionally coarser than it looks, because its key is the payload format: two AlertManager fleets pushing to one endpoint share a slot, so the first to warn silences the naming of the second's labels for the cooldown — the counter still moves for both (#350 tracks keying it on the sender instead)
infrasigns_notifications_sent_total	counter	channel, status	notification delivery attempts. channel: telegram / slack / email / pagerduty; status: success / failure
infrasigns_reports_generated_total	counter	status	trends reports by outcome (success / failure)
infrasigns_report_duration_seconds	histogram	status	trends report fetch → summarize → notify duration
infrasigns_checks_evaluated_total	counter	check, result	health check runs. result: pass / fail / error
infrasigns_check_duration_seconds	histogram	check, result	one health check fetch → verdict → notify run
infrasigns_schedule_cycles_total	counter	kind, status	lease-scheduler cycle claims. kind: digest / trends / checks; status: success / failure / skipped (nothing due to run this cycle, or a manual run held the lock) / no_runtime (org config missing or invalid — cloud degrade path) / lease_lost (lease expired mid-cycle and was re-claimed). No org label — tenant cardinality is unbounded
infrasigns_schedule_claim_errors_total	counter	—	failed lease-scheduler claim-scan queries — the scheduler cannot reach its schedule table (a DB outage). While this climbs, no scheduled cycle runs, and /readyz still reports scheduler: ok (the poller is a start-marker, not a liveness probe). Alert on rate() > 0
infrasigns_deploys_verified_total	counter	result	post-deploy verifications. result: pass / fail / error
infrasigns_deploy_duration_seconds	histogram	result	one deploy verification fetch → verdict → notify run
infrasigns_bot_questions_total	counter	transport, status	Q&A bot questions. transport: telegram / slack; status: answered / refused_budget / error / unauthorized
infrasigns_bot_connected	gauge	transport	1 if the Q&A bot transport's live connection is up, 0 once lost (non-gating; a zombie-bot signal — alert on == 0 for 5m). transport: telegram / slack
infrasigns_webhook_requests_total	counter	status	inbound alert-webhook requests. status: accepted / unauthorized / invalid / empty / rejected
infrasigns_webhook_alerts_total	counter	outcome	individual alerts inside accepted webhook requests. outcome: firing / resolved / deduped
infrasigns_mcp_requests_total	counter	status	MCP HTTP requests. status: served / unauthorized
infrasigns_llm_requests_total	counter	path, provider, status	LLM requests by outcome. A budget-exhausted call counts as success on the paths that fall back to a template summary (scheduler, trends, webhook — they still delivered a report) and as failure on the verdict-class ones, which refuse rather than synthesize (checks, deploys, bot). path is which caller made the call: scheduler (a digest collection cycle), trends (a trends report), checks (a health check verdict), deploys (a post-deploy verification), webhook (an inbound alert-webhook request) or bot (one Q&A conversation turn — a question that takes N tool-calling steps records N, which is what it spends from the daily budget). Every call site that goes through the daily LLM budget reports here
infrasigns_llm_request_duration_seconds	histogram	path, provider, status	LLM API call duration. Split by path before reading a quantile: the paths run under very different budgets — scheduler and trends under the 5m cycle budget they share with every fetch in the cycle, checks and deploys under a 2m run budget, bot under a 3m per-question one, webhook under a slice of its 28s request budget (up to 22s for a firing group; a resolved group gets only what the firing one left, which can be nothing) — so one quantile over all of them describes none of them
infrasigns_llm_budget_used	gauge	org, provider, cap	billable LLM calls consumed in the current UTC day; cap is the effective daily cap — the configured llm.max_calls_per_day self-hosted, or min(configured, subscription-tier ceiling) in cloud (#262). org is the owning organization (the default org for self-hosted; the tenant org in cloud) — the gauge is last-writer-wins state, so org keeps per-tenant budgets on distinct series rather than overwriting each other
infrasigns_llm_budget_exhausted_total	counter	org, provider	times the daily LLM budget ran out (first denied call of a UTC day); org distinguishes the owning organization

Duration histograms use custom bucket boundaries tuned to each operation's expected range (sub-second for a scrape, tens of seconds for an LLM-backed report), so latency percentiles are meaningful without hand-tuning le values.

Some labels are intentionally omitted to keep cardinality bounded: the webhook receiver's own counters (infrasigns_webhook_requests_total, infrasigns_webhook_alerts_total) carry no per-source label, deploy metrics no per-service label, and bot metrics no per-chat label. Slice those dimensions from the incident/report archive in the web UI, not from Prometheus. The one source label the receiver does emit — infrasigns_alerts_unrecognized_severity_total{path="webhook",source} — is the detected payload format, a closed three-value set, not a sender identity, so it is bounded by construction rather than by omission.

Suggested alerts

A few starting points — adapt the thresholds to your cadence:

groups:
  - name: infrasigns
    rules:
      # The process stopped exposing metrics at all.
      - alert: InfraSignsDown
        expr: up{job="infrasigns"} == 0
        for: 5m

      # A configured Prometheus source has been unreachable for a while.
      - alert: InfraSignsSourceDown
        expr: infrasigns_source_up == 0
        for: 15m

      # Notifications are failing to deliver — you may be missing digests/incidents.
      - alert: InfraSignsNotifyFailing
        expr: increase(infrasigns_notifications_sent_total{status="failure"}[1h]) > 0

      # The daily LLM budget ran out — reports fell back to template summaries.
      - alert: InfraSignsLLMBudgetExhausted
        expr: increase(infrasigns_llm_budget_exhausted_total[1d]) > 0

infrasigns_source_up and infrasigns_notifications_sent_total{status="failure"} are the two signals worth alerting on first: together they tell you whether InfraSigns can still see your infrastructure and reach you about it.

Dead man's switch

Alerting on the absence of a metric requires your Prometheus to still be scraping InfraSigns — which is exactly what breaks during some outages. For an independent liveness signal, enable the heartbeat ping (heartbeat.url): InfraSigns pings an external uptime monitor (healthchecks.io, Better Uptime, Dead Man's Snitch) after each successful cycle, and the monitor alerts you if the ping stops.