Configuration
InfraSigns is configured via a single YAML file. Pass the path with --config (default: config/config.yaml).
Full example
locale: en # LLM report language (ISO code): en | ru. Empty defaults to en.
severity_aliases: # optional: map your own severity labels onto info|warning|critical
p1: critical # without it, any other label reads as a warning
server:
port: 8080 # HTTP server: health/metrics/API/webhook (+ /app UI)
database:
# The shipped config injects this from the environment: dsn: '${DATABASE_DSN}'.
# A literal value works too; the bundled compose Postgres uses
# postgres://infrasigns:infrasigns@postgres:5432/infrasigns?sslmode=disable
dsn: '${DATABASE_DSN}'
sources:
- name: production
url: http://prometheus:9090
- name: staging
url: http://staging-prometheus:9090
llm:
provider: openai # openai | anthropic | none (deterministic engine, no key)
api_key: sk-...
model: gpt-4o-mini
notify:
telegram: # optional; delivery needs BOTH fields below
token: "123456:ABC-..."
chat_id: "-100123456789" # empty delivers nothing; the token stays available to the Q&A bot
slack: # optional; omit the block to disable. Pick ONE transport:
mode: webhook # "webhook" (incoming webhook) or "api" (Web API, threads per episode)
webhook_url: "https://hooks.slack.com/services/T00/B00/xxxx" # required when mode is "webhook"
# mode: api
# token: "xoxb-…" # bot token (chat:write scope); required when mode is "api"
# channel: "C0123ABCD" # channel ID; required when mode is "api"
email: # optional; empty smtp_host disables the channel
smtp_host: ""
from: "InfraSigns <[email protected]>"
to: [[email protected]]
pagerduty: # optional; INCIDENTS ONLY (digests are not paged)
routing_key: '${PAGERDUTY_ROUTING_KEY}' # Events API v2 integration key; empty disables the channel
severity: critical # default for unspecified severity: info|warning|error|critical (default info)
reports:
digest:
schedule: "0 8 * * *" # cron expression, UTC
trends: # optional weekly-style trends report
enabled: true
schedule: "0 9 * * 1"
window: "168h"
step: "1h"
retention_days: 90 # prune archived reports after N days; 0/absent = keep forever
webhook: # inbound alert receiver — see Webhook receiver page
token: '${WEBHOOK_TOKEN}' # min 16 chars; empty disables the endpoint
dedupe_window: "5m"
incidents: # incident history (webhook episodes)
retention_days: 90 # prune RESOLVED episodes after N days; 0/absent = keep forever
timeline: # source timeline journal (web UI per-source Timeline)
retention_days: 90 # prune events — and finished maintenance windows — after N days; 0/absent = keep forever
source_health: # frozen per-source rows: health, and severity readings (#358)
retention_days: 90 # prune removed sources' rows after N days; 0/absent = keep forever
api:
token: '${API_TOKEN}' # guards POST /api/digest/trigger, /api/maintenance, /mcp and GET /sources; empty disables all four
ui:
enabled: false # experimental web UI under /app
charts:
enabled: true # source-detail live metric charts; false disables them + the /series endpoint
heartbeat:
url: '${HEARTBEAT_URL}' # dead-man's-switch ping; empty disables
checks: # natural-language health checks — see Health checks page
- name: disk-headroom
schedule: "*/30 * * * *"
source: production
query: "Is disk usage below 85%?"
mode: alert # alert = notify on transitions; monitor (the default) records only
severity: warning # info|warning|critical — tier a FAILING check pages at
for_runs: 1 # consecutive non-pass runs before the episode opens
repeat_interval: "30m" # re-assert an open failing episode; omit = never
# the three above are notification policy — inert in monitor mode
deploys: # post-deploy verification — see its page
enabled: true
source: production
delay: "5m"
bot: # Q&A bot — see its page
telegram:
allowed_chat_ids: [123456789] # non-empty enables this transport (fail-closed allowlist)
# token: "" # falls back to notify.telegram.token when empty
slack: # Socket Mode; operators @-mention the bot
allowed_channel_ids: [C0123ABCD] # non-empty enables this transport
app_token: "xapp-..." # required (Socket Mode app-level token)
# bot_token: "" # falls back to notify.slack.token when empty
Language (locale)
locale is a top-level setting (per-org in the hosted service, a deployment
default when self-hosted). It sets the language of the LLM-generated report
prose — the digest, incident, trends, and inbound-alert summaries: the model
is instructed to write the summary and observation text in the configured
language, while severity levels and other machine-readable fields stay canonical.
The LLM summary providers (openai, anthropic) honor it for the report
prose, and the deterministic providers (llm.provider: none and the
mock/budget-exhausted fallbacks) localize their fixed chrome — headers,
verdict labels, recommendations, alert counts — through the built-in catalog,
while leaving in English the interpolated values (metric names, trend words,
engine finding details) and the change-tracking and recent-log-line sections
(shared verbatim with the English LLM prompt).
The notification report chrome is also localized across every channel
(Telegram, Slack, email, plain-text): the report title, the verdict line
(counts, severity words and recommended-action clause, with CLDR-correct
plurals), the section labels, the value-column direction word, and the email
subject/<html lang>. Locale-neutral symbols stay fixed — the status glyphs,
the value arrows (↑/↓/→), the ASCII badges ([OK]/[WARN]/[CRIT]) and the
INFRASIGNS wordmark — and the claim prose and metric values remain canonical.
One known residual: the report period line (the analysis-window label and
timestamp, e.g. 24h to 17 Jul, 09:00 UTC) uses English date formatting and
connector regardless of locale.
The web UI localizes its chrome progressively. Localized so far: the app
shell (sidebar and topbar navigation, the account and organization menus, ARIA
labels), the Dashboard, the sources, reports, incidents, incident-detail, and
settings pages (headings, filters, empty/error states), the add-source wizard
(shared by the first-run onboarding page), and the source-detail and
notifications pages (status/endpoint/queries/timeline chrome, the channel cards
and every channel edit form), and the interleaved-<code>/<a>/<strong> form
hints and channel intros across those pages (the settings intro link, the wizard
and edit-form hints, the notifications channel intros, and the dashboard
run-digest CTA), and the Go view-model prose (the Settings and Notifications
read-only view models — channel field labels, feed names and status lines,
settings rows with CLDR plurals — the source-card state labels/detail, the
browser-tab page titles, and the sign-in landing), and the organization picker —
the page a session lands on when it belongs to several organizations, or to none.
Every page whose body is localized sets <html lang> to the configured locale —
the app shell, onboarding, sign-in, the organization picker, and the members,
billing and invite pages.
These surfaces still render English pending the final web slices (#201): the
Alpine runtime labels that travel via data-* to JavaScript (the two reports
toggle labels, the wizard and healthcheck-editor runtime labels — Slice 3e-3);
the health-check and deploy-verification LLM rationales (a separate verdict
prompt, not localized yet even on the real providers); and a follow-up
Go-handler-prose slice for the members and billing standalone pages and
the dashboard check-card / incident-row timeline prefixes ("since …"). A locale
you set there validates and is accepted, but those surfaces stay English until
then.
Severity aliases (severity_aliases)
InfraSigns understands three severity labels — info, warning and critical.
Anything else your alert rules write reads as a warning, which is
conservative but costly if your fleet is on a p1/sev1/page scheme: a
collection cycle can never become an incident summary, and a pushed alert is
delivered and paged at warning urgency instead of critical (see
the alert webhook). Teach it your vocabulary instead of rewriting
every alert rule:
- The label is rewritten once, as the alert arrives — on collection and on
the inbound webhook alike — so everything downstream sees the canonical label:
the incident decision, the notification tier and PagerDuty urgency, the
ordering the LLM prompt cuts by, and the
get_active_alertsMCP tool. - Your original label is not destroyed. Only the alert's severity reading
changes; the alert's own
severitylabel keeps whatever your rule wrote. That is also why turning the table on, editing it, or removing it cannot re-open or re-key an incident episode: episode identity is built from the labels, which the rewrite never touches. - Names are matched the way the label is read — case-insensitively and
whitespace-trimmed, so one
p1:entry coversP1and a padded" p1 ". Two entries that differ only in case or padding are rejected at startup rather than letting one of them win at random. - Targets are
info,warningorcritical, spelled exactly. A typo is a startup error, not a label that quietly keeps reading as a warning at 3am. - The canonical labels themselves cannot be aliased.
critical: warningis rejected: every notification tier and every claim in these docs rests on those three words meaning what they say.warn— which InfraSigns already reads as a warning — is aliasable, since it is not one of the three. - An aliased label stops being unrecognized, which is the point: the WARN
line and
infrasigns_alerts_unrecognized_severity_totalgo quiet for it, because you answered the question they were asking. Labels you have not aliased still report exactly as before. - On the hosted service the table is bounded: at most 200 mappings, and a name of at most 500 runes (#366). Both are far above any real vocabulary — the schemes this feature exists for name a handful of labels — and the name limit is the same one applied to a sender's severity label when it is recorded, with every surface that shows a label truncating far shorter still. The name is measured as you wrote it, padding included, not as the trimmed form the matching uses. Targets need no limit of their own: they must be one of the three words above, spelled exactly.
These are limits on saving, not on loading, and they do not apply to this file: a self-hosted config is the operator's own input and is not size-checked at all. On the hosted service they are checked wherever an organization's configuration is written, so no save can leave behind a table its own editor cannot then edit.
The editor sends the whole table on every save, so that request is size-capped too. Since #370 the cap is sized to carry any table the store will accept, so a valid table is never too large to submit; a genuinely runaway paste still meets a legible limit instead of a generic bad request.
If a hosted organization is upgrading with a stored table already over one of the
two limits, nothing about its monitoring changes — collection, reports and every
page carry on. What stops is saving: any configuration change is refused until the
table is smaller. The editor still lists the whole table, and a Clear all button
(alongside each row's own delete) shrinks it; a table over the count limit now
posts and comes back naming that limit, so deleting rows until what remains will save
converges either way.
- A table that matches nothing validates cleanly. The names are checked for
shape, not against your fleet — p-1: where your rules write p1 passes
validation and does nothing. The startup log names the entry count and the
mapping it loaded — the first 50 entries, with aliases_omitted counting any
beyond that — but reading back what you loaded rules out a typo you can see,
not one you can't.
- What the table is measured against is on Settings (#358). An Unmapped
severity labels your sources send card lists, per source, the labels that
source's most recent collection cycle carried and no alias maps, each with how many alerts
carried it. An inert entry is then visible by contrast: the label you mapped is
absent from the list, and the label your fleet actually writes is still on it.
Three states are kept apart on purpose, because collapsing them is the failure
this card exists to prevent — a source reads as recognized only when its last
cycle actually assessed something (at least one alert carrying a severity:
label, all of them mapped), a source whose last cycle carried no alert with a
severity label gets its own neutral line covering both a quiet cycle and a
fleet that omits severity: entirely (neither concludes anything about your
vocabulary, and the second is read as warning by the very fallback this card
exposes), and a source with no reading at all to show is simply absent — it has
never scraped successfully, you removed it from config, or retention reaped its
last reading. A source that scrapes fine but has no alert firing is not absent;
it lands in the neutral line. Dropping a removed source is deliberate: a stale
reading must not pad the all-clear.
The counts are that cycle's reading, not a running total, and the same WARN line
and infrasigns_alerts_unrecognized_severity_total above remain the surface to
alert on.
Scraped sources only. An alert pushed to
the alert webhook is signalled under the source its labels
attribute to (#350), but
that signal is a counter and a log line — nothing persists a per-source reading
for the receiver, which is process-global rather than part of an organization's
runtime. So a pushed alert's labels still cannot appear on this card, and an
empty card is not evidence that a push-only fleet's vocabulary is understood.
- Editable in the hosted UI, per organization (#355). On the hosted service
the table lives on Settings: an Alert severity section
reports how many labels are mapped, and a Severity aliases card below it lists
every mapping — read-only for every member, edited a row at a time by an
owner (#363). The card appears once
the organization has saved a configuration at all — in practice, once it has
added a source, since the add-source wizard is the only step that creates one.
Only an owner can change the table; every other member reads it. The
severity for a new row starts unset rather than pre-picked: this table
escalates, so choosing a tier is always deliberate. A save takes effect on the
organization's next collection cycle, and it never rewrites history — rows
already recorded keep the tier they were recorded with. Self-hosted, the file
above stays the only way to change the table, and Settings reports the loaded
entry count there too, for the same reason the rest of that page exists: to see
what the process actually loaded without shelling into the container.
- Editing through the UI leaves the rest of your configuration untouched. A
save replaces the whole table and rewrites nothing else; clearing every row
removes the table rather than storing an empty one. The reverse holds as well
and always has: a severity_aliases: block an operator seeded by hand survives
every other UI edit, because each form merges into the stored document rather
than replacing it.
- Every path that reads an alert's severity label honors the table. There are
two: the collection cycle behind digests and incidents, and the
get_active_alerts MCP tool. On that tool the table now decides more than the
wording: a response returns 100 alerts per source by default, worst severity
first, so the table also decides which alerts survive the cap — an alias
that lifts a label to critical moves those alerts ahead of the cut, and one
that lowers it to info moves them behind it. (A client that passes a larger
limit sees past the cut; the ordering is the same, so the table still decides
what it reads first.) Trends read metric series and never see an alert,
and a health check derives its own severity from whether the probe passed, so
neither has a label to alias. On the hosted service only the collection cycle
applies, because MCP is not offered there — so an organization's own table covers
everything it can see. Pushed alerts are not an exception so much as outside the
question: the alert webhook is process-global, so an inbound alert is the
deployment's — recorded and delivered at the operator level and read with the
operator's table. That is the same reason incidents.retention_days is not
applied per organization; see the Settings page.
Escalation is the intended effect: mapping p1: critical means a p1 alert now
opens an incident summary and pages at critical urgency. That is a deliberate
inversion of the conservative default, and it is why the table is validated
strictly — you are declaring what your fleet already meant.
Database
PostgreSQL 13+ is required. The DSN role must own the schema and hold
CREATEROLE (migration 017 creates the infrasigns_app group role; managed
Postgres master users — RDS, Cloud SQL — have it, a hand-provisioned owner may
need ALTER ROLE ... CREATEROLE). The connection pool is capped at 10
connections (30 m max lifetime) — no tuning knobs; open an issue if a
deployment actually saturates it.
Row-level security (defense-in-depth). Tenant tables carry PostgreSQL RLS policies in addition to the application's own scoping; every statement runs in a transaction pinned to an organization. With the default single-role DSN this is transparent. Two operational notes:
- Backups:
pg_dumpof a database with forced RLS must run as a superuser or aBYPASSRLSrole — as a plain table-owner role it fails closed on the tenant tables. Do not work around a failing dump with--enable-row-security: it would silently dump zero tenant rows. - Least-privilege role (groundwork): migrations create a
NOLOGINgroup roleinfrasigns_appholding exactly the DML grants the server needs; a deployment can mint a login member of it (CREATE ROLE ... LOGIN PASSWORD '...' IN ROLE infrasigns_app;). Todayserveruns migrations at startup on its one DSN and migrations need the owner, sodatabase.dsnmust stay an owner-role DSN — the group role exists for the hosted multi-tenant deployment (where migrations and serving separate) and is what the integration tests run under. Self-hosted loses no protection:FORCE ROW LEVEL SECURITYsubjects even the table owner to the policies (only superusers bypass, e.g. the bundled compose's bootstrap user).
Sources
sources:
- name: production # display name in digests and logs
url: http://prometheus:9090 # Prometheus HTTP API base URL (bundled compose hostname)
Where a source lives
Everything in this section is about a deployment with
sign-in configured. Storing a source needs a database
document that belongs to somebody, and it needs a key to seal the credentials in
it — an encryption key (security.encryption_key or encryption_key_file) is
required once auth is set, and that is the only thing that requires one. So
without auth the start-up write below does not happen at all, this file is the
whole and only source list, and nothing of yours reaches Postgres. That is the
plain self-hosted install, and this release leaves it as it was — with one
exception, which is that sources[].provisioned and provisioned_from are
refused in a config file at every scope, because they are keys only the store
writes. Both are new, so no existing file carries one; a file that did would have
had it warn-ignored before and is refused now.
A source has two possible homes, and both are collected:
- Provisioned — declared in this file, under
sources:. This file stays authoritative: at every start the process WRITES these sources into the deployment's own organization in the database, marks them, and deletes the marked ones this file no longer declares, so removing an entry here removes the source. The web UI lists it, marks it From config file, and offers no edit or remove control for it.
A credential written as ${PROM_TOKEN} does not move with it. The file is read
a second time without environment expansion, so what the database holds is
the reference and the value is resolved at load — rotating the variable stays
live and the secret itself never reaches Postgres. The consequence is that only
the braced form is a reference in a stored document: a bare $NAME, a $$,
a ${my-var} and an unterminated ${ are each refused by name at startup, with
the form to write instead.
A credential written as a literal in this file does reach Postgres, sealed
with the deployment's encryption key, which
is why the start-up write happens only where sign-in — and therefore that key —
is configured. If you would rather it did not reach the database at all, put it
behind a ${VAR}; that is what the reference form is for.
- Stored — added through the web UI's add-source wizard, which writes it into
the deployment's own organization in the database. It can be edited and removed
from its page, and a change takes effect at the next scheduled cycle — no
restart. The wizard's second step is also where an operator with a configuration
file gets the matching config.yaml block, with Copy and Download, if they would
rather keep the source in this file; a hosted tenant, who has no file to merge it
into, no environment to export the ${…} references from and no restart, is not
offered it. DigitalOcean, Hetzner and CloudWatch sources are stored from the
first step wherever a source can be stored at all, so they never reach that step
and no block is generated for them.
Since #546 there is ONE
document: the startup write above puts this file's sources into the same stored
configuration the UI-added ones live in, so a source has one home and the runtime
reads one list. A name is an identity — it keys health records, incidents, reports
and the schedules — so two sources cannot share one: adding a source through the
UI under a name this file already declares is refused, and if this file LATER
gains a name a stored source already had, the file's definition replaces it at
the next start and the file owns it from then on, with a WARN naming the source.
A config file this deployment's own write gate refuses will not start, and provisioning is what turns rules that only ever bound on a document written through the UI into rules this file must pass. The CHANGELOG lists the full set; Pre-flight before an upgrade is the command that finds them in your file before you restart, and Rolling back is what a downgrade does and does not undo.
Only sources: is merged. Every other section a stored document can carry —
checks:, notify:, llm:, reports:, locale:, severity_aliases: — comes
from this file for this deployment, so the UI's editors for them are offered to
hosted tenants only, and the pages that read them read this file rather than the
document.
Adding a source through the UI needs sign-in. Without auth configured there
is no session to own the write and no owner to authorise it, so the UI stays
read-only — deliberately. A self-hosted operator who wants a writable UI
configures sign-in and then invites themselves into this
deployment's own organization, whose slug is default:
infrasigns org invite --org default --email [email protected] --role owner
What configuring sign-in changes, and what it no longer costs you. An OAuth
client id used to put the whole process into a hosted shape, and that cost a
self-hosted operator four surfaces in one flip. It does not any more. The manual
digest trigger (POST /api/digest/trigger), the /api/maintenance endpoints,
/mcp and the Telegram and Slack Q&A bots all mount whether or not sign-in is
configured. Each of them reads this deployment's own configuration, and every
database read they make is pinned to this deployment's own organization, so they
can only ever answer about you — what keeps them to you is the credential they
already had, api.token for the three endpoints and a per-transport allowlist of
chat ids for the bots, and those are process configuration that no signed-in
tenant can set. The CloudWatch tile in the add-source wizard is yours again for
the same reason: it now asks who is READING the page rather than what shape the
deployment is, so a member of this organization gets it, with role_arn optional
exactly as it is in this file, while a tenant of another organization still needs
a role to assume.
Three things do change when you set a client id, and it is worth knowing all three before you do.
- The plan and billing surfaces come on.
- The public site is opt-in, and off by default. Setting
ui.public_site: trueis what makes/serve the landing page to visitors who are not signed in, and what mounts the access-request form./pricingneeds the key too and a wired billing client, because its content is an offer and an offer a visitor has no way to accept is a commercial claim that applies to nobody — and the billing client is proprietary, compiled in only by theeebuild, so on the public image that page is absent whatever you configure. Leave the key unset and/keeps redirecting to this deployment's dashboard, which is almost certainly what you want on a deployment that is yours: the landing page describes a service you can be invited to. The key is refused without sign-in — with no session there is no anonymous visitor for it to be about — andservelogs one line at start-up naming it whenever sign-in is on and the key is not set, so the choice is announced rather than silent. api.tokenbecomes an operator credential in a room with other people in it. It was always the key to the digest trigger, the maintenance API and/mcp; what is new is that the room can now hold tenants. Anyone you give it to can run this organization's digest and silence its alerts. Treat it the way you treat membership ofdefault.
One residual worth naming, on a deployment that sets llm.max_fleet_calls_per_day:
questions you ask the Q&A bot spend the fleet-wide daily ceiling, the same one
every organization draws on. The bot was not running at all where that ceiling
exists, so this is new — if your own questions matter more than a tenant's
scheduled report, that is a budget to size deliberately rather than to discover.
One consequence runs the other way and is worth stating plainly, because it is a
capability rather than a loss: a member of this deployment's own organization
probes from the wizard without the SSRF egress gate. That is deliberate — it is
what lets an operator verify a Prometheus on a private address, which is the whole
point — but on a HOSTED deployment it means the account you invite into default
can make the service dial any address reachable from where it runs, and read back
whether each one refused, timed out or answered. Invite into default only the
people you would give a shell to. A tenant is unaffected: every organization but
this one dials through the gate, which refuses loopback, RFC 1918 and the cloud
metadata address.
A cross-reference is answered by the running process, not by one document.
checks[].source and a source's log_source may name a source that lives in the
other half of the merge — a check in this file may name a source you added through
the UI, and a stored source may name one this file provisions. Neither document
alone knows which sources exist, so neither refuses the reference: serve starts,
the picker on a source's page offers provisioned names alongside stored ones, and
the merged runtime is where the name is resolved.
What you get instead of a refusal is three signals. infrasigns config validate
prints each reference this file does not itself declare as a warning and still
exits 0 — it reads a file and never touches the database, so that it stays
runnable in CI, and it cannot know whether the name is provided elsewhere. The
process logs one ERROR per reference the merged source list cannot honour —
once for each distinct answer, and again whenever a config change moves it, not
on every runtime build — naming the field and the name, and for a log_source
also the source that declares it (the source[N] index would be a position in the
merge, which is not a document you can open); that covers a name nothing
declares AND a log_source naming a source that exists across the merge and
cannot fetch logs, which is a pair no single document can see. And a health check
whose source resolves in neither home records an error verdict reading
source "x" is not configured — no source of that name is declared for this
organization, which the checks card on the dashboard shows; that is a different
message from is not available (failed to initialize), which means the source IS
declared and its collector could not be built.
If you gate CI on config validate, read its output and not only its exit
code. A dangling checks[].source or log_source used to make it exit non-zero;
it now exits 0 and prints the warning block, so a pipeline step that checks the
status alone no longer fails on one. Grep the output for reference(s) this file
cannot answer if you want the old behaviour, and note that the answer such a step
would be enforcing is one this command cannot give: the name may be perfectly
resolvable at run time.
deploys.source is the exception and is still refused at load. Deploy
verification binds once at start-up to a source built out of this file and is not
rebuilt when the runtime's source set changes, so a source added through the UI
can never be its target — a dangling deploys.source is a real error rather than
a question for the runtime, and the message says so. Tracked as
#538.
Two sources of one name is no longer a state you can reach. Add db through
the UI, then later write db into this file, and at the next start the file's
definition replaces the stored one — one row, owned by the file, with the
file's URL and credential. What you added through the UI is gone rather than
shadowed, so this is worth noticing when it happens: the process says so at WARN,
once, on the start that does it —
provisioning: this deployment's config file now declares a source that was added through the UI; the file's definition replaces the stored one and the config file owns it from now on.
Removing db from the file after that deletes it; it does not bring the old
one back, because there is no old one to come back. Before
#546 the two rows both
existed and the stored one was merely hidden, which is the shadow
#539 was filed for and
which this release makes inexpressible.
${ENV_VAR} in a stored source depends on who stored it. The environment
belongs to the operator, so the answer differs by whose document it is.
- Your own organization's stored sources — a self-hosted operator adding a
source through the wizard — do expand it. The reference is what reaches the
database, and it is resolved against the process environment each time the
configuration is loaded, so rotating the variable takes effect without
re-entering anything and the credential itself is never stored at all. Only the
braced form is expanded; a bare
$NAMEis stored literally, so a password containing a$survives. - A tenant's stored sources do not, and a
${…}in one of their credential fields is refused when they save it, with a message saying why. A hosted organization has no environment of its own; expanding the reference would resolve it against the operator's, which is exactly the confused deputy the curated credential blocks exist to prevent. A tenant pastes the value, and credential encryption is what protects it at rest.
A literal $ in a config-file value
The config file is a different language from a stored document, and the
difference used to be silent. The file is expanded with Go's os.Expand before it
is parsed, which resolves ${NAME} and a bare $NAME — while the caveat both
commands print counts only the braced form. So a file whose references were all
written bare got a clean bill saying every reference resolved, and a value like
dsn: 'postgres://u:p@ss$word@db/x' was silently truncated at the $, under a
note certifying that nothing was unresolved.
Since this release the daemon warns at load about every shape that is not a
portable ${NAME}, naming the token, where it is and the remedy for it, and
infrasigns config validate refuses it — which is what the Helm chart's
pre-flight container and your CI run, so a bad shape stops a deployment without
stopping a daemon that was already running:
| what you wrote | why it is refused | write instead |
|---|---|---|
$NAME |
os.Expand resolves it and the ${...} caveat does not count it |
${NAME}, or $$ if you meant a literal dollar |
$1, $@, $?, $* |
resolved as a shell variable of the process that started the daemon | $$ |
${my-var} |
os.Expand resolves it, but the ${...} caveat counts only the portable shape, so the file would be certified as having no unresolved references |
rename the variable |
${} |
deleted outright by os.Expand, so the value loses those characters |
name a variable, or $$ |
${NAME |
an unterminated brace is deleted the same way | close the brace |
Each refused token is named with its line and key path, and the token itself
is abbreviated — line 3 (llm.api_key): $B… is not a reference this config file
may carry — … — so a file with several is a list you can work down rather than a
hunt for a $. The abbreviation is there because the commonest value this rule
fires on is a password: in postgres://u:p@ss$word@h/db the offending token is
half the secret, and the line and key path are what tell you which value is meant
without printing it. The same clauses go to the boot log with the same
abbreviation, and a provisioned sources: entry refused on the stored path is
abbreviated the same way. How short it is depends on the list: each token keeps
growing by one letter until it differs from the others printed beside it, so
$DB_HOST, $DB_NAME and $DB_PASS read $DB_H…, $DB_N… and $DB_P… rather
than three identical $D…. It never grows to the whole name, so two names
differing only in their last letter read alike — each still has its own clause and
its own line.
The list reads down the file, and where a file has more offenders than the list prints, the ones dropped are the ones furthest down it.
Three things that position does not promise. It is the token's first
occurrence: one token written on two lines is one clause, naming the first, which
is what keeps the refusal a stable list you can diff between runs. For a value
whose text does not lie on the line YAML reports for it, the clause says
line 8 or below rather than claiming a line it does not know. That is every |
and > block — their line is the HEADER's and the value starts below it, whatever
the value holds — and any plain, single-quoted or double-quoted value written over
several lines, including the three styles that fold each line break to a space and
so leave no trace of it in the value you get back. And
the key path is a name for the reader rather than a selector: it spells a source
source[0].url, matching the source validators' own messages rather than the
file's sources: key.
To write a literal dollar, double it. password: 'p@ss$$word' loads as
p@ss$word. Inside a URL or a DSN, %24 is the alternative and is what a
URL-encoding tool produces: postgres://u:p%40ss%24word@db/x.
One exception, and it is worth knowing before you reach for $$. If this
deployment provisions its file sources into the database — that is, if it
configures sign-in — the copy written to the database is the
file's verbatim text, and a stored document expands nothing. A $$ would
therefore be stored and sent with both dollars, so provisioning refuses it — and
the single $ it used to ask for is exactly what the file refuses. The practical
consequence is that on a provisioning deployment a $ followed by a letter cannot
be written into a source's credential at all: use %24 in a URL, or keep the
value out of the file and pass it through ${NAME}. That last is what the stored
gate's own message now names, so a reader following it is not sent back and
forth.
What the refusal does NOT cover, stated as a rule rather than as a list of cases: the guarantee is about the scalars of the parsed document, judged on the file's own source bytes — never about the whole file's bytes, and never about the value YAML hands back once it has unescaped it.
- A
$in a comment is still expanded — into a comment the YAML parser then discards. It is not a scalar, so it is refused by nothing and it is harmless; the shippedconfig/config.yamland this project's own dogfood config both carry one. - A
$that exists only after YAML unescaping —\x24inside a double-quoted scalar — is not a$os.Expandcan see, so it is not refused. The file contains no$byte there; the value does. - A file whose raw bytes do not parse cannot be walked, so it is not gated. It fails for its own reason instead.
- Only the first YAML document is read, as before;
os.Expandcovers the whole file.
A substituted value may not contain a line break
The rule above is about the $ shapes the file may write. This one is about
what a ${NAME} may resolve to, and it is a separate gate because a comment
really is harmless for the first rule and is not for this one.
${NAME} is substituted into the file's text before anything parses it, and
os.Expand does not know what a comment, a key or a value is. So a resolved value
carrying a line break ends whatever it was written into and starts a new line of
configuration at the top level — measured: a ${ROTATE_CMD} sitting in a
comment, resolving to a value whose first character is a newline, turned on a
digest schedule the file does not contain and config validate answered ✓ Config
valid, exit 0. Any multi-line secret does it; a PEM key is the obvious one.
Six characters count, not two: LF, CR, CRLF, U+0085 NEL, U+2028 LINE
SEPARATOR and U+2029 PARAGRAPH SEPARATOR — measured against the YAML parser
this product uses, all six of which split a line. (VT and FF are refused by
the parser itself with control characters are not allowed, so they are a loud
failure rather than a silent one.)
Nothing that worked before is refused: a literal newline in a substituted value
has never survived the YAML scalar parser, which folds it to a space — the
long-standing advice on this page is to URL-encode one as %0A, and that is still
the remedy.
The disposition differs by command, exactly as for the reference rules above.
serve WARNS and loads, so a deployment whose secret happens to carry a trailing
newline — kubectl create secret --from-file, a base64 without -w0 — keeps
booting. infrasigns config validate, the chart's pre-flight container and CI
REFUSE, and infrasigns config check and infrasigns migrate carry the verdict.
Before you upgrade, check your own file for a bare reference:
The sed is not decoration: it deletes the $$ escape this page prescribes
before the search, and without it the reader most likely to run this check — the
one who has already applied that remedy — sees their own fix reported back as a
problem. Line numbers survive it, because sed deletes within lines.
That is deliberately POSIX -E and not the shorter Perl form: grep -P is a GNU
extension, so on macOS or busybox it fails to stderr and prints no lines — which
reads exactly like a clean file, on the one command whose whole job is to tell you
your file is not clean.
The pattern reads bytes and cannot see YAML, so it still errs wide in two ways:
a hit inside a # comment is not refused, and neither is a $ with nothing
after it. Treat a hit as a place to look and infrasigns config validate as the
verdict. That pattern is a FLOOR, not a total: it finds the bare and
shell-special forms, which is what an existing working file is likely to carry,
and by construction it skips everything that opens ${, so a ${my-var}, a
${} or an unterminated ${ is not in its output. Those are named by the load
itself, one clause each.
That applies to the whole document, url: included, and since
#544 both places you can
write one agree about it. Every save of this deployment's own stored
configuration RESOLVES a ${…} before it is judged — that is what makes the
value judged the value the runtime reads — and so does the paragraph after this
one, about what a refusal may say. What is particular to the two surfaces named
next is that they also DIAL the resolved address in a live probe. The
add-source wizard and the source editor resolve a
${…} address for an owner of your own organization before they judge it, so the
live probe dials the address the collector will dial and the save is judged on the
same value the runtime reads — while what is written stays the reference, in the
generated snippet and in the database alike. Before that release the two surfaces
answered "the URL must be absolute and use http or https" for a value the save
itself accepts. Three things follow, and only the first is new behaviour:
- A reference in an address is resolved for an owner of your own organization and for nobody else. A member of it, and any hosted organization, is told to enter the address itself — the reference would have to be read from the environment of the process InfraSigns runs in, which is the operator's.
- A variable with no value is named in the refusal, because an unset one silently becomes an empty string and the address it leaves behind is not the problem to go and look at.
- A variable that resolves to something unusable is named too, and the messages these two surfaces compose for it never quote what it resolved to — they name the variable, and where the code has established that the resolved value is the problem they tell you to look at what the environment holds. That includes a probe that could not reach the address, which names the variable where it would otherwise have named the host, the IP it dialled or the name on the certificate. (The validator's own withheld verdict, below, is the case where the code has NOT established that, so it says where the refusal lies and leaves the looking to whoever can do it.) The reference, never the expansion, is also what the generated snippet and the stored document carry.
That covers the shared validator's own verdict too, by a different rule, and it covers it wherever a configuration is saved — the wizard, the source editor, and every other form that writes this deployment's stored document (#560). Some refusals are rendered word for word from the configuration validator, which quotes the value it was handed — an address, an OAuth2 token endpoint, a health check's schedule — and it is handed the resolved one, because the whole point is to judge the value the runtime will read. Two families of rule do this: the ones that judge the document as it loads, and the ones only a save runs. Both go through the same gate now, and both go through it one rule at a time, so a rule's refusal is shown only when it is identical to that rule's verdict on what you typed, and everything in it is something you can see in your own form.
Otherwise you get a sentence naming the variables the rule judged and nothing
else: "…is refused by a rule that judges what this deployment resolves for
${PROM_URL}, so the message is withheld". It does not say the variable is at
fault, because the rule that withheld it never asked that — the fault may be in
what your environment holds for it, or in a field judged alongside it, and both
are named. It does not tell you to go and check either, because that message is
anchored to a form control in a browser and the reader at it may have no way to
read this deployment's environment at all; what it ends with is what THAT reader
can do — ask whoever runs the deployment, or, for a document already stored, ask
them to run infrasigns config check. A ${PROM_URL} that resolves to a cleartext http:// host with a
credential set no longer comes back with that host in it; nor does a reference in
oauth2.token_url, nor a ${CRON} in a health check's schedule.
At a shell you still get the precise verdict. infrasigns config check
reads this deployment's own stored configuration and prints its verdict with the
resolved value in it, and the boot log prints the same thing when a start-up
refuses it — because a reader there already has the environment the value came
from and withholding it would leave them with nothing to act on. The asymmetry
is deliberate: the browser is the wider audience, and it is the one the default
is written for.
One redaction survives at the shell, and it is the same one the daemon's own log
applies: a credential written into a URL as user:pass@ is replaced. The field,
the scheme, the host, the port and the rule that refused are all printed, so the
diagnosis is intact; what is withheld is a password that would otherwise land in
a pod log wherever this command is run as a pre-upgrade hook or in CI.
Run it where the daemon runs. The rules expand ${VAR} before they judge,
and the expansion reads the environment of whoever runs the command — not the
daemon's. A systemd unit with EnvironmentFile=, or a container taking its
values from a Kubernetes Secret, gives that environment to the service and not
to your login shell, so a shell without it resolves every reference to the empty
string and answers for a different document: a refusal for one the daemon
accepts, or an accept where the daemon degrades. The command says which
references it could not resolve, on the accepted answer as well as the refused
one, so a verdict taken from the wrong shell is visible rather than silent. Run
it from a shell that has sourced the same environment file, or, with docker
compose,
or, in Kubernetes,
kubectl exec -n '<namespace>' 'deploy/<release>-infrasigns' -- /infrasigns config check --config /etc/infrasigns/config.yaml
These are the two lines the command prints for itself when it has references it
could not resolve. Both absolute paths are load-bearing, and so is running the
binary rather than a shell. Both placeholders are quoted, because each
carries < and >: an unquoted <namespace> or deploy/<release>-infrasigns
is a pair of shell REDIRECTIONS rather than a word, and pasted as-is the line
answers bash: namespace: No such file or directory and never reaches kubectl.
Quoted, it reaches kubectl, which names the placeholder you still have to
substitute. The image is distroless and has no shell, so there is
nothing to exec into — docker compose exec infrasigns sh is the obvious
reach and it fails the same way; the binary lives at /infrasigns, which is on
no PATH, so a bare infrasigns answers executable file not found in $PATH;
and --config defaults to a relative path while the image sets no working
directory, so the default resolves to /config/config.yaml and the command dies
on the file before it reads a variable. <release> is a placeholder: the chart
names the Deployment after its fullname, which is <release>-infrasigns
unless the release name already contains infrasigns, in which case it is the
release name alone — so helm install infrasigns … gives deploy/infrasigns
and helm install prod … gives deploy/prod-infrasigns. A
nameOverride/fullnameOverride changes it again; kubectl get deploy -n
'<namespace>' is the reliable read.
infrasigns config check needs the database, since the document it reads lives
there. It reads the config file only for database.dsn and the encryption key,
so a file the rest of which does not validate — an unset ${PROM_URL} in a
source, say — no longer stops it: you still get the stored document's verdict,
and the file's own refusal is reported beside it, naming the file's references
this shell could not resolve. It names them when the file is ACCEPTED too: a file
this shell validates can be a file the daemon refuses, for exactly the reason
above, so a ✓ computed without the daemon's variables says which ones were
missing rather than reading as a clean bill. Once the file has been read, its
verdict is printed on every answer this command can give, including the ones
that never open the database — a file it cannot load is a plausible reason it
found no database.dsn. infrasigns config validate says the same thing about the same
file, on both of its answers, since it is that file's judge and it expands
${...} against your shell exactly as this one does
(#566). Neither command
is a substitute for the other: config validate opens no database and therefore
never reaches a stored document at all. Run both.
What either command names is the references in the document, not every
${...} in the file's bytes — a variable written into a comment, or sitting in
one of the commented-out examples the shipped config.yaml is full of, is
nobody's reference and no environment will ever set it. A file that does not
PARSE has no document at all, so neither command says anything about its
references: the syntax error is the whole answer, and it is the same answer from
both. Every YAML document in the file is read, not only the first — a --- used
to end both the count and the $ rule, while os.Expand had always read the
whole file.
One shape parses only AFTER substitution: a ${VAR} written inside a flow
collection ({a: ${VAR}}, [${VAR}]) closes the collection, so the file's own
bytes do not parse while the expanded text does. Such a file LOADS, and nothing
walked the document you can edit — so the reference gate was not applied to it
at all, and both commands say so, and name that class, rather than certifying
it. A file of SEVERAL documents is judged on the ones that do parse: a ---
followed by a mistake of its own cannot turn the rules off for the document
above it. It is the REFERENCE gate specifically: the sibling rule that refuses a
substituted value carrying a line break reads the expanded text and still runs,
so a $ this file may not carry is what goes unrefused, and a value like
p@ss$word quietly loses the $word.
And within that document each reference is read from the value's own source
bytes, not from what YAML hands back after unescaping it
(#576). A double-quoted
"\x24{VAR}" decodes to ${VAR} while the file contains no $ byte at all, so
neither command names VAR: it is not a variable your file references, and no
export could ever clear it. This is the same reading the refusal above uses, so
the caveat and the verdict now agree about what a $ in your file is. It reads
the file's own $$ escape the same way os.Expand does, so $${VAR} names
nothing (it resolves to the literal text ${VAR}) while $$${VAR} — the escape
followed by a real reference — still names VAR.
Where it can still err wide, stated as a rule rather than as an example: a
double-quoted scalar whose own bytes this reading cannot locate in the file is
judged on its decoded value instead, which is the fallback scalarSource in
internal/config/fileenvrefs.go documents, and such a value can carry a $ the
decoder produced. That direction is the deliberate one — naming a variable your
own text mentions is the safe way to be wrong.
Who the withholding is FOR, since it is not "an owner who could read the
environment anyway": infrasigns org invite admits a second owner through a
browser, and that person may never have had access to the host — it is also the
only way back into an organization whose owners have lost access. Without the
rule, a message written to help them fix a URL can hand them the value of
${DB_PASSWORD}.
What that rule costs, stated plainly, because on a deployment that uses
references it is not a corner case. Each rule is judged on its own, so a rule
that reads none of your ${…} values says exactly what it would say if you had
typed the address out in full: with a stored source url: that is itself a
reference, a bogus Slack mode, an over-long recipient list, an unknown severity
alias or a malformed schedule all still come back word for word. What a rule
withholds is bounded by what that rule reads, and since
#562 one rule reads
widely:
- The structural pass that applies this file's field rules reads the whole
document and reports everything it found at once, so a reference in a field it
constrains —
llm.provideris the one that bites — moves that batch's verdict. It is also the pass that echoes what it judged, inllm.provider must be one of: …, got …; in a browser that value is withheld like any other. The names it lists are narrowed — a reference in a field no clause of the pass judges is not named — but the pass itself is not decomposed, so its refusal is still about the whole rule rather than about one field, and it fires first. -
The one that checks source names used to be the other, and is not any more. It is one rule per source now, so a reference in one source's address no longer withholds a malformed name on a different source: that refusal comes back word for word.
Within that batch the withholding is per MESSAGE. Each message the pass produces is compared on its own, so the ones that read identically either way are printed and only the ones that moved are replaced:
llm.provider: ${LLM_PROVIDER}beside a missingllm.api_keynow answers "llm.api_key is required; llm.model is required" and then says, in one further clause, that something else was refused by a rule it cannot quote. A message moves exactly when it QUOTES the value it judged, so what is still withheld here is the enum message itself; a plain "… is required" reads the same either way and is printed.
A withheld message no longer silences the rest of the document, and it is no
longer dropped either. The rules are reported one at a time and the first
failure is what you see; before this, a rule whose verdict could not be shown
ended that list, so a reference in llm.provider hid a bogus Slack webhook
further down it. Now a rule that cannot speak is skipped over, the first one that
CAN is what you get, and the withheld one rides beside it as one further clause —
in a browser, "…; this submitted configuration is additionally refused by a rule
that judges what this deployment resolves for ${LLM_PROVIDER}…". At a shell
infrasigns config check prints both verdicts, one line each, in the order the
rules ran. That matters because before it the two arrived one at a time: you
fixed the Slack mode, restarted, and only then met the first — a second round
trip on the one path where withholding buys nothing, since that reader already
holds the environment the value came from. It holds in both directions and
across both families of rule: whichever of the two refusals is the withheld one,
the other does not swallow it. When NEITHER can speak, infrasigns config check
prints every withheld rule rather than the first one alone — the browser's
sentence is unchanged there, since it names the variables and not the rules.
What is genuinely lost in a browser is the withheld message itself, and there is
no log level that brings it back. infrasigns config check at a shell prints it,
resolved value included — and a refusal of the STORED document names that
command, so a reader with no shell has something to ask for rather than a dead
end.
Which of the two you get follows what was REFUSED, not which page you were on
(#565). A refusal of a
submitted document — a save from any of these forms — reads "this submitted
configuration is refused …" and ends "Ask whoever runs this deployment what
those values are": nothing has stored what you sent, so no command can read it
back, and on a self-hosted install with no auth: section this deployment stores
nothing at all. A refusal of the stored document reads "this stored
configuration is refused …" and names infrasigns config check, which is the
command that reads exactly that. The add-source wizard and the healthcheck
endpoint list end with the same "ask whoever runs this deployment" clause but
name a NARROWER subject, because what they judge is one field rather than the
whole document: "this source is …" and "these endpoints are …".
Before this the save path took the stored wording and the stored command: you
were told your saved configuration was broken and sent to a command that, run on
the same deployment in the same minute, answered "✓ … accepted by the write
gate" — because it had read the other document.
One further shape pays for it with a less specific message. A reference whose own
text is shorter than the sixteen characters a bearer token must have —
${TOK} — is refused as too short as typed, while the resolved value gets past
that check and is refused for something else; the two verdicts differ, so the
generic sentence is what you see. Note that what a longer variable name
(${PROMETHEUS_BEARER_TOKEN}) restores is the MESSAGE, not the source: whatever
the resolved value was refused for is still there to fix.
An unset variable expands to the empty string, as everywhere else in this file.
For the credentials that are required — a basic-auth password, an
authorization.credentials, an OAuth2 client secret — the save is then refused;
for an optional one such as a bearer token the source simply stores no
credential, so check the variable is exported before blaming the source. In a
browser that refusal names the variable rather than the field, because an unset
one is exactly the case where the two readings of your document disagree; run
infrasigns config check if you want the field named — it judges what this
deployment has STORED, so it answers for a save that landed, not for one the form
has just refused.
Most of the surfaces that read a deployment-wide source list now see a save
without a restart. The vocabulary /mute <source> is validated against, the
source data behind the MCP and Q&A-bot tools, and the set a health check uses to
tell "no source of that name" from "its collector could not be built" are all read
live: a committed save rebuilds this deployment's runtime in the background, so
the new name is usually accepted within seconds. The tool catalog is no longer
derived from that list either (#551):
get_active_alerts, query_metrics, query_prometheus and query_loki are
always offered, and a deployment that has nothing to point one of them at is told
so when it is called, with the remedy — rather than being served a tool list the
add-source wizard cannot change.
There is one state in which those live reads stop moving: if this deployment's
stored configuration exists and can no longer be read — a rule tightened by an
upgrade is the ordinary cause — they keep the source set they last had rather than
narrowing to the config file's
(#550). A save cannot reach
them while that lasts, nor can it commit, since it merges into the same refused
document — so they are stale, never smaller, until that document is repaired. The
on-demand digest surfaces (POST /api/digest/trigger and the trigger_digest
tool) are held back with them, so the whole tool catalog answers about one source
set. The scheduled engines meanwhile run the config file's sources, which is what
keeps a rule tightened by an upgrade from taking your monitoring down, and the boot
ERROR that reports the refusal names all of this — including that the Sources
page is not where you repair it, since that page asks the same question of the same
document and falls back with it.
Two are still derived once, at startup, and a source you add through the UI
reaches them only at the next restart: the deploys.source binding, and the
inbound-alert receiver's alert_labels attribution.
Since #546 every one of them sees the whole list — the file's sources and the stored ones together — because provisioning writes the file's into the same document.
Three surfaces deliberately still see the config file's own sources alone.
/readyz's per-source checks: that used to be because a single unhealthy
source answered 503 for the whole process, so a source added through the UI
must not be able to take the pod — and with it the UI that would remove that
source — off the network. Since
#507 no source gates the
pod at all (see Readiness), so that reason
is gone; what decides it now is narrower and still decides it — these are the
OPERATOR's own declared sources, and a process-wide probe body is not where a
tenant-added source belongs. Widening it is a candidate, not a defect. A stored
source's own failure is reported on its card on the sources page instead.
Live charts on a source's page, because the endpoint behind them queries the
process's own collectors and dials with the credential map built at startup. And
the separate infrasigns mcp process, which loads this file itself and opens
no database.
The keys a source takes:
- Source names may use only letters, digits, dots, underscores, and dashes
(
^[A-Za-z0-9._-]+$) and are at most 60 characters — they double as metric label values, deep-link IDs, the source's web detail-page URL, and the source token in the/incidents <source>commands the bot prints for you to copy, which shortens anything longer and so would print a command that matches nothing. The bound is exactly that chat display cap, so it rejects only names the printed command would already have cut.new,gridandrecentare reserved: the first two collide with the/sources/{new,grid}UI routes, andrecentis the newest-first switch in the bot's/incidents [<source>] [recent]grammar, so a source with that name could never be filtered from chat. The charset rule (but not the length bound or the reserved list) applies to check names too. Renaming a source leaves three things behind, all keyed on the OLD name literally. They are worth doing in one sitting: - A live maintenance window. Its scope is matched literally and is checked
against your sources only when the window is opened, so a window on the old
name silences nothing and that source starts paging again mid-silence.
Re-open it under the new name (
/mute <new-name> …, or the maintenance-window API). - Open incident episodes. An incident records the source it was attributed
to when it arrived, and nothing rewrites that on a rename — so an episode
that is still firing disappears from
/incidents <new-name>and from the source's timeline on its detail page, while/incidents <old-name>still finds it. Either close those episodes before renaming, or keep the old name to hand until they resolve. - Dashboards and alert rules. The name is a metric label value, so every
query pinned to
source="<old-name>"goes flat rather than erroring. Update them alongside the rename. - Every URL key must name a host, checked on the host NAME rather than on the
authority.
https://:8080— a port and nothing else — parses, looks well-formed and used to load; but a hostless authority resolves to localhost, so such a value silently points InfraSigns at its own machine.config validaterejects it for a sourceurl, a healthcheckendpoints[].url,heartbeat.url,bot.mcp_servers[].url,auth.base_urlandauth.github.enterprise_urlalike; the add-source wizard,infrasigns digest trigger --addrand the Slack webhook channel refuse it where they meet it. - A wildcard bind address is refused on the same keys.
0.0.0.0and[::](and the spellings that normalize to them, such as[::ffff:0.0.0.0]and a zoned[::%25eth0]) are what a listener binds to; as a URL they reach the local machine exactly as a hostless authority does, so a value copied off alisten/bindline silently probes, scrapes or pings InfraSigns' own host. Every key in the list above rejects them, with a message naming the mistake rather than "missing host". Two deliberate exceptions: only the unspecified address itself is refused —0.0.0.1is an ordinary routed address and is accepted — andinfrasigns digest trigger --addrstill takes a wildcard, because that flag addresses your own daemon on purpose. typeselects the collector:prometheus(the default when omitted),cloudwatch(see AWS CloudWatch below),digitalocean(see DigitalOcean below),hetzner(see Hetzner Cloud below),healthcheck(see Healthcheck URLs below), orloki(see Loki (LogQL) below). A Prometheus source needsurl(and takes an optional bearertoken, see Authenticated Prometheus-compatible backends below); a CloudWatch source needsregionand nourl; a DigitalOcean or Hetzner source needs atokenand one or moreresources; a healthcheck source needs one or moreendpoints; a loki source needs aurland one or more metricqueries(with optionallog_queriesfor log context, and an optional bearertoken).- Multiple sources are supported — each produces an independent digest, and Prometheus, CloudWatch, DigitalOcean, Hetzner, healthcheck and loki sources can be mixed freely
- At least one source is required, with one exception: a config file that
configures sign-in (Sign-in (hosted)) may leave
sourcesout entirely. That process monitors nothing of its own — every source it collects belongs to an organization and comes from that organization's stored configuration — so a minimum here only ever forced operators to invent a dummy entry. It still holds everywhere else: a self-hosted config file needs one, and so does an organization's stored configuration — which is what the sources page's refusal to remove your last source backstops (#477). Watching the hosted instance itself is a good idea, but do it as its own deployment or as ahealthchecksource on a self-hosted instance — not as a source the validator extracted from you - The
severity:label on your alerts decides how a collection cycle is reported: a firingcriticalalert makes it an incident summary, delivered at critical severity, instead of the regular digest. On a Prometheus source the label is whatever your alert rules wrote, so it is read case-insensitively and whitespace-trimmed —CriticalandCRITICALcount. An unrecognized scheme (p1,sev1,page) and an alert carrying noseverity:label at all are read as a warning: they ride the next digest and never turn a cycle into an incident summary. That is the default, not the only outcome: it is deliberate — escalating every label InfraSigns does not recognize would turn a typo into an incident — so a fleet on a non-standard scheme gets digests only until it declares the scheme withseverity_aliases, which rewrites such a label as it arrives and lets it open an incident like any other critical. Everything below describes the labels no alias covers. On a collection cycle — scheduled, triggered manually through the digest API, or requested through the MCP tool — you do not have to read this page to find out: a cycle carrying an unrecognized label logs a WARN line containing the phraseunrecognized severity labels rank as warning— each path then spells out its own consequence, so grep the phrase, not a whole line — with thesourceand the labels themselves, and incrementsinfrasigns_alerts_unrecognized_severity_total{path="scheduler",source}on every such cycle — the un-throttled surface to alert or graph on. The log line is throttled to at most once per source per 24h, so a label that first appears inside that window waits it out; each line names up to five labels plus the count of the ones it cut, so the next warn describes the remaining vocabulary rather than only what changed — with the residual that a label past the cap is counted but never named, so a source with more than five unrecognized spellings shows the five carried by the most alerts and a count (#358 — before it, the five that sorted first alphabetically). The throttle is in memory on the running scheduler, so it is not a promise about a source for all time: a process restart re-warns on the next cycle, and in cloud a rebuilt org runtime — a config save, a plan change, or an org idle long enough to age out of the runtime cache — does the same. What it bounds is the repetition that made the line unreadable: a handful of lines a day for a misconfigured source instead of one on every cycle. Two labels that rank as warning are deliberately not reported, because neither is something to fix:warn, which InfraSigns already understands as a warning, and noseverity:label at all, which is just an alert rule that never set one. A healthcheck source's down alerts take the same path, but theirseverity:is configuration rather than a free-form label and is validated case-exactly at startup (see Healthcheck URLs).The inbound alert webhook raises the same signal, with a worse consequence and a coarser key (#345). Everything above is about collection cycles. A pushed alert's
severity:goes through the very same reading and falls back to warning the same way, and the webhook receiver now logs a WARN line prefixedwebhook:and carrying the sameunrecognized severity labels rank as warningphrase, and incrementsinfrasigns_alerts_unrecognized_severity_total{path="webhook",source}on every such request. Two things differ. What goes wrong is worse: a pushed alert is not merely capped at digests, it is delivered — every channel routed to theincidentsfeed (by default, all of them — see feed routing) renders it as a warning and PagerDuty pages at warning urgency instead of critical, since onlycriticalmaps to a PagerDuty critical and this is the only path that carries a sender-supplied severity there (a health-check transition also pages, but at the tier its ownseverity:sets — your config, not a label a sender can distort). Andsourceis resolved differently there: it is the configured source name the alerts' labels match (Incident attribution), not the collector's own name, since the receiver is pushed to rather than scraping (#350). Matching only: the single-source shortcut that section describes does not apply here — naming a fleet no label identified would be a guess — so an install that configures noalert_labelskeeps reportingalertmanager, exactly as before. Matching is per alert, so one request carrying two fleets' alerts warns once for each and increments the counter once for each — the 24h throttle is per fleet, as it is on the collection path. An alert nothing matches falls back to the detected payload format, in practicealertmanager, since a CloudWatch payload carries noseverity:label for the receiver to fail to recognize. Only a process restart clears the receiver's throttle either way. The remedy is the same one the collection path uses:severity_aliasesapplies to pushed alerts too, and it is the only way to make this path page at critical urgency for a label InfraSigns does not know on its own. log_source(optional) — the name of another configured source whose logs ground an incident on this one. See Grounding an incident in another source's logs below.- InfraSigns is read-only — a Prometheus source calls only
GET /api/v1/alertsandGET /api/v1/query; a CloudWatch source calls onlyListMetrics/GetMetricData; a DigitalOcean source calls only the read-only Monitoring and resource-listing endpoints; a Hetzner source calls only the read-only Metrics and resource-listing endpoints; a healthcheck source only issuesGET/HEADrequests to the URLs you configure; a loki source calls onlyGET /loki/api/v1/query_range - No changes to your Prometheus configuration are required
Pre-flight before an upgrade
An upgrade is one of two ways to meet these rules. They bind whenever this
deployment configures sign-in, so turning auth: on — unchanged file, unchanged
binary — meets them for the first time too, and the daemon then refuses to start
in exactly the same way. Read this section before either move.
Run it with the binary that carries the rules, which for an upgrade means the
version you are upgrading TO. That is the load-bearing half and the easiest to get
wrong: these rules ship in the new binary, so the version you are running today
reports nothing about them, and a version older than
#546 has no
--strict-provisioning flag at all and answers unknown flag. (If you are only
turning sign-in on, the binary you are already running is that binary — add the
auth: block to the file first, then run the command against it, because the
command decides whether to check at all from that block.) Run it against your
current file, with the same environment the daemon has — the rules expand ${VAR}
before they judge, so a variable unset in your shell is judged as empty and
answers for a different document. Docker has the
docker run form; for a bare binary, download the new one and run:
It names every offending source and prints the write gate's refusal verbatim —
which names the offending field, and the block to move the value into, only
where the rule it broke has one to name. There is a second answer it can
give, and it names no source: if this file's sources: section cannot be read as
it is written at all — a sources: ${SOURCES} supplied whole from the
environment, or a section the parser refuses — then nothing was judged, and the
command says so rather than reporting a count of sources it never saw.
It exits 1 on either answer, with or without
--strict-provisioning — that flag is accepted and implied, and is still worth
passing only because a binary older than this one needs it. Both answers are
decidable from the file alone, which is why they move the exit code where
the cross-reference warnings deliberately do not: one of
those may legitimately be answered by a source that lives in the store, and this
one cannot. The refusal is still scoped to these answers alone and does not turn
those warnings into failures.
The command is WIDER than the daemon here, deliberately and in the safe
direction: serve refuses only when the file's sources differ from what is
already stored, which needs a database this command does not open, so a
stored-and-unchanged offender fails here and boots there. What it does NOT cover:
it reads whether sign-in is configured from the EXPANDED file, so a
auth.github.client_id: ${GITHUB_CLIENT_ID} whose variable your shell does not
hold is judged as a deployment that provisions nothing, and these checks are
skipped. The run now says so, in a block of its own, rather than passing over
it in silence — it reads the file's own bytes to see that sign-in is declared, and
tells you what was skipped and that the daemon will run it.
What is skipped is more than this section's subject, and the block names all of
it: the whole auth: block (ui.enabled: true, each declared provider's client
secret, auth.base_url), the security.encryption_key a deployment holding other
organizations' credentials must carry, and this file's sources: against the
write gate. On such a file with the variable set, the refusal you get is commonly
auth requires ui.enabled: true — not a source at all — so a block naming only
the provisioning pass would send you to the wrong section. It cannot tell you what
any of them would have found: that needs the variables your shell does not have.
One more shape draws the same block: a file whose own bytes are not YAML while the
expanded text is, because a ${VAR}'s braces are being read as YAML syntax. That
happens in two places, and the run names whichever applies — inside a flow
collection ({a: ${VAR}}, [${VAR}]), where the reference's } closes the
collection, and straight after an &anchor or a !tag, where the token ends at
the {. (An alias, b: *${VAR}, is not one of them.) The read that looks for
auth: parses the file as written, so where it could decode no document at all it
cannot tell whether sign-in is declared either way, and it says that rather than
answering as though nothing were declared. Where the break is in a LATER document
of a --- stream it could read the first one — which is the only document this
product loads — so it answers from that and says nothing. Running the command with
the daemon's environment — which the paragraph above already asks of you for a
different reason — is what turns the caveat into an answer.
That caveat has an exit code of its own: 3. A pre-flight reads $?, so a run
that could not apply every rule must not be indistinguishable from one that did,
and it must not be indistinguishable from a refusal either. There are more
verdict lines than codes, because exit 1 is deliberately the same for a document
a rule refused and for a file whose bytes were never obtained — the table gives
each code a row, and the row under 1 carrying two lines is why:
| Exit | Verdict line | What it means |
|---|---|---|
| 0 | ✓ Config valid |
every rule this command carries was applied, and none refused |
| 3 | ! Config valid under the rules this run could apply |
nothing refused, and a set of rules was SKIPPED — this shell could not resolve the sign-in client id, or the file's own bytes are not YAML, so the document was not walked in full and the $ reference gate did not cover all of it |
| 1 | ✗ Config invalid / ✗ This deployment cannot store this config file's sources |
a rule refused the file — it was read, and rules were applied to it |
| 1 | ✗ Could not read the config file |
no rule ran at all: the process never held the file's bytes |
The two verdicts under 1 are not interchangeable, and the line is the only
thing that separates them — the code is the same on purpose, because a
pre-flight must stop on either. ✗ Could not read the config file means the
open or the read failed and nothing was judged: a mode the process cannot read
(a 600 config on a docker bind mount — see
The container runs as UID 65532),
a path that does not exist, a path that is a directory. There is no document
there for a rule to have refused, so a syntax error is not what you are looking
for (#594). The other two
are statements ABOUT a document the command read.
If 3 is not a failure for your gate — a CI job that lints the config file has no
reason to hold a sign-in secret, so on such a job the third verdict is the normal
answer — pass --accept-unjudged and it exits 0 instead. The block still prints;
only the code moves. The Helm chart's pre-flight container CAN pass it, behind a
values key of its own (preflight.acceptUnjudged) that is off by default, for
the reason Upgrade gives: the flag carries an image floor
that --strict-provisioning beside it does not, so an install pinned to an older
image.tag must not meet it by accident.
infrasigns config check carries the same third verdict, the same code and the
same flag, over a NARROWER union — and the difference is worth knowing if you
gate CI on it. That command's subject is the STORED document, but its exit code
already answers for the config FILE as well (a file it refuses is exit 1), and it
reads the file through the same $ reference gate. So it exits 3 on the second arm
above — the file's own bytes are not YAML, so the gate covered only the documents
that parsed — and never on the first, because it does not apply the sign-in rules
at all and so has nothing there to skip. Before this it exited 0 on exactly the
file config validate reports 3 for, while printing the same caveat above the
zero, and answered unknown flag to the --accept-unjudged this page tells you to
reach for.
It exits 0 with nothing to say where sign-in is not configured AT ALL, because there is no start-up write to refuse anything. That silence is a fact about your deployment, not a clean bill for a deployment that does provision — and it is distinguishable from the third verdict above, which is the one that means "this run could not tell". A file that declares sign-in with an EMPTY client id is exit 0 too, and says so in its own words: the daemon reads the same empty value, so nothing was skipped that it will apply, and running the command elsewhere answers the same.
The Helm chart carries it as a preflight init
container on the migration Job, so where it runs a refusal fails the pre-upgrade
hook while the previous Deployment is still running rather than crashlooping the
new pod. The container renders on every install — it has a switch of its own,
preflight.enabled, defaulting on and independent of migration.enabled; what is
conditional is the FLAGS, not the check. (An earlier version of this paragraph said
the container "renders only where the chart renders an auth: block", which was
never true of the container and is what templates/migration-job.yaml and
Upgrade both say.) What the chart cannot reach is the
provisioning refusals specifically: it renders no auth: section, so a config it
produces declares no sign-in and that class has no subject. A compose or
bare-binary deployment has no hook at all: run the command yourself before you
restart, and see Docker.
If you cannot run the new version first
There is one thing you can check with nothing but the file in front of you, and
it is the class that carries a credential: a user:pass@ inside a url:,
an endpoints[].url: or an oauth2.token_url:.
A hit is a URL whose authority carries userinfo, which is the shape all three refusals are about. Two things to know before you act on it:
- It over-matches, on purpose. It keys on the URL shape and not on the YAML
key, so it also reports your
database.dsnand any other credential-bearing URL in the file. Read the hits; false positives cost you a glance, and the alternative — anchoring on the key — would miss a spelling. - It is not the whole check. It answers for the userinfo class only. Every
other refusal on the breaking list — a credential under four characters, an
over-long
unit:, atags/labelsselector, a NUL, a bare$NAMEwritten without braces — is invisible to it, and so is anything that arrives through${VAR}: the grep reads the file as written, so a variable is just a variable to it. (For the three userinfo fields specifically that is not a gap — the gate reads the raw document too, so a reference stores no credential and is accepted by both. For the rules that run on the expanded document it is a real one.)
For the sources already in the database — the ones added through the UI,
which no config file mentions and no config validate opens a connection to see
— What is encrypted, and what is not has
the SQL. Run it as a superuser or a BYPASSRLS role, for the reason stated
there.
Rolling back to a version before this one
There is no schema migration to undo and no ladder step to walk back: the stored
document's shape is unchanged, so an older binary reads it. It logs
config: ignoring unknown or unsupported field for provisioned and
provisioned_from on every read of that document — noise, not damage — and it
merges this file's sources over the stored ones by name exactly as it used to, so
the fleet it collects is the same one.
One of those lines is not noise, and nothing else reports it. A binary from
before #545 does not know
endpoints[].basic_auth or endpoints[].authorization, so it warn-and-ignores
them exactly like the two keys above — and then probes every authenticated
endpoint with no credential at all. The endpoint answers 401, the probe
reads that as down, and you get a down-alert at whatever severity that endpoint
carries, for a service that is up. The stored credential survives the rollback
untouched and comes back when you roll forward — unless you save that source
while you are rolled back, which is the one thing that makes the loss
permanent. The older endpoints editor replaces the list wholesale from a form
that has no credential fields at all, so the save writes each endpoint without
one and the encrypted value is gone; there is nothing to roll forward to. The
paragraph on Edit and Remove further down describes the same save costing you
something else at the same time. While rolled back, treat a healthcheck source's
endpoints editor as destructive.
Once you have rolled back there is no detector for this, and we would rather
say so than imply coverage: the code that would have to notice is the code you
have just rolled back to. What you get is the warn line, which names basic_auth
or authorization without saying which endpoint. Before you roll back there is
one — the version you are still running is the version that knows which endpoints
carry a credential, in endpoints[].basic_auth / endpoints[].authorization in
this file and in the endpoints editor on each source's page for the ones added
through the UI. Write that list down first. Then, if you are rolling back a
deployment that authenticates a health probe, expect those alerts, and either
mute the source for the duration
(maintenance windows) or drop the endpoint from the
config file until you roll forward. The same applies on a mixed-version fleet
— a replica on the older binary probes unauthenticated while its neighbour does
not.
What you lose while rolled back is the direction this release added. The stored copies survive the downgrade, and an older process does not delete them, so removing a source from this file no longer removes the source — the stored copy simply reappears as though it had been added through the UI, which is the behaviour this release exists to end. Rolling forward again prunes it, because the marker that says the file owns it is still on the row.
The one thing that does not heal itself: while rolled back, that reappeared source has a working Edit and Remove on its page, because the older version decides ownership from a snapshot of this file's names taken at boot and the file no longer names it. Saving from that page rewrites the entry from the form and drops the marker with it, so rolling forward can no longer tell it from a source somebody added through the UI, and it will never be pruned — it keeps running as a stored source. If it is a healthcheck source, that same save also drops every endpoint credential, for the reason the paragraph above gives. If you did not want the source, use Remove on that page rather than Edit.
If the deployment also has credential encryption on, read Turning it off, and rolling back as well: a binary from before encryption existed has its own trap, and it is the more expensive one.
Authenticated Prometheus-compatible backends (token)
Many OpenTelemetry-native and hosted backends expose an authenticated
Prometheus-compatible query API — Grafana Cloud, Dash0,
Grafana Mimir, VictoriaMetrics Cloud, and Thanos behind an auth proxy. Point a
Prometheus (or Loki) source at their query endpoint and add an optional bearer
token:
sources:
- name: grafana-cloud
url: https://prometheus-prod-01.grafana.net/api/prom # the query API base
token: '${GRAFANA_TOKEN}' # Authorization: Bearer
queries:
- name: http_error_rate
query: 'sum(rate(http_requests_total{status=~"5.."}[5m]))'
signal: errors
tokenis sent asAuthorization: Bearer <token>on every query. Leave it unset for an unauthenticated Prometheus — the field is optional. When set it must be at least 16 characters, and theurlmust behttps(a bearer is refused over cleartexthttpto a non-loopback host, so it can't be sniffed on the wire — a loopback host is exempt for local dev:localhost(alsoLOCALHOSTandlocalhost., the same name), or an IP literal that parses as loopback, including an IPv6 zone such as[::1%25eth0]). Short address literals like127.1,0177.0.0.1and2130706433are not loopback here — Go's parsers do not implement that syntax, so the value goes to DNS as a name, and a resolver search domain could expand127.1to127.1.corp.example.comand carry the bearer token to a remote host in cleartext. The same goes for a Unicode look-alike of the name (localhoſt).- The token is a secret: inject it via an environment variable
(
token: '${GRAFANA_TOKEN}') exactly like every other credential (see Kubernetes). Self-hosted YAML and the cloud add-source wizard both collect the token (#251); the wizard's live probe carries it (before #480 a token made the wizard skip the probe, because an unauthenticated probe answers 401 for a source that is perfectly healthy), and the source-detail page lets an owner rotate it. A cloud organization stores the token in its config; self-hosted copy-paste emits a commented${PROM_TOKEN}placeholder to uncomment and wire from the environment. Never put a bearer in theurlquery string — a?token=…(or a bare trailing?) is rejected at config load, because a query parameter is a credential carrier that can't be reliably redacted from error logs. Put it intoken: the bearer rides anAuthorizationheader, so it never appears in the source URL, in metric labels, in the web UI, or in sanitized error logs. - The header is scoped to the configured host: if the endpoint issues an HTTP redirect to a different host, the token is not forwarded (it stays on your backend, never a redirect target).
- Digests, incidents, trends, and the source-detail page's live-charts widget all read the authenticated source normally — the charts dial sends the same bearer header (#252). (In cloud the live-charts endpoint is not yet served per organization, so those charts show no data there regardless of authentication.)
- This is the deliberate shape of OpenTelemetry support — InfraSigns is a view + analysis layer over your existing backend, not a TSDB. Apps instrumented with OTel reach InfraSigns through a backend that ingests OTLP (Prometheus ≥3.x native OTLP endpoint, Dash0, Grafana Cloud, …). See OpenTelemetry for end-to-end recipes and metric-name normalization notes.
Other authentication schemes
token covers a bearer. A prometheus or loki source can also authenticate
with basic_auth, authorization, oauth2, custom headers, tls or sigv4
(#480):
sources:
# Grafana Cloud, Last9, a Prometheus behind nginx-ingress or Traefik.
- name: grafana-cloud
url: https://prometheus-prod-01.grafana.net/api/prom
basic_auth: { username: '123456', password: '${GRAFANA_TOKEN}' }
# A raw Authorization header, for a vendor whose docs print a scheme word other
# than `Bearer` — `ApiKey`, `Token`, `DSN`. Leave `type` out and it defaults to
# `Bearer`, which is what the `token:` shorthand writes.
- name: gateway-apikey
url: https://metrics.example/prometheus
authorization: { type: ApiKey, credentials: '${METRICS_TOKEN}' }
# Azure Monitor managed Prometheus (the v1.0 `resource=` form).
- name: azure-monitor
url: https://<workspace>.prometheus.monitor.azure.com
oauth2:
client_id: '${AZURE_CLIENT_ID}'
client_secret: '${AZURE_CLIENT_SECRET}'
token_url: https://login.microsoftonline.com/<tenant>/oauth2/token
endpoint_params: { resource: https://prometheus.monitor.azure.com }
# Google Managed Service for Prometheus (RFC 7523 service-account flow).
- name: gmp
url: https://monitoring.googleapis.com/v1/projects/<p>/location/global/prometheus
oauth2:
client_id: '${GCP_SA_EMAIL}'
token_url: https://oauth2.googleapis.com/token
grant_type: urn:ietf:params:oauth:grant-type:jwt-bearer
signature_algorithm: RS256
client_certificate_key: '${GCP_SA_PRIVATE_KEY_PEM}'
scopes: [https://www.googleapis.com/auth/monitoring.read]
# An origin behind Cloudflare Access.
- name: behind-cf-access
url: https://prom.example.com
headers:
CF-Access-Client-Id: ['${CF_ACCESS_CLIENT_ID}']
CF-Access-Client-Secret: ['${CF_ACCESS_CLIENT_SECRET}']
# An internal CA and a client certificate (service mesh).
- name: internal
url: https://prom.internal:9090
tls:
ca: '${INTERNAL_CA_PEM}'
cert: '${CLIENT_CERT_PEM}'
key: '${CLIENT_KEY_PEM}'
server_name: prom.internal
min_version: TLS12
# AWS Managed Service for Prometheus — SigV4 only, there is no token option.
- name: amp
url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/ws-xxxx
sigv4:
region: eu-west-1
role_arn: arn:aws:iam::123456789012:role/infrasigns-amp-read
- At most one of
token/basic_auth/authorization/oauth2/sigv4may be set on a source;headersandtlscompose with any of them.tenant_idis unaffected — it is the Loki/Mimir tenant selector, not authentication. - The
urlmust behttpsfor a non-loopback host whenever any of these is set, exactly as fortoken, and for the same reason. Atlsblock on a non-httpsurl is refused rather than ignored: it would do nothing there. - Every credential is scoped to the configured host. If the endpoint redirects to another host, the request is re-dialed with none of it — basic auth, oauth2 token, custom headers and the SigV4 signature alike.
tlsapplies to every dial the source makes, including theoauth2token request. A source with both blocks connects twice — to its own URL and tooauth2.token_url— and onetlsblock configures both, so a Prometheus behind an internal CA reached through an SSO proxy works without a second block. Two consequences when the two are on different hosts, and the second is the one to decide about:careplaces the trust store rather than adding to it, so an identity provider on a public CA needs its root in the same bundle.cert/keyare a client certificate, and it is now presented to whatever hosttoken_urlnames — commonly a third party. TLS offers it during the handshake, before any application-level decision, so that host learns this deployment's mesh identity simply by being dialled. If yourtlsblock exists for mutual TLS to your own Prometheus and your token endpoint is somebody else's, that is a disclosure to weigh; there is no way to configure it away today short of removing the block, and a per-endpointoauth2.tls_configis the shape that would separate them.
SigV4 is the exception, deliberately — its credential hops dial IMDS and STS,
which the AWS SDK picks and you never write down, so they keep the system trust
store and are offered no certificate.
- headers cannot set a name the transport owns (Authorization, Host,
Content-Type, the X-Amz-* signing headers, …); those are rejected at load.
Use token/authorization for a bearer and basic_auth for basic.
- oauth2.token_url takes no query string, and no user:pass@ — the same two
refusals the source url gets, and for the same reasons. The userinfo half
binds on a WRITE, so on a deployment with sign-in configured
it reaches a config file too, through the start-up pass that provisions it — see
Where a source lives. With no sign-in there is no write
and the file is unaffected. Its remedy
depends on the grant: client_id/client_secret on client_credentials, where
that pair is what userinfo was carrying anyway; nothing on jwt-bearer, where
the signed assertion is the authentication and client_secret is refused — there
the userinfo simply has to go. Extra token-request parameters (an Auth0
audience, an Azure resource) go in endpoint_params, which the token
request carries in its form body. An IdP whose token endpoint genuinely needs a
query parameter in the URL cannot be expressed here; open an issue if you meet
one.
- sigv4 signs for the aps service in region. With access_key/secret_key
it uses that static pair (both or neither); leaving both empty is a
configuration, not a missing secret — it uses the standard AWS credential
chain (environment variables, ~/.aws, an EC2/EKS instance role or IRSA),
assuming role_arn on top when set, the same way an
AWS CloudWatch source resolves credentials. That is the shape
to reach for on a machine that already has an AWS identity, and it is the one
the add-source wizard cannot write: the wizard's SigV4 arm always emits a static
pair, and its hint on the Access key field points here.
- There is no password_file, credentials_file, client_secret_file,
ca_file, *_ref, proxy_url or proxy_from_environment. Those exist in
Prometheus' own http_client_config and are deliberately absent here: in the
hosted service a source config is written by the tenant, so a field that reads
a file off the worker's disk would be a way to exfiltrate the operator's own
config, and a proxy field would route the dial around the egress gate. Writing
one is a load error that says so. Keep secrets out of the file with
${ENV_VAR}, which config loading expands.
- Self-hosted only: a source using one of these blocks dials directly, so an
HTTP_PROXY/HTTPS_PROXY in the process environment does not apply to it. A
source using the bare token field is unchanged and still honours the
environment proxy.
- Most of this is reachable from the add-source wizard
rather than only from this file: it offers a scheme picker covering token,
authorization, basic_auth, a custom header, oauth2 client-credentials, and
sigv4 with a static key pair. Three shapes stay file-only, and the wizard does not offer them —
tls (a PEM chain is not something a form can guide), the
jwt-bearer grant above (same reason, and it takes a different secret from the
client-credentials grant), and a sigv4 block naming a role_arn or carrying no
keys (in the hosted service the shared worker would then sign a tenant's source
with the operator's own AWS identity). Self-hosted, the wizard never writes the
value you type into the snippet it generates: it emits an
INFRASIGNS_<SOURCE>_<FIELD> reference and the export line to set, which is
the ${ENV_VAR} expansion above used as intended.
If you are on the hosted service
Everything above is written for someone who edits this file. A tenant of the hosted service has no config file, no process environment and no restart, so the three file-only shapes are not "set them elsewhere" for you — they are not available, and the add-source wizard says so where you would look for them:
| you need | on the hosted service |
|---|---|
a client certificate (mTLS) or a custom CA (tls) |
not offered — ask your operator |
Google's service-account grant (jwt-bearer) |
not offered — ask your operator |
sigv4 with an IAM role, or with no keys at all |
not offered — enter a key pair of your own |
The last row is the one with a reason rather than a limitation behind it. A
keyless sigv4 block signs with whatever AWS identity the process has, and on
the hosted service that process is the operator's shared worker — so a tenant
source configured that way would be signed with the operator's credentials, which
is the confused-deputy shape AWS CloudWatch solves with a
per-source role_arn and an External ID. Until sigv4 grows the same thing, a
hosted tenant supplies a key pair.
What an operator can do about the first two: add the source to the organization
from the config file yourself, or run the affected backend behind a gateway that
takes one of the schemes the wizard does offer (a bearer token, an Authorization
header with a scheme word of its own, basic auth, a custom header, or OAuth2
client credentials).
Incident attribution (alert_labels)
Inbound webhook alerts can be attributed to a configured source, so its detail-page timeline shows them under the Incidents chip:
sources:
- name: production
url: http://prometheus:9090
alert_labels: # attribute alerts whose labels contain ALL pairs (AND)
cluster: production
- With exactly one source and no
alert_labelsanywhere, every alert is attributed to it automatically — zero config for the common single-source install. Adding a second source (or anyalert_labels) switches to explicit label matching. This shortcut is scoped to the incident timeline: the receiver's unrecognized-severity signal (infrasigns_alerts_unrecognized_severity_total, #350) uses label matching only, so it names a fleet on evidence rather than on the assumption that one source is the whole install. - An alert matching no predicate stays unattributed (it still appears on the Incidents page, and its "Attributed to" filter has an explicit Unattributed option); an alert matching more than one source is left unattributed and logged, never guessed. Predicates that make ambiguity inevitable (one a subset of, or equal to, another) are rejected at config load.
- CloudWatch alarms carry only a
regionlabel, soalert_labels: {region: eu-west-1}is the only usable predicate for them; AlertManager and Grafana alerts pass their full label set through. - Attribution happens at receive time — there is no bulk backfill for
incidents recorded before it was configured. An episode still delivering
events catches up on its next delivery that reaches the store (a repeat
firing past
webhook.dedupe_windowrefreshes the open episode — dedupe-suppressed repeats don't; a re-sent resolution fills an unattributed history row), but fully quiet history stays unattributed.
Query hints for the analysis engine
Each custom query optionally carries hints for the deterministic analysis
engine (used in every mode; with llm.provider: none they drive the report
directly):
sources:
- name: production
url: http://prometheus:9090
queries:
- name: cpu_usage_percent
query: '(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100'
signal: utilization # latency | traffic | errors | saturation | utilization
warn: 70 # optional static thresholds; higher-is-worse —
crit: 90 # a finding fires at value >= threshold
unit: "%" # optional display suffix for signal value columns
signaloverrides the name-based golden-signal classification (unhinted metrics are classified by substring heuristics: "latency", "error", "queue", "cpu", "request", …).unitis an optional display suffix appended verbatim to every number of this query's value column, in a digest, a trends report or an incident alert alike (23.7 → 11.2 ↓becomes23.7% → 11.2% ↓; a threshold breach's95.4 (limit 90)becomes95.4% (limit 90%)). Use"%"for a percentage," GB"(leading space) for a word unit; omit it to render bare numbers. It never affects a threshold or the analysis (a metric value carries no unit on its own, and a name is not a reliable unit) — but it is no longer cosmetic: since #524 it is also what the summary and verdict prompts tell the model the numbers are measured in, so a wrongunit:is a wrong statement to the model, not just a wrong label. It is bounded (a short display suffix, not a sentence) where a value is SAVED — the wizard, the source editor and any org-config write — rather than at load, so an over-long unit stored before that bound existed keeps its source running and is refused only when something tries to save it again; a config file carrying one loads with a warning — and, on a deployment with sign-in configured, is then refused by the start-up write that provisions it, which is one of the shapes Where a source lives is about. Its leading space is meaningful and is never trimmed, and a line break inside it is flattened to a space before it reaches a value column. DigitalOcean, CloudWatch, and the node fallback set carry a unit from their curated metric definitions where the API guarantees one (cpu/memory%, storage/network bytes, latenciess/ms); metrics with no reliably-known unit render bare (#237). Hetzner carries one for every curated metric (#524), transcribed from the unit table in Hetzner's own OpenAPI spec:%forserver_cpu,B/sfor the disk and network bandwidth series,iop/sandpackets/sfor the counters,/sfor the load-balancer rates.lb_open_connectionsstays bare because a count is dimensionless.server_cpuearns its%only because the collector divides the wire value by the server's vCPU count first — see Hetzner Cloud.- When you leave
unit:empty, the QUERY is consulted (#524) — and only when the query is a bare instant selector:node_memory_MemAvailable_bytes, or the same thing with a label matcher,node_memory_MemAvailable_bytes{job="node"}. Its metric name is then read against the base-unit suffixes the Prometheus/OpenMetrics naming convention defines:_seconds→" s",_bytes→" B", and_celsius,_volts,_amperes,_joules,_grams,_meters(each with the same leading separator space), plus_ratioas dimensionless. A trailing_totalor_sumis peeled first, so a barenode_network_receive_bytes_totalis bytes;_countand_bucketare not, because those count observations rather than measure the thing observed.
Anything that is not a bare selector renders bare, and that is most real
queries: rate(node_network_receive_bytes_total[5m]) is bytes per second,
rate(node_cpu_seconds_total[5m]) is a dimensionless core fraction,
free_bytes / size_bytes * 100 is a percentage and x_bytes / 1024 / 1024 is
megabytes — none of them is in the unit its metric name carries, so none gets
one. A function, an operator, an aggregation, a range selector, an offset or an
@ modifier all disqualify the query, and so does anything the recogniser is not
certain about: it is a deliberately narrow subset of PromQL's selector grammar,
and every doubt resolves to no unit. A LogQL query is never a bare PromQL
selector, so a Loki source's unit is whatever you set and nothing else.
_percent, _percentage, _ms, _kb and _mb are deliberately not
recognised — the convention does not define them, and a name saying "percent"
does not make a value a percentage (Hetzner's server_cpu is the
counter-example, below). An explicit unit: always wins, so set one whenever
your query is anything more than a bare selector, or its name does not describe
what it returns.
Two consequences worth knowing. A derived unit is not prompt-only: it is the
same field an explicit unit: fills, so it also labels the numbers in a
delivered digest's value column and in the web archive. And an empty unit: now
means "work it out", not "render bare" — there is no spelling for "this metric is
deliberately dimensionless", because _ratio renders exactly as an unrecognised
name does. If you want a bare number from a query whose name ends in a base unit,
rename the query's metric or accept the suffix.
- The unit is no longer display-only. Since #524 every metric line a summary or
verdict prompt renders states it, as a (unit: X) token beside the series
identity, with (unit: none) where nothing is known — and the prompt tells the
model to quote values in exactly that unit, to invent none where the line says
none, and not to read one out of the metric name either. The values themselves
stay bare in the prompt: a suffix welded onto a number is what would tell the
model a byte count is a percentage.
- warn / crit produce threshold-breach findings in digests and trends
reports. Thresholds are higher-is-worse; for lower-is-worse metrics (free
space, availability) invert the query expression instead. warn must be
<= crit when both are set.
- Baseline anomaly detection (recent window vs the rest of the fetch window,
z-score >= 3) and trend direction need no hints; anomalies need at least
16 data points in the analysis window (for trends: window / step >= 16 —
below that only trend and threshold findings fire; the digest's derived
window always yields enough points). Escalation to critical additionally
requires the deviation to be large relative to the metric's own level (≈30% of
the baseline mean), so a rock-stable gauge whose tiny variance turns a benign
excursion into a many-σ outlier stays a warning instead of escalating to
critical. This is range-free by design; for a bounded metric where danger is
proximity to a ceiling (a disk filling to 95%), configure a crit threshold —
that fires independently of the anomaly detector.
AWS CloudWatch
A cloudwatch source pulls metrics natively from AWS, for teams without a
self-hosted Prometheus:
sources:
- name: aws-prod
type: cloudwatch
region: eu-west-1
# uses the standard AWS credential chain: env vars, ~/.aws, or an IAM role
namespaces: [AWS/EC2, AWS/RDS] # optional; defaults to all supported
- Credentials come from the standard AWS chain (env vars, shared
~/.aws/config/credentials, or an IAM role / IRSA on EKS) — never from this file. The minimal read-only IAM policy:
{
"Effect": "Allow",
"Action": ["cloudwatch:GetMetricData", "cloudwatch:ListMetrics"],
"Resource": "*"
}
role_arn(optional) makes the source assume a specific IAM role viasts:AssumeRoleinstead of using the ambient chain above:
sources:
- name: aws-prod
type: cloudwatch
region: eu-west-1
role_arn: arn:aws:iam::123456789012:role/infrasigns-readonly
The assumed role carries the read-only policy above; the process's own identity
needs only sts:AssumeRole on that role. In self-hosted this is optional (leave
it out to use the ambient chain). In the hosted multi-tenant service it is
required — see the cross-account note below.
namespacesscopes which curated metric sets are fetched. Supported:AWS/EC2,AWS/RDS,AWS/Lambda,AWS/ApplicationELB,AWS/ECS. Omitting it fetches all of them. For each namespace InfraSigns discovers the running resources viaListMetricsand fetches the standard golden-signal metrics (CPU, latency, errors, saturation) in one batchedGetMetricDatacall. To bound cost, at most 500 series are fetched per cycle (excess is dropped with a log warning).
Narrowing this list lowers your AWS bill,
directly and proportionally, and this is the whole reason the key is worth
setting rather than leaving at its permissive default. AWS bills
GetMetricData per metric requested — $0.01 per 1,000 metrics requested —
and it is one of the three CloudWatch operations excluded from the free tier.
(The unit matters: this is the per-metric-requested tariff, not the standard
per-request one, which is a different line item.) An account that runs no
Lambda also gains nothing from AWS/Lambda being polled, and the noise dilutes
the digest. Leaving the list empty fetches every supported namespace, which is
the right default on day one and the wrong one a month in. See
CloudWatch pricing and
Identifying resources driving GetMetricData charges.
The bill is yours in both modes, but only a hosted tenant reaches this page
through a walkthrough — so the reason lives HERE, in mode-neutral reference
material, rather than only inside Recommended setup (hosted) below, which is
where a self-hosted operator following the wizard's own link would never have
found it (#499 review round 2, X5).
- metrics optionally replaces the curated set with an explicit list, e.g.
a custom or application namespace:
sources:
- name: aws-prod
type: cloudwatch
region: eu-west-1
metrics:
- name: queue_depth
namespace: AWS/SQS
metric_name: ApproximateNumberOfMessagesVisible
stat: Average # Average | Sum | Minimum | Maximum | SampleCount
dimensions: { QueueName: jobs } # optional; omit to fetch every instance
signal: saturation # same engine hints as Prometheus queries
- CloudWatch alarms are ingested on the push side, not here: point an SNS
topic at an InfraSigns webhook subscription (
X-Infrasigns-Source: cloudwatch). Metrics (this section) and alarms (the webhook) are independent — configure whichever you need. - Cross-account (hosted service). In the multi-tenant hosted service a
CloudWatch source must set
role_arn: without it the source would resolve against the worker's own AWS identity, reading the operator's metrics rather than yours (a confused-deputy), so a role-less cloudwatch source is skipped for hosted organizations. To grant access, create a role in your AWS account that trusts the InfraSigns hosting account and pins your organization id as the STS external ID:
{
"Effect": "Allow",
"Principal": { "AWS": "<the principal your operator declares>" },
"Action": "sts:AssumeRole",
"Condition": { "StringEquals": { "sts:ExternalId": "<your-organization-id>" } }
}
The Principal.AWS placeholder stands for the WHOLE value, not for a fragment
inside an ARN: your operator's
aws.worker_principal_arn is substituted for it in one
piece, so it becomes something like arn:aws:iam::123456789012:role/infrasigns-worker
— never an ARN nested inside another. You do not have to construct it: the
add-source wizard, which collects the region and
the role_arn, prints a ready-made copy of this policy carrying the
real principal and your organization id already filled in. A hosted deployment
that declares no principal offers the CloudWatch tile disabled and shows no
policy, because there would be nothing for it to name.
InfraSigns always sends your organization id as the external ID (it is
server-supplied, never taken from your config), so a role scoped to your org id
can be assumed only for your organization — no other tenant can name a role
under it. Self-hosted deployments need no external ID (a single AWS account); a
role_arn there is assumed directly.
Same partition, both ends. AWS does not support sts:AssumeRole across
partitions, so your role must live in the same partition as the principal above:
an aws-cn or aws-us-gov role cannot be assumed from an arn:aws:… principal,
or the other way round. Nothing checks this for you — InfraSigns validates the
shape of each ARN, not that the two agree. What you will most likely see is the
same failure a mistyped principal gives: IAM refuses to save a trust policy
naming a principal it cannot resolve, and a principal in another partition is
one it cannot resolve, so the error arrives while you are creating the role
(Invalid principal in policy) rather than later. We have not exercised the
cross-partition case, so treat that as reasoning from how IAM stores a
principal rather than as a measurement; if a partition does accept the save, the
assume itself still fails and the symptom is an AccessDenied at collection.
Operator setup (hosted-service deployments). A cross-account assume needs BOTH ends, and only the first is the tenant's:
- the tenant's role trusts your principal, with the external-ID condition — the policy above;
- your worker principal carries an identity policy allowing
sts:AssumeRoleon the tenant's role. Without it every collection fails withAccessDeniedno matter how correct the tenant's trust policy is.
Operator hardening, and the trade-off in (2). role_arn is tenant-supplied
and can name any account — including the hosting account. The per-tenant
isolation above holds only if the worker's own IAM principal cannot assume a role
that ignores the external ID. Two obligations, and they pull against each other:
ensure no role in the hosting account trusts the worker/account root without an
sts:ExternalId condition — otherwise a tenant naming such a role would be
assumed despite the external-ID mismatch, re-opening the confused-deputy one
level up — and decide deliberately how the grant in (2) is scoped. Naming
specific customer role ARNs is the tightest form and it does not survive
self-service: a new organization's role ARN does not exist until they type it, so
an enumerated grant means editing an IAM policy per signup. The practical shape
is a grant broad enough to cover roles you have not seen yet — excluding your own
account, which is the part that matters — with the external-ID condition doing
the per-tenant isolation, since InfraSigns supplies that value itself and a
tenant cannot forge another organization's. Whichever you pick, this is a
deployment-side obligation InfraSigns cannot enforce in code.
Do not delete and recreate the worker role under the same name. IAM resolves a role-ARN principal to the role's unique id when a trust policy is SAVED, not when it is used, so a recreated role — same name, same ARN, new unique id — leaves every tenant policy already stored pointing at a principal that no longer exists, and every one of them has to be re-saved. The stored policies do not report this: they simply stop matching.
Recommended setup (hosted)
Everything above is reference material: what each key means, what the policies contain, what the operator owes. This is the ordered walkthrough — what to do, in what order, if you are a tenant of a hosted InfraSigns and you are staring at the CloudWatch step of the add-source wizard. The wizard links here, and it keeps only what you must type (region, role ARN) plus the one thing only it can give you: your organization's External ID.
-
Copy the External ID from the wizard. It is the value under External ID on the CloudWatch step. It is not a secret — it names your organization, and its whole job is to appear in your role's trust policy so that role can be assumed for your organization and nobody else's.
-
Create an IAM role in your AWS account with a custom trust policy. Paste the JSON the wizard prints under Show the role trust policy — it already carries the hosted deployment's principal and your External ID, so there is nothing to fill in. (If the deployment has declared no AWS identity of its own you never reach this step: the wizard offers the CloudWatch tile disabled, with a sentence saying so under the row. Ask your operator.)
-
Attach the read permissions. The minimal policy is the one at the top of this section:
cloudwatch:GetMetricDataandcloudwatch:ListMetrics, onResource: "*". Those TWO are the whole of it — InfraSigns callsListMetricsto discover what exists andGetMetricDatato read it, and never writes. Nothing else is needed, and granting more is a wider role than the product uses. -
Paste the role ARN back into the wizard, pick the region the metrics live in, and add the source. One source is one account and one region; a second region is a second source.
-
What to expect on the first cycle. Nothing is probed when you add the source — a live probe here would be an
sts:AssumeRolefrom a web form — so the source is verified on its first collection instead. If the trust policy or the identity grant is wrong you will seeAccessDeniedon the source's page rather than at save time. Give it one digest cycle before concluding anything. -
Then narrow the namespaces, from the source's own page — not from the wizard, which is why the wizard no longer asks. Two reasons, and the second is the one that costs money:
-
Relevance. An account that runs no Lambda gains nothing from
AWS/Lambdabeing polled, and the noise dilutes the digest. - Cost. A shorter namespace list is a smaller AWS bill, directly and
proportionally, because
GetMetricDatais billed per metric requested and sits outside the free tier. The tariff, the unit it is charged in and the two AWS references have ONE home so the figures cannot drift:namespacesin the reference above.
DigitalOcean
A digitalocean source pulls metrics natively from the DigitalOcean Monitoring
API, for teams on Droplets, managed databases, and load balancers without a
self-hosted Prometheus:
sources:
- name: my-do-account
type: digitalocean
token: ${DO_TOKEN} # personal access token, read scope
resources:
- type: droplet
tags: [production] # optional: only droplets carrying ALL listed tags
- type: database # managed MySQL clusters
ids: [db-abc123] # optional: these exact ids (skips listing)
- type: load_balancer # optional: omit ids/tags for all of the type
tokenis a DigitalOcean personal access token with read scope. It is required:${DO_TOKEN}expands to the empty string when the variable is unset, and start-up then refuses the config withsource[N]: token is required for digitalocean sources. Export it before you restart — that is what the wizard's "Set DO_TOKEN…" line beside the generated snippet is telling you. It rides anAuthorizationheader — it never appears in logs or error messages. No write scope is needed; InfraSigns only reads metrics and lists resources.resourcesselects what to monitor — at least one selector is required. Each names atype(droplet,database, orload_balancer) and optionally narrows it:ids: [...]— monitor exactly these resource ids (no listing performed). Config-file only: a hosted organization's edit page does not offer it, but a stored list is shown there and is preserved through every edit made from it.tags: [...]— list resources of the type and keep only those carrying every listed tag (AND). DigitalOcean load balancers and databases also support tags. A hosted organization sets this from the source's edit page; a config file is the self-hosted route.- neither — monitor all resources of the type in the account.
A tag written from the edit page may not contain ,, = or ! (the
separators of a Hetzner label selector — one rule covers both providers so the
two cannot drift), nor a space or an invisible character. That rule binds on
write only, and it binds on the whole organization's configuration: a value
already stored keeps collecting, but until it is repaired every save that
organization makes is refused, including saves about something else entirely. A
config file is never subject to it, and neither is a selector pinned to an
explicit ids list — ids take precedence and the collector never reads that
kind's tags, so there is no control on the page to repair and nothing to
police.
- Curated metrics per type (the analysis engine finds anomalies and trends on
all of them; only the percentage metrics carry static thresholds):
- droplet — CPU, available memory, free filesystem, 1-minute load, public
inbound/outbound bandwidth.
- database — CPU %, memory %, disk % (with 80/95 warn/crit thresholds).
DigitalOcean exposes monitoring metrics only for MySQL clusters, so
non-MySQL databases are skipped during discovery (logged once per cycle).
Per-service metrics (connections, query latency) need a per-cluster service
identifier and are not wired yet.
- load_balancer — requests/sec, current connections, average response time,
and HTTP responses (for error-rate analysis).
- No alarms. DigitalOcean has no AlertManager, so InfraSigns anomaly detection
is the only alert path for these sources — there is no push/webhook side.
- Cost bound. At most 200 (resource × metric) Monitoring calls are issued per
cycle (excess is dropped with a log warning), fanned out with bounded
concurrency. Each cycle additionally makes one reachability probe plus the
resource-discovery listing calls (paged), so the DO API call count is ~200
metric calls plus discovery — a tags filter over a large account pages the
fleet to find matches.
- Per-source token, in both modes. A DigitalOcean source carries its own
token, so a hosted organization can add one from the add-source wizard and it is
stored with that organization's configuration — there is no shared account. In a
self-hosted deployment the token comes from the environment
(${DO_TOKEN}) like every other secret.
Hetzner Cloud
A hetzner source pulls metrics natively from the Hetzner Cloud Metrics API —
the cost-conscious choice for EU/Go teams on Hetzner without a self-hosted
Prometheus:
sources:
- name: my-hetzner
type: hetzner
token: ${HETZNER_TOKEN} # API token, read scope
resources:
- type: server
labels: {env: production} # optional: only servers with ALL these labels
- type: load_balancer
ids: ["4711"] # optional: these exact numeric ids (skips listing)
tokenis a Hetzner Cloud API token with read permission. It is required:${HETZNER_TOKEN}expands to the empty string when the variable is unset, and start-up then refuses the config withsource[N]: token is required for hetzner sources. Export it before you restart — that is what the wizard's "Set HETZNER_TOKEN…" line beside the generated snippet is telling you. It rides anAuthorizationheader — it never appears in logs or error messages. No write permission is needed; InfraSigns only reads metrics and lists resources.resourcesselects what to monitor — at least one selector is required. Each names atype(serverorload_balancer) and optionally narrows it:ids: [...]— monitor exactly these numeric resource ids (no listing). Config-file only: a hosted organization's edit page does not offer it, but a stored list is shown there and is preserved through every edit made from it.labels: {k: v, ...}— list resources of the type and keep only those whose labels contain every listedkey=valuepair (AND), applied server-side as a Hetzner label selector. A hosted organization sets this from the source's edit page; a config file is the self-hosted route.- neither — monitor all resources of the type in the project.
A label key or value written from the edit page may not contain ,, = or
!, nor a space or an invisible character; a key may not be empty, while an
empty VALUE is legal (env= selects resources carrying the label with no
value). Those three characters are the label selector's own separators, and the
selector is built by joining key=value pairs with commas — so a value carrying
one would forge additional terms and quietly enumerate a different subset of
your project. The rule binds on write only, and it binds on the whole
organization's configuration: a value already stored keeps collecting, but until
it is repaired every save that organization makes is refused, including saves
about something else entirely. A config file is never subject to it, and neither
is a selector pinned to an explicit ids list — ids take precedence and the
collector never reads that kind's labels, so there is no control on the page to
repair and nothing to police.
- Curated metrics per type (the analysis engine finds anomalies and trends on
all of them). Every one carries the display unit Hetzner's own OpenAPI spec
documents for it (#524):
- server — CPU (%), disk read/write IOPS (iop/s) and bandwidth (B/s),
network in/out packets-per-second (packets/s) and bandwidth (B/s).
Multi-disk / multi-NIC servers produce one series per device (a device label
disambiguates).
- load_balancer — open connections (a bare count), connections/sec and
requests/sec (/s), and inbound/outbound bandwidth (B/s).
- server_cpu is normalized per vCPU, and that is a change in the numbers.
Hetzner's wire value for cpu is SUMMED across the server's vCPUs — its spec
calls the series "percent" and never states the range, and a 2-vCPU server in a
real account peaks at 187.6 — so a raw reading runs 0..vCPU×100 and is not a
percentage. InfraSigns divides it by the server's vCPU count at ingestion, so the
series it stores, charts, analyzes and reports IS a 0-100 percentage of the whole
machine. That is what earns it the % unit. No Hetzner metric carries a static
warn/crit: normalization makes one possible for server_cpu and does not
decide it, and a curated threshold here would have no opt-out (a hetzner source
rejects queries:) — tracked separately.
- Readings from before and after the upgrade are not comparable. They are
cores× apart on a multi-core server, so a chart spanning the release shows a
step and nothing in the product marks it. This reaches every consumer of the
series, not just reports: a health check assertion written against the old
scale ("CPU stays under 150" on a 4-vCPU box) can now never fail and reports
pass forever, a deploy verification spanning the upgrade compares two
scales, and MCP query_metrics returns the new numbers. Re-read anything
that names a Hetzner CPU number.
- A server whose vCPU count cannot be resolved reports no CPU. The count comes
free with the discovery listing; a selector pinned to explicit ids never lists,
so there it costs one bounded API read per configured server id — issued
concurrently and memoized for an hour (with a jittered expiry, so a whole
selector does not re-resolve in one burst), since a vCPU count changes only on a
resize. A FAILED lookup is memoized for five minutes rather than not at all:
everything that fetches from a source — the digest cycle, each due health check,
deploy verification, the reporter and MCP query_metrics — shares the memo but
not the call, so an unmemoized failure meant every one of them re-asking for
every failing id, with the client's own five retries inside each call. Five
minutes keeps a transient failure short-lived — on a five-minute schedule the
entry expires around the next cycle or the one after. An id Hetzner
reports as nonexistent is dropped and named, and costs no
metrics call. If the count is genuinely unavailable (the API returned no server
type, or the lookup was throttled), that target's server_cpu series is
omitted and the failure is logged, rather than published on an unknown scale
under the same name and unit as every normalized one. The target's other metrics
are unaffected — eight further metric definitions for a server, and one series
per device where a machine has several disks or NICs.
Because the omission is per target, a fleet can split under one metric name: some
servers reporting server_cpu and some not, with a fleet-wide sentence computed
over the ones that answered. The per-target error in the log is what makes that
visible.
- A sanity guard on the divisor. The measurement behind the normalization is
one live reading of one server type (a shared-vCPU cx23); the dedicated
(ccx*) and Arm (cax*) lines were not measured. A normalized sample above 110
is counted and warned about once per fetch, which is the tell that the divisor is
too SMALL for that server type. It cannot see the other direction: a divisor that
is too LARGE deflates the series, which then stays inside 0-100 and trips no bar
on the value. That direction is only reachable through a stale memo, so it is
bounded rather than detected — a listed target re-reads its count every cycle,
and an ids: target can be stale for at most the memo's hour after a downsize,
during which a pinned server reads low and a check asserting "CPU stays under 90"
reports pass.
- 5-minute resolution. The Hetzner Metrics API returns 5-minute averages for
anything but very short ranges, not raw samples — so anomaly detection over
Hetzner series is coarser than over a fast-scrape Prometheus. This is a Hetzner
API limitation, not a configuration knob.
- No alarms. Hetzner has no AlertManager, so InfraSigns anomaly detection is
the only alert path for these sources — there is no push/webhook side.
- Cost bound. At most 200 resources are monitored per cycle (excess is dropped
with a log warning). Hetzner returns every requested metric type for a resource
in one Metrics call, so a cycle makes at most ~200 metric calls (fanned out with
bounded concurrency) plus one reachability probe and the paged resource-discovery
listings.
- Per-source token, in both modes. Like DigitalOcean, a Hetzner source carries
its own token, so a hosted organization can add one from the add-source wizard
and it is stored with that organization's configuration; self-hosted reads it
from the environment (${HETZNER_TOKEN}).
Healthcheck URLs
A healthcheck source periodically probes a list of HTTP endpoints and reports
each one's availability and response latency — synthetic monitoring for a URL
with no metrics backend, or for an operator with no Prometheus at all.
sources:
- name: public-endpoints
type: healthcheck
endpoints:
- url: https://api.example.com/healthz # required; http or https
- url: https://example.com/status
method: HEAD # GET (default) or HEAD
expect_status: 204 # exact status; omit for "any 2xx"
timeout: 5s # default 10s, max 30s
severity: warning # down-alert severity: critical (default) | warning | info
endpoints— one or more probes. Each needs aurl;method,expect_status,timeoutandseverityare optional (defaults above) — but the UI editors require an explicitseverity(#357): a form row has a select the user simply never touched, and the tier is what pages, so the add-source wizard and the source-detail edit form refuse an unset one while this file keeps its default. A stored source that omits it keeps working and opens its edit form showingcritical, the tier in force. A probe is up when the endpoint answers with the expected status (any2xxwhenexpect_statusis unset), down on any other status, a transport error, or a timeout. Redirects are followed and the final status is checked, so anexpect_statusin the3xxrange is rejected at config load (it can never be the final status).- At most 200 endpoints in one source, and at most 50 through the web UI.
A source probes its whole list every digest cycle, and 200 is the most this
product will store for one source. Since
#500 a longer list is
refused by the org-config write gate rather than stored and silently
truncated at probe time. The refusal quotes a position and the limit —
source[0].endpoints lists 201 endpoints, more than the maximum of 200— because that is how every source rule in the validator spells one element ofsources:. What that position MEANS depends on which path produced the message, so the three cases below say so individually. Drop the endpoints you do not need, or move them into a secondhealthchecksource, which spends another source wherever a plan limit applies.
Where that bites depends on how this deployment is set up, and the three cases differ:
- Sign-in configured. Your
sources:are provisioned into the stored organization config, one entry at a time, through that same gate — so an over-long list in this file is refused at startup and the process exits rather than running a configuration it cannot store. Each entry is checked inside a one-source document of its own, so the message always readssource[0]whatever the entry's place in your file; the log line beside it names the source bynameand says so. Catch it first withconfig validate --strict-provisioning, which is what that flag is for. There is no plan on a deployment like this one — tier limits exist only for a hosted organization — so "spends another source" costs you nothing here. - No sign-in. Nothing is stored and the gate is never reached: this file still loads and still probes the first 200. The collector logs a warning naming the source, the maximum and how many endpoints are being ignored — each time that source's collector is built, which is at startup and again whenever the runtime is rebuilt, not once per process.
- Saved through the web UI. No web save can create a list this long: the
add-source wizard and the source-edit endpoints editor refuse at 50 first,
with their own wording. That is deliberate — a form bounds what it will
generate, the store bounds what it will hold — and the two must stay in that
order. It can still be answered with the 200 message, because a save
canonicalises the organization's whole stored document: if a list already in
the column is over 200 (seeded by an operator, or written before this rule),
every org-config save is refused until it is shortened, not just that
source's. Shorten it in that source's edit card — and note that the card
refuses at 50, so cutting to exactly 200 is refused again there.
infrasigns config checkreports the same refusal by name. - No secrets in the URL. The
urlis shown in labels, the web UI, and every digest/incident delivered to your channels and the LLM. Userinfo (user:pass@) and the entire query string are stripped from that displayed/labelled form, so a?token=…would silently vanish there — put credentials in the endpoint's ownbasic_authorauthorizationblock instead (below). Because two endpoints that differ only in userinfo or query then look identical, they are rejected as duplicates at config load: distinguish endpoints by scheme/host/path, not by a query string. ${ENV}in an endpoint URL — and in its credential. In this file it is expanded like every other value — the whole document is expanded before it is parsed. Since #544 anendpoints[].urlstored through the web UI resolves one too, on a self-hosted deployment's own organization: the wizard's live probe dials the resolved address, both write paths judge the resolved value, and the generated config snippet emits the reference — so what is stored is the reference and rotating the variable takes effect without re-entering anything. Before that release both surfaces judged the string you typed, so an endpoint behind a reference was refused with "scheme must be http or https" — its own wording for the rule a sourceurl:stated as "the URL must be absolute". That now covers the endpoint'sbasic_auth.passwordandauthorization.credentialson the same terms — see Authenticating a health probe. Resolution takes an owner of that organization (#552); a member of it, and any hosted organization, is told the reference is not resolved for them, with the offending field named and the remedy it asks for — enter the address itself for aurl, the value itself for a credential. The rules and the reasons are the same ones a sourceurlfollows — see Where a source lives.- Metrics. Each endpoint emits
up(1/0) andlatency_msseries, labelled by that strippedurl— so the digest shows current availability and response time. - Down = alert. A down endpoint raises an alert at the endpoint's
severity; acriticalone (the default) makes the cycle a paged incident, awarningorinfoone rides the next regular digest. - No history / limited trends. A healthcheck has no metrics backend — each cycle probes live and records a single point, so trend analysis (which needs several points) is effectively flat. The value of a healthcheck source is the down alert and the current state in each digest, not a trend. Each endpoint is probed twice per digest cycle (once for the alert, once for the metric series) and once more per trends cycle, so size your endpoints' own rate limits accordingly.
- Readiness. A down endpoint never marks the pod NotReady. That is no longer
an exception this source type earns — since
#507 no source does
(Readiness) — but it is true twice over
here: this source's
/readyzentry reflects the prober itself and is always ready, so a down endpoint does not even move the top-levelstatustowarningthe way an unreachable Prometheus does. Endpoint health is reported through the digest and theupmetric instead. - Cloud-safe. Unlike the token-based cloud collectors, a healthcheck source carries no ambient credential, so a hosted deployment routes its probes through the same egress (SSRF) gate as a tenant Prometheus URL — a probe that resolves to a private or metadata address is refused at dial time.
Authenticating a health probe
An endpoint that needs a credential carries its own — per endpoint, not per source, because one healthcheck source commonly probes several services:
sources:
- name: public-endpoints
type: healthcheck
endpoints:
- url: https://api.example.com/healthz
severity: critical
basic_auth:
username: prober # optional; some vendors issue a token-as-password
password: '${PROBE_PASSWORD}'
- url: https://internal.example.com/status
severity: warning
authorization:
type: Bearer # optional; Bearer is the default
credentials: '${PROBE_TOKEN}'
- Two schemes, mutually exclusive.
basic_authsends anAuthorization: Basicheader;authorizationsends a rawAuthorization: <type> <credentials>one. Setting both is a config error. A health probe is a singleGETorHEAD, so the oauth2, sigv4, custom-header and TLS blocks a Prometheus or Loki source can carry are deliberately not offered here. passwordandcredentialsare encrypted at rest in the hosted service and wherever a self-hosted deployment stores its configuration in the database. They are never displayed, never logged and never pre-filled in a form. Each must be at least 4 characters, like every other stored credential: a shorter value cannot be masked in an error message, so the product would be able to store one it could not hide. That bound is enforced when the configuration is written, not when it is read, so a document that predates it keeps loading and is repaired on the next save.- https is required whenever an endpoint sets a credential and its host is not
the loopback — the same rule a source
urlwith a token follows. ${ENV}works in the credential, not only in the URL. In this file the whole document is expanded before it is parsed, sopassword: '${PROBE_PASSWORD}'has always worked — and it is the form the wizard's own generated snippet emits, as'${INFRASIGNS_<SOURCE>_ENDPOINT_1_PASSWORD}'. The wizard's live probe now resolves one inbasic_auth.passwordandauthorization.credentialstoo, so an authenticated endpoint written that way previews as itself instead of as a 401; until #545 the check that decided whether to resolve anything read the endpoint'surlalone, which left exactly the credential the snippet mints unexpanded. Resolution is an owner's capability on this deployment's own organization, as it is for an address, and what is stored — and what the snippet emits — is still the reference.
A failed probe of an endpoint whose URL came from a variable names the
variable rather than the host it resolved to ("check what this deployment's
environment holds for ${PROBE_HOST}"), because the underlying transport error
would otherwise carry that host in its own words. Nothing about a healthcheck
source's address reaches the process log either: a verify failure for this source
type is logged against the fixed label (healthcheck endpoints). That is not so
for a Prometheus or Loki source, whose verify and preview failures still log the
resolved address — deliberately, since the only reader who can cause a
resolution at all is an owner of this deployment's own organization, who may read
that log.
- The wizard and the source's edit card both offer it. In the edit card the
secret field is left blank and blank means keep the stored value — while the
row still points at the same address on the same scheme: type a new value to
rotate it, or pick No credential to remove it. A credential travels only with
its own address. If you change an endpoint's URL, enter its credential again;
a submission that changes the URL while leaving the secret blank is refused on
the endpoints control rather than sending a credential stored for one address to
another one. The editor says so before you submit: a hint under the credential
fields carries the rule (leave blank to keep the stored one) and turns into a
visible warning the moment the URL stops matching. Switching between the two
schemes leaves the new scheme's field empty, because it is a different
credential. Reordering the rows or deleting a neighbour keeps every other row's
credential. If the endpoints changed in another tab while your page was open,
the save is refused and asks you to reload rather than guessing which endpoint
your row meant. A save refused for any other reason discards the credential you
typed and says so, so enter it again with your correction.
- Redirects drop it. Probes follow redirects and check the final status. Go's
HTTP client strips an Authorization header when a hop crosses to a different
host, so a probe that redirects off-host arrives unauthenticated and will report
whatever that host answers — but its test reads the host NAME only, so it treats
a different port and a subdomain of the same domain as the same host, and it
says nothing about the scheme. InfraSigns adds the missing floor: a hop that
leaves https arrives unauthenticated too, whatever host it names.
Loki (LogQL)
A loki source runs your LogQL metric queries against a
Grafana Loki instance and feeds the resulting
numeric series into the same digest / incident / trends engine as Prometheus —
so logs-derived signals (error rate, per-level counts, anomaly rates) sit next to
your metrics. It is the near-twin of a Prometheus source: same url + queries
shape, only the query language and the endpoint differ.
sources:
- name: app-logs
type: loki
url: http://loki.monitoring.svc:3100 # Loki HTTP API base URL
tenant_id: team-a # optional; X-Scope-OrgID for multi-tenant Loki
queries: # at least one; each is a LogQL METRIC query
- name: error_rate
query: sum(rate({app="api"} |= "error" [5m]))
signal: errors # optional engine hint
unit: /s # optional display suffix in the digest
- name: warn_logs
query: sum by (level) (count_over_time({app="api"} | logfmt | level="warn" [5m]))
log_queries: # optional; raw log lines for LLM context
- name: api_errors
query: '{app="api"} |= "error"' # a raw LogQL LOG selector (returns lines)
limit: 30 # optional per-query line cap (default 20, max 100)
- name: api_errors_on_the_firing_instance
query: '{app="api", instance="%INSTANCE%"} |= "error"' # correlated: filled from the firing alert
url— the Loki HTTP API base. Each query is issued asGET {url}/loki/api/v1/query_range.queries— LogQL metric queries —rate,count_over_time,sum,bytes_rate,quantile_over_time, etc. — whose result is a matrix of numeric series, exactly like a PromQL range query.name,signal,warn,critandunitbehave exactly as they do for a Prometheus query (see Query hints for the analysis engine). A raw LogQL log query (a bare stream selector, which returns log lines) is rejected here — wrap it in a metric aggregation, or put it inlog_queries.log_queries(optional add-on; needs a metric query) — raw LogQL log selectors (e.g.{app="api"} |= "error") whose recent lines are captured as LLM context on a critical cycle (#246). Because a loki source emits no alerts, a "critical cycle" for it means a critical finding — e.g. an error-rate metric query breaching itscrit:threshold. When that fires, the most recent lines (per-querylimit, default 20, max 100; capped in total; max 10log_queries) are attached to the digest / incident summary so the model can reason over the actual logs. On a normal cycle no log query runs.log_queriescarry nosignal/warn/crit/unit(a line is not a numeric series) and are not citable claim evidence — they are context only. They are an add-on, not a substitute: a loki source still needs at least one metricqueries, because the metric queries are the numeric digest/trends signal — a source with none has nothing to summarize. That holds for a loki source configured purely as another source'slog_sourcetarget too, whoselog_queriesfire on that source's critical cycle.Privacy: captured log lines are sent to your configured LLM provider (for
provider: openai/anthropic, a third-party API) and may be quoted into the stored report. Application logs often contain secrets or PII (tokens, emails, request bodies) — scope yourlog_queriesselectors to streams safe to share, especially on a hosted LLM. Control characters and embedded newlines are neutralized, but secrets are not redacted.- Placeholders — scoping a
log_queryto the firing alert (#347). A selector may carry a placeholder, which is filled from the labels of the alert that made the cycle critical before the query is sent:
| Write in the query | Filled from the alert's label |
|---|---|
%INSTANCE% |
instance |
%JOB% |
job |
%POD% |
pod |
%LABEL_<name>% |
the label named, verbatim — %LABEL_url% reads url |
The first three are shorthands for the generic form, so %POD% and %LABEL_pod%
are the same query. The token family is upper-case, but the label name inside
%LABEL_<name>% is taken verbatim and case-sensitively, because it is a
Prometheus label name: %LABEL_url% reads url while %LABEL_URL% reads a label
literally spelled URL. The name must follow the Prometheus label-name grammar
([A-Za-z_][A-Za-z0-9_]*), so %LABEL_% and %LABEL_9x% name no label: they are
never substituted, and they are reported as unrecognized. A case near-miss —
%instance%, or %label_pod% since the LABEL_ prefix is upper-case too — is
reported as well, with the spelling that would have worked; it is never
substituted either (see the typo note under
log_source).
log_queries:
- name: errors_on_the_firing_instance
query: '{app="api", instance="%INSTANCE%"} |= "error"' # correlated
- name: errors
query: '{app="api"} |= "error"' # standing
- Correlated vs standing. A selector holding at least one recognized
placeholder is correlated; one holding none is standing and behaves
exactly as it did before. The correlated selectors are preferred, not
exclusive: they run first, and if they return no lines at all the standing
ones run too and their lines are used source-wide. That matters because a
selector can fill perfectly and still match nothing — a Prometheus
instanceishost:portwhile a Loki stream label is usually the bare host — and without the fallback the incident would then be summarized with no logs. So a capture issues as many Loki queries as it has correlated selectors, or that many plus the standing ones in the case that had already produced nothing; if no correlated selector fills, exactly the standing ones run. - Whose alert. The identity is the root-cause alert (the earliest firing
alert) of the top correlated group of the source the incident is about —
with
log_sourceset, that is the alerting source, not the Loki source whoselog_queriesare being filled. Onlyprometheusandhealthchecksources emit alerts, so a critical finding — a Loki source breaching its owncrit:threshold — has no firing alert to narrow to and always resolves to the standing set. - Which labels there are to use. A Prometheus alert carries the labels its
alerting rule and the underlying series set (
instanceandjobare the usual ones;podonly if your rules carry it). A healthcheck alert carriesurl,methodandsource— plusstatus_codewhen the endpoint answered at all — and neverinstance/job/pod, so a healthcheck-driven incident needs the%LABEL_url%form. -
Values are gated, not escaped. A substituted value must be non-empty, at most 300 characters, and built only from
A-Za-z0-9and_ . : / -— so a realinstance(host:port) and a real healthcheckurl(scheme://host/path) pass, while a quote, a brace, a space or a%is refused. A refused value means the query does not run; nothing is ever sent half-substituted. See prompt injection for what that buys and what it does not.A selector mixing a recognized placeholder with an unrecognized one (
%INSTANCE%beside a typo'd%PDO%) is refused whole, not sent with the typo in it; so is one whose filled form exceeds 4096 bytes. Both count as unscopable and the standing selectors cover the capture. - A literal%needs no escaping. Only a run shaped like one of the tokens above is treated as a placeholder, so|= "50% of %total%"is left alone, and a selector with no recognized placeholder is sent byte-for-byte as you wrote it. That conservatism is what makes the token family upper-case, and it has a consequence: a lower-case%instance%— or%label_pod%, since theLABEL_prefix is upper-case too — is not a placeholder and is never substituted, but because it differs from a recognized token only in letter case it is reported, with the spelling that would have worked, just as a misspelled upper-case%INSTNACE%is (below). A run that is neither —%total%— stays silent.
What happens when a placeholder does not fill — and what a typo costs — is under
log_source below;
it applies to a Loki source grounding its own findings too.
- tenant_id (optional) — the Grafana Loki tenant to query. A multi-tenant
Loki (the common self-hosted setup) requires an X-Scope-OrgID header to select
the tenant and rejects a tenant-agnostic query with no org id; set tenant_id
and InfraSigns sends it on every query. InfraSigns does not treat it as a
credential (it names a tenant), so it may be a literal or a ${VAR} — put real
authentication in front of Loki (a reverse proxy, or the bearer token below)
rather than relying on X-Scope-OrgID as access control. It is sent only on the
query endpoint, never on Loki's tenant-agnostic /ready readiness check.
The value must be free of control characters (it rides an HTTP header) and at
most 150 bytes — a longer or CR/LF-bearing value is rejected at config load.
- Authentication. For an authenticated Loki endpoint (e.g. Grafana Cloud
Logs) set an optional bearer token — sent as Authorization: Bearer <token>,
exactly like a
Prometheus source. It is
a secret; inject it via ${VAR} and keep it out of the URL.
- No secrets in the URL. A query string (?token=…, or a bare trailing ?)
is rejected at config load, exactly like a Prometheus source — put the
bearer in token, or use Loki's own auth (e.g. a reverse proxy). Userinfo
(user:pass@) is allowed in the config file on a deployment with no
sign-in and still sent on the wire to connect; it is
stripped from the displayed URL in the web UI and logs, so a credential there
never renders. Everywhere a source is WRITTEN it is refused since
credential encryption, because that URL is
left readable in the database and token is the sealed place for a credential
to live — a stored loki source on save, and, once sign-in is configured, the
config file too, at the start-up write that provisions it
(Where a source lives).
- Readiness. Healthy / /readyz probe Loki's GET /ready — a loki source
that cannot reach Loki reports its error in the /readyz body and moves
the top-level status to warning; it does not gate the pod
(#507). The probe is
cheap and unmetered.
- Cloud-safe. Like a healthcheck and Prometheus URL, a tenant's loki url is
dialed through the egress (SSRF) gate in a hosted deployment.
Grounding an incident in another source's logs (log_source)
In a real deployment the alerts live in Prometheus and the logs live in Loki —
two different sources. Log capture (log_queries, above) is per-source, so on its
own it can only ground a loki source's own critical finding: an
alert-driven incident, on a prometheus or healthcheck source, has no logs to
attach because neither of those source types can fetch any.
log_source closes that gap. It names another configured source whose recent
log lines are attached to this source's incident summary:
sources:
- name: prod
url: http://prometheus:9090
log_source: app-logs # a critical alert on prod is summarized WITH app-logs' logs
- name: app-logs
type: loki
url: http://loki.monitoring.svc:3100
queries:
- name: error_rate
query: sum(rate({app="api"} |= "error" [5m]))
log_queries: # required for a log_source target
- name: api_errors
query: '{app="api"} |= "error"'
- The target must exist, be a
lokisource, and carry at least onelog_queries, so a target that would silently produce nothing is named rather than shipping an empty log section. WHERE it is named depends on whose document it is, and Where a source lives has the rule: a hosted organization's blob carries all its own sources, so all three are checked at load and the save is refused. This FILE is half of a merge, so only a target the file does not itself declare is left to the runtime — the process logs anERRORper reference it cannot honour, once for each distinct answer, and again whenever that answer changes. A target the file DOES declare is still checked here, type andlog_queriesincluded, andserverefuses to start. - Naming yourself is rejected. Leaving
log_sourceunset already means "my own logs", so a self-reference is a misunderstanding rather than a shorthand. - Any source type may set it (including another
lokisource), and it changes nothing until a cycle turns critical — a firingcriticalalert, or a critical finding from the analysis engine. On a normal cycle no log query runs. - Unset behaves exactly as before: a loki source with
log_queriesstill grounds its own critical findings in its own logs. - It REPLACES this source's own logs, it does not add to them. Setting
log_sourceon a source that can fetch logs itself — alokisource with its ownlog_queries— means its critical cycles are grounded in the named source's lines and no longer in its own. There is one origin per cycle, which is what keeps the "Recent log lines from source …" line in the prompt honest. Set it on a log-capable source only when the other source's logs are the ones worth reading. - Settable from the hosted UI as well (#346). The source-edit form on a
source's detail page carries a log context source picker for every source
type. (The add-source wizard offered the same one until #499 moved the choice
here alone: a source being added has collected nothing, so there is no basis to
choose yet.) It lists your
organization's own log-capable sources — a Loki source with at least one
log_query— and never the source being edited. Sources beyond a hosted plan's source limit are left out too: they are not collected, so their logs could never arrive. Choosing None clears the key. The picker appears only once there is something to pick, so add the Loki source first. Self-hosted keeps editing the file: the wizard's generatedconfig.yamldescribes a single source, so a reference to a sibling could not resolve in it — which is why the snippet now carries a commented# log_source:placeholder naming the key instead. - The prompt (and the
provider: nonereport) names the origin and what the lines are scoped to — "Recent log lines from source app-logs, scoped to {instance=web-1.prod:9100}", or "…, source-wide" when nothing was substituted — so the model is not told another source's logs are this one's, nor that a source-wide sample is the firing instance's. - A missing target degrades, it does not fail. If the named source is not running (it failed to initialize, or a hosted plan's source cap truncated it), the cycle logs a warning and is summarized without log context.
The lines are correlated to the firing alert only where you ask for it (#347). What is captured is the log source's
log_queriesover the cycle's window. A selector carrying a placeholder —%INSTANCE%,%JOB%,%POD%,%LABEL_<name>%— is correlated: it is filled from the labels of the alert that made this source's cycle critical, so its lines are the firing instance's. A selector carrying none is standing and runs source-wide, exactly as everylog_querydid before. The correlated selectors are preferred, not exclusive: they run first, and if they return no lines at all the standing ones run too; if no correlated selector fills, exactly the standing ones run. A standing selector on a large, busy log source still gives you "what this source was logging while the alert fired", not "what the alerting instance was logging" — so either scope the selector yourself, or put the placeholder in it.
Three consequences of that fallback are worth knowing before you rely on it:
- A typo stays silent.
%INSTNACE%is not a recognized token, so it is not a placeholder: it is copied into the query as literal text. A selector whose only token is misspelled is therefore standing and runs source-wide forever — which looks exactly like a correlated selector that simply never narrows. A selector that carries a misspelled token beside a good one is refused whole rather than sent with the typo in it, and the placeholder-free selectors cover that capture. The signal is a warning in the process log, emitted when the Loki source's collector is built (self-hosted: once, at startup), naming the source, thelog_queryand what is wrong with it: an unrecognized token is answered with the vocabulary,%LABEL_%with the form to write, and a token differing from a real one only in letter case (%instance%) with the spelling that would have worked. On the hosted service you cannot read that log — it is the service's, not your organization's — so the add-source wizard says the same three things beside the field when you save the source. It does not block the save. - All correlated and none filling captures nothing at all. If every selector on the log source carries a placeholder and none of them can be filled — the alert does not carry that label, its value is refused by the gate, the selector mixes a good token with a misspelled one, or the cycle is a critical finding rather than a critical alert — then zero queries run. That is not an error (nothing was asked, so nothing failed): the incident is simply summarized with no log lines. Keep at least one placeholder-free selector alongside the correlated ones and that case degrades to a source-wide capture instead of an empty one.
- Healthcheck alerts do not carry
instance/job/pod. A down-endpoint alert is labelledurl,methodandsource(plusstatus_codewhen the endpoint answered), so grounding a healthcheck incident means writing%LABEL_url%—%INSTANCE%would never fill for it, which by the rule above means its correlated selectors never run.
A log query narrows only on a cycle that a critical alert made an incident. A cycle made critical by an engine finding runs the standing queries even when other alerts are firing — none of those alerts is why the cycle is critical, so narrowing to one would ground the report in an unrelated host.
Alerts are correlated by instance first, so N hosts firing the same alert form N
groups of one and the capture is narrowed to just one of them — the host that
fired earliest, which is not necessarily the worst or the most explanatory. On an
incident that is still spreading, that host can change between consecutive reports
as earlier alerts resolve. Whenever the capture returned lines, the log section's
header names the scope it applied, so a report says which host's lines it is
showing; it never claims they are the incident's. (A capture that returned no
lines has no scope to name: if it skipped queries its line says only how many, and
if it skipped none there is no log section at all.)
If a correlated query references a label the firing alert does not carry —
%LABEL_URL% against a url label is the common case, since the token family is
uppercase but the label name inside it is verbatim — the query is skipped for that
capture. Two surfaces say so. The report carries a row stating how many queries
were skipped — on every channel the report reaches and under every LLM provider,
so a hosted
organization reads it without access to the process log, subject to that channel's
own row budget: the row is informational and sorts last, so a report carrying more
signals than the channel shows loses this one first — and the log section of
the summary states it as well, which under provider: none is the same text. The
process log carries a warning naming the source, and that is where the diagnosis
lives: the row is a count, never the query text or the label values. The row
appears on incident reports only, because those are the only captures that ask for
a narrowing; a cycle made critical by an engine finding runs the placeholder-free
queries, which is not a skip. A source that keeps at least one standing
query alongside its correlated ones still asks a query — whether it returns lines
is up to what that selector matches; a source whose queries are all correlated and
none of which fills asks nothing at all, and the report says so.
The privacy note under log_queries applies unchanged: the
captured lines are sent to your configured LLM provider and may be quoted into
the stored report.
LLM
provider: none runs the deterministic analysis engine with no API key and
no external calls; checks: and deploys.enabled are rejected in that mode.
See LLM Providers for the standalone mode details, model
recommendations, cost estimates, and the daily cost guard
(llm.max_calls_per_day).
This block is the OPERATOR's, and in the hosted service it is the operator's for
every organization: an organization's own stored llm: section contributes only
max_calls_per_day, and its per-report model overrides are cleared. See
Who provides the model in the hosted service.
llm.max_fleet_calls_per_day (integer, 0 or absent = no ceiling) caps
billable calls per UTC day across every organization the operator's key
serves, on top of each organization's own max_calls_per_day
(#472). It is
hosted-only and refused at load anywhere else:
llm.max_fleet_calls_per_day is only meaningful in hosted mode
(auth.github.client_id or auth.google.client_id set): self-hosted runs one
summarizer and llm.max_calls_per_day already bounds every billable call —
set that instead
So it belongs in a hosted deployment's config file, beside the auth: block that
turns hosted mode on. It is deliberately not shown as a settable example in
config/config.yaml or in the Helm chart's values.yaml: both are self-hosted
examples and neither renders an auth: section, so a value uncommented there
fails config validate exactly as printed. The chart's ConfigMap template and
JSON schema still carry the key, so a value set anyway is rendered and then
refused loudly rather than dropped silently. A ceiling BELOW the operator's own
max_calls_per_day is accepted, not refused — bounding the two default-org
push-ingress paths harder than the fleet is a real configuration. See
the fleet ceiling for the arithmetic and
Health checks for what a refused check does.
Like max_calls_per_day, the key is decodable from an organization's stored
llm: section — but nothing ever reads it from one, and no tenant write path can
produce a value, so an operator-seeded blob carrying it is preserved and inert.
Notifications
notify:
telegram: # delivery needs BOTH token and chat_id
token: "..." # bot token from @BotFather
chat_id: "..." # group, channel, or user chat ID; empty delivers nothing
# feeds: [digest, trends, incidents, deploys, checks] # optional; omit for all of them
slack: # omit to disable; see docs/notifications.md for the api (threaded) transport
mode: webhook # "webhook" | "api"
webhook_url: "https://hooks.slack.com/services/T00/B00/xxxx" # required when mode is "webhook"
email:
smtp_host: "" # SMTP host; empty disables the channel
smtp_port: 587 # optional; defaults to 587 (STARTTLS)
smtp_user: "" # optional; set with smtp_password for authenticated relays
smtp_password: ""
from: "InfraSigns <[email protected]>"
to: [[email protected]]
pagerduty: # incidents only — see note below
routing_key: '${PAGERDUTY_ROUTING_KEY}' # Events API v2 integration key; empty disables the channel
severity: critical # default for unspecified severity: info|warning|error|critical (default info)
# feeds: [incidents, checks] # optional; these two are the only values this channel takes
Channels are optional and independent — configure any combination. With no
feeds: key a channel receives every notification (digests, trends, incident
alerts, deploy verdicts and health-check transitions), which is the default;
feeds: narrows a single channel to the producers you name, and an explicit
feeds: [] keeps a channel configured but silent. See
Feed routing for the vocabulary and its three
states. PagerDuty is incident-only: a firing incident — an inbound alert
group or a failing health check — triggers a PagerDuty alert and its resolution
resolves it (correlated per episode), while digests, trends and deploy verdicts
are not paged — which is also why its feeds: list accepts only incidents and
checks, and rejects the other three at load. See
Notifications for setup instructions.
Scheduled reports
Both the daily digest and the trends report run on a cron schedule under reports:
reports:
digest:
schedule: "0 8 * * *" # daily health digest
model: "" # optional llm.model override for this cycle (digest + incident summaries)
stale_episode_after: "24h" # an open incident episode nothing has re-asserted for this
# long stops standing the verdict up; defaults to 24h
trends: # optional resource-utilization + trend report
enabled: true
schedule: "0 9 * * 1" # e.g. Mondays at 09:00
window: "168h" # analysis look-back (Go duration); defaults to 7d
step: "1h" # range-query resolution; defaults to 1h
timezone: "UTC" # IANA timezone for the schedule; defaults to UTC
model: "" # optional llm.model override for this report
retention_days: 90 # prune archived reports after N days; 0/absent = keep forever
The digest analyzes a range window derived from its own schedule: the gap
between two consecutive runs, clamped to [1 hour, 7 days], at a resolution
of about 120 points per series (step >= 30s). A daily digest therefore looks
back 24 hours — anomalies and trends that fired and resolved between cycles
still show up. There is no knob: the window follows reports.digest.schedule.
On sub-hourly schedules consecutive windows overlap (the 1h minimum keeps a
useful anomaly baseline), so a short-lived anomaly may appear in more than one
digest.
Each digest (and incident summary) also carries a "What changed" section derived from the same window (no configuration, no extra state):
- New / removed scrape targets — the target set is
upevaluated at the window's start and end; a target present at the end but not the start is new, one present at the start but not the end was removed. A target merely down (up=0) still emits samples, so it is not reported — only a change in service discovery membership counts. Membership is evaluated over a 15-minute lookback at each boundary, so targets scraped as slowly as every 15 minutes (and brief scrape gaps at a boundary) do not read as spurious changes. Detection needs a populated baseline, so a freshly started Prometheus with less than a window of history reports no new targets rather than flagging the whole fleet. - Newly firing alerts — alerts whose start time falls inside the window, as opposed to recurring ones already active before it. (An alert that both fired and resolved within the window is not shown — the digest sees the currently firing set.)
- Threshold crossings — metrics that went from below a configured
warn/critthreshold at the window's start to at/above it at the end. Only queries with awarn/crithint are eligible; the comparison is between the window's first and last samples.
It is fed to the summary alongside the metric snapshot so the analysis can correlate a change — a new exporter, a freshly crossed threshold — with an anomaly. The section is omitted when nothing changed. Because the analysis window follows the schedule, a sub-hourly cadence (whose windows overlap the 1h minimum) may report the same change in several consecutive digests, and a cadence longer than 7 days can miss a change older than the 7d window cap.
Open episodes on the digest
Everything above is derived from the analysis window, which means it can only describe what the window contains. An incident that opened last Tuesday and was never resolved is older than every window a daily digest looks at, so the digest used to say nothing about it — and could legitimately render "All healthy" over a fire that had been burning for three days.
Each digest now carries one row per open episode, with the alert's name, when it started, and how long ago that was:
An open episode is one the inbound alert receiver opened and nothing has resolved. (An episode nothing has re-asserted recently renders differently and does not stand the verdict up — see the last bullet below.) The rows are facts read from the incident archive, not something the model is asked to mention — but the model is told about them, so its prose can correlate a long-running fire with what the metrics are doing and will not write "no issues observed" above a row that says otherwise.
Details worth knowing:
- The verdict answers for them. The report you are sent — Telegram, Slack,
email, the plain-text floor, PagerDuty — can never read "All healthy" while it
carries an episode a sender is still re-asserting, and no open-episode row ever
draws the green healthy accent. Such an episode at critical or warning tier is
counted like any other offending signal ("1 critical"); one whose sender graded
it
severity: info— the case a count cannot see — renders "N still firing" at the warning tier. The tier the sender chose is never overruled; the digest only declines to call it health. An episode nothing has re-asserted lately is the exception, and it is deliberate — see An episode nothing re-asserts is demoted, not hidden below. - The archive says the same thing. Since
#418 a delivered report
RECORDS what its verdict resolved to, and the web archive, the dashboard digest
card, the source-detail timeline and the MCP
get_latest_digest/list_reportstools all read that instead of re-deriving a badge from the stored live alert count — which could never see an episode that opened days ago. A digest that was sent as "1 still firing" is badged DEGRADED · 1 still firing in the archive, andget_latest_digestcarries averdictobject (severity/healthy/unresolved) beside the oldercriticalandalert_countfields. Two consequences worth knowing: reports generated BEFORE the upgrade recorded no verdict, so they render a neutral NO VERDICT RECORDED badge rather than the green HEALTHY they used to (the count behind that verdict was a live read at generation time and cannot be reconstructed, so there is no backfill — green returns for every report generated after the upgrade); and averdictofnullin the MCP output means exactly "this report's verdict is not known" —critical: false, alert_count: 0does not stand in for it. It is an explicitnullrather than an absent key on purpose: see theverdictobject. - Which episodes appear on which source's digest. A digest is per configured source, while episodes arrive by webhook and are matched to a source only best-effort (see Incident attribution). A source's digest therefore carries the episodes attributed to it plus every episode nothing could be attributed to — so no episode is invisible, which is the whole point, while an episode attributed to another source stays on that source's digest. With no attribution rules configured, nothing is attributed and every digest carries every open episode.
- Long lists are cut, oldest kept. Episodes are ranked worst-severity first
and then longest-running, and every cut keeps that prefix — the read itself
asks the database for the 200 longest-running open episodes, the report renders
at most 10 of them, and the prompt sees at most 10. The rendered list is closed
by an "…and N more still open" note ("open", not "firing", because the cut takes
the tail of the list — which is exactly where the demoted rows sort); that note
is a count, not another incident, so it never adds to the verdict's own count. Past 200 the counts
become a floor — a fleet in that state should be reading the incidents page,
not a digest row. Two consequences of ranking by age at the read: at the 200
boundary an old
infoepisode is kept ahead of a newer critical one, and since a demoted episode is an old one by construction, a digest whose read filled that 200 keeps its verdict amber even when every row it managed to read is demoted — the rows it did not reach are the newer ones, and it will not report an open set it could not finish reading as healthy. - It degrades quietly. If the archive cannot be read, the digest ships without the section rather than failing; the failure is logged as a warning.
- An episode nothing re-asserts is demoted, not hidden. A deleted Alertmanager
rule, a sender configured with
send_resolved: false, a curl'd test alert or a decommissioned host leaves an episode open permanently — nothing ever resolves it, and retention prunes only resolved rows. Once nothing has re-asserted such an episode for longer thanreports.digest.stale_episode_after(default 24h), the digest stops standing its verdict on it: the row still appears, reading
but at the info tier, outside the "N still firing" count, and sorted below every episode that is still being re-asserted. A digest whose only open episodes are stale can read "All healthy" again — which is the point: a permanently amber verdict is noise that hides the next real episode. The row never disappears, because nothing resolved it and the report must not claim otherwise.
The clock is the episode's last re-assert, not its start: a repeat firing from the sender refreshes it, so an alert Alertmanager is still repeating never goes stale however long it has been open. Two things have to be true for that refresh to happen, and InfraSigns enforces both rather than assuming them:
- The sender has to repeat at all. Alertmanager and Grafana re-notify an
unchanged group every
repeat_interval. CloudWatch does not — an SNS alarm notification is a state change, so an alarm sitting in ALARM sends nothing more. Itsupdated_atwould freeze at the transition, so CloudWatch episodes are never demoted on this clock, however long they have been open. Nothing can tell an abandoned CloudWatch episode from a burning one, so the remedy for those is the incident page's Mark resolved, not a timer. - The repeat has to survive the deduper. A repeat inside
webhook.dedupe_windowis suppressed and never reaches the archive, so the real refresh period is at least the longer ofdedupe_windowand the sender'srepeat_interval.stale_episode_aftermust therefore be longer thanwebhook.dedupe_window; a config where it is not is rejected at load, with both values in the message.
The 24h default is six missed re-asserts at Alertmanager's own default
repeat_interval of 4h (with the 5m default dedupe window, which is nowhere
near binding), so a receiver outage or a paused sender cannot demote a live
fire. There is deliberately no "off"
value — 0s and negatives are rejected at config load, since a zero would be
indistinguishable from an absent key; to keep the pre-stale_episode_after
behaviour, set something very large ("87600h").
To make the row go away entirely rather than be demoted, close the episode: either make the sender send its resolution, or use the incident page's Mark resolved action. The two halves are deliberately independent — the demotion handles the sender that went quiet without anyone noticing, the manual close handles the alert an operator knows will never resolve. - Cloud: incidents are received process-wide today, so a hosted organization has no incident rows of its own and the section stays empty for it. The read is organization-scoped, so it starts working the moment per-tenant reception does.
Trend detection needs at least 4 data points: for trends,
window / step >= 4 is validated at config load (e.g. window: 1h at the
default 1h step is rejected with a clear error instead of silently reporting
every trend as "stable").
Each report — and the webhook receiver via webhook.model — can override the
global llm.model; see LLM Providers.
Every delivered report lands in the web archive (the reports table).
reports.retention_days prunes rows older than N whole days — the prune runs
at startup and after each persisted report, and deletes the report's feedback
with it. 0 (the default) keeps everything forever; retention is opt-in so an
upgrade never silently deletes history.
Incident episodes from the webhook receiver accumulate in the incidents
table the same way; the sibling incidents.retention_days (top-level) prunes
RESOLVED episodes older than N days, at startup and after each persisted batch
of incidents. Open (still-firing) episodes are never pruned, however old.
The source timeline journal (source_events: fetch failures/recoveries and
check-verdict changes, shown on the web UI's per-source Timeline) has its own
sibling knob, timeline.retention_days — pruned at startup and after each
journaled transition. Only transitions are journaled, so growth is slow; the
default 0 keeps them forever.
The same knob also reaps finished maintenance windows, the other append-only record of something that happened to this deployment. That prune runs at startup only — a window is created by a person, not by a cycle, so there is no write path for it to ride — and it can only remove windows that ended more than N days ago, never one that is still silencing something.
The source_health table holds one last-known health row per source. When a
source is removed from config its collector stops, so that row freezes and its
web UI card becomes a neutral "removed" card. source_health.retention_days
prunes those frozen rows once they are older than N days — pruned at startup and
after each journaled transition, the same path as timeline. A live source's
row keeps updating every collection cycle, so only genuinely-removed sources are
reaped; the default 0 keeps them forever. Independently of this knob, the web
UI hides a removed source's card after 30 days so the sources grid stays bounded
even with retention off.
Since #358 the same knob also governs source_unrecognized_severities, the
per-source reading behind the Unmapped severity labels your sources send card on
Settings — same shape and same prune path, because a live
source rewrites its reading every cycle too. The freeze behaviour is nearly the
same, with one difference worth knowing: a health row advances on every collection
attempt, while a severity reading advances only on a successful one. So a
source that keeps failing for longer than the retention window has its reading
reaped while its health row stays fresh — the card then reports it as never having
reported, which its health card contradicts, and correctly: there is no reading
because there was no successful cycle to take one from. Two more consequences:
lowering this knob deletes the evidence behind that card as well as the health
rows, and — as with the sources grid — the card applies its own view-side bound, so
a source you removed from config stops being listed and stops counting toward its
all-clear regardless of whether you ever set retention.
In the hosted (cloud) service these three per-org knobs — reports, timeline,
and source_health — are additionally capped by your subscription tier (Free
7 days / Solo 90 / Team 365). The effective retention is min(configured
retention_days, tier cap), and the 0=keep-forever default is likewise capped to
the tier, so history is never kept longer than your plan allows. A value below the
cap is honored as-is; raising retention_days above the cap has no effect (the
Billing and Settings pages show the effective, tier-clamped value). Self-hosted has
no plan, so the configured value is used verbatim. incidents.retention_days is
not capped per-org because incidents arrive through the process-global webhook
receiver.
Each schedule is a standard 5-field cron expression. Examples:
| Schedule | Meaning |
|---|---|
0 8 * * * |
Every day at 08:00 |
0 8 * * 1-5 |
Weekdays at 08:00 |
0 8,20 * * * |
Twice a day at 08:00 and 20:00 |
Digest schedules are evaluated in UTC. The trends report uses reports.trends.timezone
(defaults to UTC).
Web UI (experimental)
ui:
enabled: true # serves the embedded web UI under /app; off by default
charts:
enabled: true # source-detail live metric charts (default on when the UI is on)
An embedded dashboard over the same store that backs your digests, incidents,
and reports, plus an add-source wizard that probes Prometheus live and generates
config for you. The wizard also generates config blocks for CloudWatch,
DigitalOcean, Hetzner, healthcheck, and Loki sources (region / resource types /
endpoints / LogQL queries / Loki tenant_id + log_queries; a cloud token is
emitted as a ${…_TOKEN} env placeholder) — but it does not live-verify these: that would require a token
(or arbitrary-URL probing) on this unauthenticated endpoint, so they are
added-and-applied and verify on their first collection. It shares
server.port and, with no sign-in configured, is
unauthenticated and read-only — restrict /app at the network layer.
Configuring sign-in is what makes it writable, and what puts a session in front
of every page; see Where a source lives for the invite an
operator issues themselves. See the Web UI page for the full tour, the
wizard flow, and the security posture.
Each source's detail page draws live metric charts for its configured
queries, fetched on view from that source's Prometheus. They add per-view range
queries to Prometheus (bounded: at most 24 charts × 8 series × ~120 points per
view, 4 concurrent, one 12s budget) — but with no sign-in configured /app is
unauthenticated, so the /series endpoint is a live-query path anyone who can
reach the port can trigger, and that is one more reason to restrict /app at the
network layer. The charts are drawn for provisioned sources only, and the
endpoint answers only a request acting as this deployment's own organization: it
queries the process's own collectors, so it has nothing to say about a hosted
tenant's sources and must not answer about the operator's on their behalf.
Concurrent identical requests (same source and window — a scripted loop or many
viewers on the same page) are coalesced onto a single Prometheus fan-out, so
the cross-request amplification is bounded to one in-flight fetch per
source/window rather than one per caller; this endpoint has no rate limit yet,
and adding one is deferred until /app grows real auth (which changes the threat
model). Read that as the claim it is — it is about /series, not about the
application: the public access-request form does run behind one
(web.md), because that route is anonymous by
design rather than merely unauthenticated today.
Set ui.charts.enabled: false to turn the charts and their /series endpoint
off while keeping the rest of the UI; absent, the charts are on whenever the UI
is.
Sign-in (hosted)
auth:
base_url: https://app.example.com # public origin; fixed paths are appended (no query/fragment/userinfo)
github:
client_id: "Iv1.abc123" # setting EITHER provider turns ON cloud sign-in
client_secret: ${GITHUB_OAUTH_SECRET} # from the environment, never committed
# enterprise_url: https://ghe.example.com # optional; a GitHub Enterprise Server instance
google: # optional; enable one or both providers
client_id: "1234.apps.googleusercontent.com"
client_secret: ${GOOGLE_OAUTH_SECRET}
# session_ttl: 720h # login lifetime (default 30 days)
Setting either auth.github.client_id or auth.google.client_id switches
the deployment into cloud mode: the web UI stops resolving the single
implicit organization and instead resolves the acting organization per
request from a login session, and mounts the sign-in routes (/login,
/auth/{provider}/login, /auth/{provider}/callback, /auth/logout). The
login page shows a button per configured provider. Leaving auth empty (the
default) keeps InfraSigns self-hosted single-tenant — no login, every request
scoped to the default organization.
Cloud sign-in is invite-only, always, with no key to turn it off: an identity completing OAuth is admitted only if it already belongs to a member of some organization or has a live invite for its verified email. The UI creates no organizations at all, so every one of them comes from the command line — see Seeding the first organization below.
- Disabled until a provider's
client_idis set. Cloud mode requiresui.enabled: true(the login flow is served by the web UI) andauth.base_url; each configured provider also requires itsclient_secret(an unconfigured provider is simply omitted, not validated). auth.base_urlmust be https for any non-localhost host: the session cookie is__Host--prefixed and thereforeSecure-only, so a plain-http origin would mint a cookie the browser refuses.http://localhost(in any case, and the root-qualifiedlocalhost.) or a loopback IP literal (including an IPv6 zone such as[::1%25eth0]) is accepted for local development. The name match is ASCII-case-insensitive only, so a look-alike such aslocalhoſt(U+017F) is a different DNS name and is refused; short address literals like127.1are refused too — Go resolves those as DNS names, where a search domain can send the traffic to a remote host.auth.base_urlmust be an origin, optionally with a path prefix. Every consumer appends a fixed path to it — each provider's registered callback is<base_url>/auth/github/callback/<base_url>/auth/google/callback, the members page shows the owner<base_url>/invite/<token>to copy and send to the invited teammate, and Stripe returns to<base_url>/app/billing— so credentials (@), a query string (including a bare trailing?) and a fragment (including a bare trailing#) are rejected byconfig validaterather than silently rewritten. A trailing slash is fine. "Origin" includes a host, andhttps://:8080— a port and nothing else — is rejected for the reason Sources gives for every URL key: every URL built from such a base would have been hostless, so the registered OAuth callback could never match and the invite and Stripe return links named no origin. A wildcard (https://0.0.0.0:8080,https://[::]:8080) is rejected too, for a reason of its own: this value is handed out — to the OAuth provider as a redirect target, to a teammate as an invite link — so it has to name an origin those readers can reach, and a wildcard names only what your own machine listens on. The same applies toauth.github.enterprise_url, whose authorize endpoint a browser navigates to.auth.github.enterprise_urlpoints GitHub sign-in at a GitHub Enterprise Server instance instead of github.com. Leave it unset for github.com. A GHE instance serves the whole flow from one origin, so one base URL is enough: InfraSigns derives the endpoints it requests —<base>/login/oauth/authorize,<base>/login/oauth/access_tokenand the REST API base<base>/api/v3— from it (a trailing slash is fine). Give it the instance origin, not the API base — a value whose path resolves to an/api/v3base is rejected (so/api/v3,/api/v3/and/api/v3/.alike), because/api/v3is appended to derive the REST base. It must be https unless the host is loopback — the same setauth.base_urlaccepts, described above — and it must carry no credentials (@), query string (including a bare trailing?) or fragment (including a bare trailing#) —config validaterejects those rather than silently rewriting the value. When sign-in is enabled, setting it withoutauth.github.client_idis an error: the GitHub provider would never be registered and the key would have no effect. It does not by itself turn cloud mode on — so if no provider client id is set anywhere, the wholeauthblock is off, this key is validated against nothing and is simply unused. What the https rule buys is transit protection and nothing more — the authorize redirect carries the CSRF state, the callback carries the authorization code, and the token exchange carries the client secret and returns the access token, all readable and replayable over plain http on a routed network. It is not a phishing or endpoint-pinning defence: this is an instance-level operator key at the same trust level asauth.github.client_secret,api.tokenanddatabase.dsn— it is read once at process start, is never part of an organization's config, and cannot be set by a tenant, so anyone able to point it at a hostile host could already hand over the client secret. Endpoints are configurable for GitHub Enterprise Server; InfraSigns is not tested against a live GHE instance. There is deliberately no Google equivalent: Google has no on-prem analogue — Workspace, Cloud Identity and GCP all authenticate against the same endpoints regardless of tenant. To confirm the key took effect, check the startup log: thecloud sign-in enabledline reportsgithub_origin, which isgithub.comby default and the resolved instance origin otherwise (a misspelled or mis-nested key is only warned about, so this line is the way to tell it apart from a key that was ignored).- A GHE instance behind a private CA needs its bundle mounted. The token
exchange and both profile requests use Go's default HTTP transport against the
system root pool, and the published image is
gcr.io/distroless/static-debian12— the public Mozilla bundle only. If your instance presents a certificate from an internal CA, mount the CA bundle into the container and point Go at it withSSL_CERT_FILE=/path/to/ca-bundle.crt(orSSL_CERT_DIR); otherwise the first symptom isx509: certificate signed by unknown authorityon the OAuth callback, after the browser round-trip has already succeeded. - GitHub sign-in requires a verified email, on GHE too. InfraSigns takes the
account's primary verified address (falling back to any verified one) and
refuses the login if there is none — an unverified address cannot establish
identity. On github.com that is rarely an obstacle; on a GHE instance it
depends on the instance having outbound email configured, and accounts
provisioned through LDAP or SAML commonly report
verified: false. If every user is refused while the endpoints are demonstrably right, this is the thing to check first. - Sessions are opaque and server-revocable (a row in the database keyed by
the SHA-256 of the random cookie value, so a database read yields nothing that
can be replayed as a live session), not JWTs — logout and ban take effect
immediately.
Expired sessions are reaped in the background; correctness never depends on the
reaper (an expired session is never honored).
session_ttlbounds a login before re-auth (default 30 days). - Sign-in is invite-only. An identity reaching the callback is admitted only when it already belongs to a member of some organization, or has a live (unaccepted, unexpired, unrevoked) invite for its OAuth-verified email. Anything else is refused before any write: a refused sign-in creates no user row, no organization, no subscription and no session. The decision reads the provider-verified profile and never the invite token in the return URL, so a forwarded invite link admits nobody but the address it was issued to. Membership is matched on the provider id — the key the identity itself is stored under — so changing your primary email with GitHub or Google does not lock you out; the verified email is a second match for an account that has not linked that provider yet, and an invite is always matched on the address alone. A user who had an account and has since lost every membership is refused too, in the same words — the page gives no way to tell "no account here" from "an account with nothing attached". Re-inviting that address restores access on their next sign-in; nothing about their history is deleted.
- An admitted newcomer joins the organization that invited them. A first
sign-in no longer creates a personal organization: the invite names the
organization, so the sign-in routes to
/invite/{token}. A user who belongs to several organizations picks the active one on an org-picker (a pending invite to some other organization does not divert them; its own link still works). Deauthorizing a member takes effect on their next request (the acting organization is re-checked against live membership every time), and REMOVING one additionally deletes every session they hold — under invite-only sign-in a live session is a standing admission that would otherwise outlive the membership that justified it, for as long assession_ttl. - An admitted user cannot create an organization. There is no
create-organization page; every organization comes from
infrasigns org createand is enrolled on the plan that command was given (Team unless--plansays otherwise), active, with no expiry. That is what bounds the operator's exposure: the hosted service runs every organization's summaries on the operator's LLM key with the daily allowance counted per organization, and the fleet-wide ceiling (llm.max_fleet_calls_per_day) is optional and off unless the operator sets one — so organizations have to be something the operator issues rather than something an admitted user mints. It is also what keeps admission from becoming permanent — an account removed from its last organization has no way to become a member of one of its own.
Seeding the first organization
No organization comes from the UI — not the first one and not any later one, so
the heading understates it: infrasigns org create is the only way an
organization exists at all. These commands run against database.dsn from
--config, one-shot:
infrasigns org create --name "Acme" # --plan free|solo|team, default team
infrasigns org invite --org acme --email [email protected] --role owner
infrasigns org list # every organization: members, plan, dates
infrasigns org invites --org acme # pending, expired and accepted
infrasigns org invites # every organization's, with an ORG column
infrasigns org revoke --org acme --email [email protected]
infrasigns org member remove --org acme --email [email protected]
org create makes the organization with no members and prints its id and
slug, plus the plan and the daily LLM allowance it will actually run on. The
subscription it writes is active with no expiry — nothing here seeds a trial,
because a trial exists to push a customer at a checkout and this build has none.
An organization created by this command therefore stays on the plan you gave it
until you change it. Its display name is bounded at 60 characters and must not
contain control characters.
--plan defaults to team, and that is deliberate rather than a fallback: Free
is one source, no Slack, no watchdog and a retention pruner that irreversibly
deletes anything older than 7 days, so defaulting to it would quietly hand every
beta participant a crippled install. Pass --plan free or --plan solo when an
organization should be limited. There is no command to change an EXISTING
organization's plan — that is a deliberate SQL UPDATE subscriptions, and
org list is how you check the result.
One organization is NOT created by this command and is always present: the
deployment's own, seeded by a migration with the slug default. It owns the
sources this config file provisions, the incidents the process-level webhook
receiver writes, and the operator's own LLM budget. org invite --org default
is what gives a human a writable UI over it — see
Where a source lives — and it is the only way its FIRST
member gets in, because minting that invite needs a shell on this server. After
that they are an ordinary owner and can invite others from the Members page, at
owner or at member. Treat ownership of default as equivalent to server
access when you grant it: an owner can write this organization's sources; can, in
the add-source wizard, make this server dial an address they type without the
egress gate in front of it — the loopback and private-network reach a self-hosted
operator needs; and is the only role for whom a ${VAR} in a submitted credential
is resolved against this process's environment before that dial. A member
gets none of the three: the write endpoints are owner-only, their wizard probes
dial through the egress gate exactly as a hosted tenant's do, and a ${…} they
submit is left as the literal characters (see the web UI). No hosted tenant can reach this organization on their
own at any role.
org create is not idempotent — running it twice makes two organizations, the
second with a random slug suffix — so if you lose the output, org list is how
you find what was created. It prints one row per organization with its slug, name,
member count, plan, subscription status, plan-end date, creation time and id
(both timestamps in UTC). A member count of 0 means either "just created, nobody invited
yet" or "an orphan from a retry"; those are the same row until you decide. A
trialing status with a date in the PLAN ENDS column is the other row worth
looking for: such an organization drops to Free the moment that date passes, and
the PLAN cell then reads team (now free) — the plan every gate in the product
is actually applying. Rows created by this build never reach that state; rows that
predate it can. A - in PLAN ENDS means the subscription has no period
end, which is what org create writes and what makes the row unable to lapse. A
- in PLAN and STATUS means something else entirely: the organization has
no subscription row at all, and an unrowed organization is treated as unlimited
rather than as Free — only the self-hosted default organization is legitimately in
that state.
There is no org delete: removing an organization cascades to its configuration
and its history, which is a deliberate SQL operation rather than a command a
mistyped slug can aim at the wrong row. org invite prints an absolute
/invite/{token} link — so it requires auth.base_url and refuses without it,
naming the key. Open the link, sign in with GitHub or Google as the invited
address, and accept: you are the owner, and you got in through exactly the path
every later teammate takes.
The seeder deliberately writes no users row. One that did would have to
choose between an unverified row, which account-linking refuses to attach to — so
your first OAuth login would fail permanently, with no UI to repair it — and
asserting verification from a shell, which hands the organization to whoever
controls a typo'd address. A memberless organization plus an owner invite has
neither problem.
org invite is also the recovery path: if every owner of an organization
loses access, it is the only way back in. --role defaults to member; it
accepts owner because the CLI's gate is the shell (the database DSN and a login
on the host), where the browser has no owner-granting path at all. Re-running it
for a still-pending address refreshes the token and the expiry, which is what
makes "the invite expired, run it again" safe. Invites last 7 days.
Undoing an invite, and un-admitting a member
A typo in org invite --email grants admission to whoever controls the mistyped
address, and with --role owner it grants ownership. Nothing is emailed, so the
mistake is invisible — and possession of the address is what admits, not
possession of the link: the sign-in gate matches the invited address against the
one your provider verified and hands the invite to the login itself, so a
forwarded link admits nobody else and an unforwarded one still admits the address.
Re-read the address before pressing return, and prefer --role member for anyone
who does not have to be an owner.
org invites shows what an organization has issued, one row per invite with
its address, role, state and who issued it. Accepted invites are listed too,
and that is the point: the row leaves the pending world at the exact moment
somebody uses it, so this is what answers did anybody get in on my typo?. The
STATUS column says pending, expired or accepted — an expired row admits
nobody but is still worth clearing, and an accepted one is spent, so its remedy is
a removal rather than a revoke. EXPIRES and ACCEPTED are separate columns
and exactly one of them is filled per row; the other reads -. An invite the
operator minted shows - under INVITED BY — it has no inviting user, by
design.
Without --org it lists every organization's invites, adding an ORG
column. That is the only fleet-wide view of invites there is — every other
surface, the members page included, is scoped to one organization.
org revoke --org <ref> --email <addr> deletes a pending invite, and it is
the address you typed that addresses it, not an id you never saw:
infrasigns org revoke --org acme --email [email protected]
It refuses, with a non-zero exit, when there is nothing pending — and it
distinguishes the two reasons, because the remedies are opposite. If the address
already accepted, the person is in the organization and has to be removed. If
there is no invite at all, the address is worth re-reading against org invites.
Deleting the row is only half of it, because the gate admits on the address and
creates the session before any acceptance: a stranger at a mistyped address can be
signed in for the full auth.session_ttl while holding no membership at all. So a
successful revoke also signs out that address's live sessions — but only when
all three of these hold: the row you revoked was still live (an expired one
admitted nobody), the address holds no membership in any organization, and no
other live invite names it anywhere. The gate admits on the address with no
organization scope, so a live invite in some other organization justifies the
session exactly as a membership does; a revoke here must not take that away. The
command reports how many went, and says nothing when there were none — which can
mean nobody was signed in, something else still admits them, or the row had
already lapsed.
Revoking is decisive only while nothing else admits the address — the gate takes
membership in any organization or any live invite naming it. The session
eviction asks that same question and one more (was the row you revoked still
live), so a revoke that reports no sessions is often telling you that something
else still admits the address; org invites with no --org is how to see what.
org member remove --org <ref> --email <addr> removes a member, including an
organization's last owner. The web control refuses that removal — an
organization must keep an owner — which is right between members and is a trap for
the shape this page produces: org create plus one owner invite, i.e. every
organization on its first day, where the only member is the person you might need
to un-admit. The operator is not acting as a member, so this path is allowed to
empty the organization; the browser still cannot.
infrasigns org member remove --org acme --email [email protected]
It prints what it removed and what that cost. Three things are worth knowing before you run it:
- Every session that person holds is deleted, in every organization. A session is not organization-scoped, so leaving the others alive would leave the eviction one click away from being undone. Somebody who legitimately belongs to other organizations signs in again and lands where they still belong.
- Removing the last member leaves the organization memberless. No page in the
product lists it:
org listis where it stays visible from the shell, andorg inviteis the way back in, exactly as at creation. - Removal un-admits only somebody left belonging to no organization at all, and even then only while no live invite names their address: the gate admits on membership or on an invite. The command says which of those hold; it does not claim an un-admission it cannot see.
Invites the removed person had issued for that organization are revoked with them
— an invite must not keep granting membership once the authority behind it is
gone. Invites issued to them are not, so revoke those too if the point is to
keep them out. org invites with no --org is what makes that followable: it
lists every organization's invites in one table, so you can find the ones naming
that address without walking org list by hand, and clear each with
org revoke --org <ref> --email <addr>.
What the seeding path still cannot undo
- A delivered invite link cannot be un-sent. Revoking kills the admission, not the message; the 7-day TTL is what bounds a typo nobody notices.
- There is no
org delete— see above. An organization emptied byorg member removestill exists, on its plan, with its history. -
A plan is still changed only by SQL.
--plandecides it at creation andorg listis how you check it. -
Accounts are linked by verified email. Signing in with GitHub and later with Google (or vice versa) using the same verified email resolves to one account, not two — the second provider is attached to the existing identity.
The requested scopes are minimal: GitHub read:user + user:email, Google
openid + email + profile — the verified email (used as the identity and
billing address) plus display name and avatar, no repository, Drive, or Gmail
access. Helm support for the hosted deployment lands with the cloud rollout; the
published chart remains self-hosted single-tenant.
Credential encryption (security)
security:
encryption_key: ${INFRASIGNS_ENCRYPTION_KEY} # 32 bytes, standard base64
# encryption_key_file: /etc/infrasigns/keys/root # instead of the above — never both
# previous_encryption_key: # keys older rows are still wrapped under
# - ${INFRASIGNS_ENCRYPTION_KEY_PREVIOUS}
An organization's whole configuration lives in one org_configs row, and that
row is where every credential it has stored ends up: source tokens, basic-auth
passwords, OAuth2 client secrets and private keys, TLS client keys, SigV4 keys,
DigitalOcean and Hetzner API tokens, a
health probe's per-endpoint credential,
Telegram and Slack tokens, the SMTP password, the PagerDuty routing key, the
dead-man's-switch ping URL. The list is an illustration, not the source of truth
— the set is whatever carries the secret:"true" classification in
internal/config (grep -rn 'secret:"true"' internal/config/), and the seal
and the error redactor both read it there. Set a key
and each of those values is encrypted before it is written, so a database
read, a replica, or a pg_dump yields ciphertext instead of credentials.
It is required as soon as sign-in is on. A deployment with auth configured
holds other people's credentials, so infrasigns serve refuses to start
without a key and says which setting is missing. A single-operator install with
no sign-in holds only its own, and is not required to set one — but may, and
gets the same protection if it does. Encryption engages on the key's presence;
the requirement is what is gated on sign-in.
Generate one with infrasigns secrets keygen. The key goes to stdout and
every word of guidance to stderr, so infrasigns secrets keygen > root.key
writes a file usable directly as encryption_key_file. Use
encryption_key_file on Kubernetes, where the External Secrets Operator
projects a secret as a mounted file — see
Kubernetes. Setting both forms is a configuration error, not a
silent precedence rule.
What is encrypted, and what is not
Only the credential fields are. Source names, URLs, queries, schedules, check definitions and every other part of the document stay readable, which is deliberate: an operator can still inspect and query a stored configuration, and the pre-flight SQL this documentation gives elsewhere keeps working.
A URL is left readable, so a stored URL may no longer carry a credential in
it — where there is somewhere sealed to put it instead.
https://user:[email protected] puts a live credential in a field
nothing encrypts, and the product has always treated userinfo as one — it is why
such a URL is required to be https. Two fields are therefore refused on save:
sources[].url— the remedy is thebasic_authblock, which is curated, validated and encrypted.sources[].oauth2.token_url— and here the remedy splits by grant. Onclient_credentialsit is that block's ownclient_id/client_secret, which is exactly the pair userinfo was carrying, moved into a field that is encrypted — a relocation. Onjwt-bearerthere is nowhere to relocate to: the signed assertion is the authentication,client_secretis refused outright there, and the remedy is to drop the userinfo. So for that grant this refusal removes a capability rather than moving it, and we would rather say so.
The config file is no longer exempt from either, and that changed with
#546. A deployment with
sign-in configured copies its file's sources: into the database at start-up, so
both rules now bind on the file too — and there they are fatal at boot rather than
a refused save. See
Pre-flight before an upgrade for the command that
finds them first. A deployment with no sign-in stores nothing, so its file is
still unaffected.
sources[].endpoints[].url— added by #545, which is what gave it a remedy. A healthcheck endpoint had no credential block at all, so refusing userinfo would have removed the only way to authenticate a probe; it now carries its ownbasic_authorauthorization, encrypted like every other credential, and the refusal points at it. See Authenticating a health probe.
One carrier is still stored in the clear, and only in an endpoint URL: a
credential in the QUERY. https://probe.example.com/healthz?ping_key=SECRET is
the standard shape for a status endpoint, and no credential block can carry a
query parameter — it is part of the request the endpoint expects, not a header —
so refusing it would remove a capability with nowhere to move it to. That value
does reach org_configs in the clear. What is protected is how it renders:
every display of an endpoint URL is rebuilt from scheme, host and path, so
userinfo, query and fragment are all dropped. A source url and an
oauth2.token_url are not affected — a query string is refused outright on both.
#546 widened who all of this reaches, so it is worth restating rather than
leaving to inference: before it, only a hosted tenant could put such a URL in the
database. Now a self-hosted operator with sign-in on puts one there too, out of
their own config file, on every start.
The three refusals bind on writes only, so an organization that stored such a URL before the rule keeps running and can repair it in the editor. On a deployment with sign-in they also bind on the config FILE at start-up, which is why the pre-flight command exists — run it before you upgrade. To find them in the database:
SELECT org_id, s->>'name' AS source, 'url' AS field
FROM org_configs
CROSS JOIN LATERAL jsonb_array_elements(COALESCE(config->'sources','[]'::jsonb)) AS s
WHERE s->>'url' LIKE '%@%'
UNION ALL
SELECT org_id, s->>'name', 'oauth2.token_url'
FROM org_configs
CROSS JOIN LATERAL jsonb_array_elements(COALESCE(config->'sources','[]'::jsonb)) AS s
WHERE s->'oauth2'->>'token_url' LIKE '%@%'
UNION ALL
SELECT org_id, s->>'name', 'endpoints[].url'
FROM org_configs
CROSS JOIN LATERAL jsonb_array_elements(COALESCE(config->'sources','[]'::jsonb)) AS s
CROSS JOIN LATERAL jsonb_array_elements(COALESCE(s->'endpoints','[]'::jsonb)) AS e
WHERE e->>'url' LIKE '%@%'
ORDER BY 1, 3;
Run it as a superuser or a BYPASSRLS role. org_configs carries forced row
level security with a restrictive policy, so the ordinary owner DSN this
deployment connects with returns zero rows for every organization — not an
error, just an empty result that reads exactly like a clean bill. (Measured, not
assumed: as the owner with no app.org_id set, SELECT count(*) FROM org_configs
is 0; as a superuser the same query finds the row.) The alternative, if you have
no superuser to hand, is to set app.org_id and run it once per organization.
The query over-matches — an @ anywhere in the path hits it — so check each
result by eye. It reads the URL in the clear precisely because that field is not
sealed, which is the point.
A sealed value is self-describing — it records which data key sealed it — so a document may hold sealed and unsealed values at once, and no separate flag says which is which. That is what lets the one-time sweep below run without a maintenance window.
The first start after upgrading
On the first start with a key configured, InfraSigns sweeps every organization
and seals whatever is still in the clear. It is idempotent, it does not count as
an edit (a configuration's version is untouched, so nothing is invalidated and no
editor loses a concurrent save), and it reports what it did. You can run the same
pass by hand with infrasigns secrets seal.
Losing the key loses the secrets
There is no recovery path. Without the key the stored credentials cannot be opened by us or by anyone else, and every tenant has to enter their tokens again. So keep a copy somewhere other than the machine — and, specifically, not in the same archive as the database backup. A backup that contains both the ciphertext and the key it opens is a backup with no encryption at all.
Rotating
Two different operations, and only one of them needs a restart.
A single organization's data key rotates live, no restart:
That is key hygiene — it bounds how much ciphertext one data key protects and moves an organization onto fresh material. It is not a remedy for a tenant credential that may have been exposed, and it is worth being blunt about why, because the command reads like one. A data key exists only wrapped in the database and in the memory of a process that also holds the root key. So either whoever read it held both — in which case they already have the plaintext, and the retired key is deliberately kept under that same root key so the ciphertext they copied still opens — or they held neither and never had plaintext at all. Rotating takes nothing from anybody.
If a tenant's credentials may have been exposed, re-issue them where they were issued: revoke and reissue the Prometheus bearer token, the Slack webhook, the SMTP password, the PagerDuty routing key, the LLM key, and save the new values. That is what actually invalidates the old ones. Saving re-seals them here as a side effect.
The root key takes exactly one restart, and it can ride an ordinary deploy:
- Put the new key in
security.encryption_keyand move the old one intosecurity.previous_encryption_key. - Restart. The process now holds both, and each stored data key records which root key wrapped it, so nothing is guessed.
infrasigns secrets rewrap. This rewraps the data keys only — no credential is re-encrypted, so the work is one row per organization and finishes in seconds.- Drop the old key from the configuration at your next deploy.
previous_encryption_key is a list because a rewrap is a background job and a
second rotation can legitimately start before the first has finished. Two
identical entries are refused rather than ignored: pasting one key into both
settings would otherwise look like a rotation that never happened.
Turning it off, and rolling back
infrasigns secrets unseal --write-cleartext writes every stored credential back
in the clear. It is the exit ramp, and it is a required step before some kinds of
rollback. Two rules govern it, and getting either wrong is expensive.
Run it while the key is still configured. Only a process that can decrypt the values can write them out. Remove the key first and the ramp stops working along with everything else.
The ramp survives a restart, but not a save. unseal marks the deployment for
cleartext before it walks, so serve stops sealing at boot and a pod restart no
longer quietly undoes what you just did — and a boot that was already halfway
through sealing the fleet when you typed the command re-reads that mark before
each organization and stops there, rather than racing you to the end. What a running daemon still does is seal each
organization that is saved while you are on the ramp — so either stop the
daemon, or re-run unseal immediately before the rollback step.
infrasigns secrets seal is the way back, and while you are on the ramp every
boot says so at WARN.
Rolling the SCHEMA back past the migration that creates the data-key table
destroys the ability to decrypt anything still encrypted — the keys go with the
table. So: unseal first, then migrate down.
Rolling the BINARY back — to a version from before encryption existed — has
its own trap, and it is the one that bites during an incident. An older process
has no notion of a sealed value: it reads enc:v1:… as the credential itself and
sends it as a bearer token, a Slack token, an SMTP password. Sources start
answering 401, health checks flip to failing, tenants are paged about services
that are fine, and the delivery of those pages fails too. Worse, it looks like a
tenant-specific fault rather than a deployment-wide one, because only the
organizations whose runtime happens to be rebuilt in that window break.
So a rollback of the image is unseal first, then roll. If you have already
rolled back and are seeing that pattern, roll forward, unseal, and roll back
again — the values are recoverable as long as the key and the data-key table are
both still there.
What this does not defend against
- Anyone holding both the process environment and the database. The key is in the environment by design, so an attacker with both reads everything. This protects the database, the replicas and the backups — not the host.
- Backups taken before a tenant was deleted. Deleting an organization destroys its data key, so backups taken afterwards cannot be opened for it. A dump taken earlier contains both the ciphertext and the wrapped key, and the root key is in the environment, so it stays readable.
- A running process's memory. Everything above the storage layer works with plaintext; the encryption is at rest.
Billing (hosted)
Stripe billing turns the plan limits (Free / Solo / Team) into a paid product on the hosted service. It is cloud-only and requires sign-in (above) — every checkout and webhook is scoped to an organization.
billing:
secret_key: ${STRIPE_SECRET_KEY} # Stripe secret API key (sk_…); its presence turns billing ON
webhook_secret: ${STRIPE_WEBHOOK_SECRET} # verifies the /billing/webhook signature (whsec_…)
price_solo: ${STRIPE_PRICE_SOLO} # Stripe Price ID (price_…) the Solo plan maps to
price_team: ${STRIPE_PRICE_TEAM} # Stripe Price ID the Team plan maps to
Setting billing.secret_key switches on the paid boundary: the /app/billing
page gains Subscribe / Manage actions (owner-only), POST /billing/webhook starts
verifying Stripe signatures, and the source/member/Slack quota gates fail closed on
a plan read error (a read failure must never hand out a higher tier's limits once
money is involved). Leaving billing empty keeps the read-only billing page and
the fail-open gates — every org stays on whatever plan infrasigns org create
enrolled it on (or whatever SQL set afterwards), with no way to change it from
the browser.
- Requires sign-in and at least one price. Billing validation fails closed
unless
authis enabled (it needs the session/org spine),billing.webhook_secretis set (an unverified webhook could forge plan state), and at least one ofprice_solo/price_teamis configured. - Secrets come from the environment, never the config file:
secret_keyandwebhook_secretare${ENV}references. The price IDs are not secret but are referenced the same way for parity. - The webhook is unauthenticated by design — Stripe calls it, and the only
trust boundary is the HMAC signature over the raw body (verified with
webhook_secret). The route is mounted whenever theeebuild is running, configured or not: an unconfigured one answers the same400 signature verification faileda bad signature gets, so its presence tells an anonymous caller nothing about whether this deployment sells anything (#558). On the public image, which carries no billing client, it is absent regardless — the same build-not-configuration distinction the/pricingrow above draws. Point your Stripe webhook endpoint at<base_url>/billing/webhookand subscribe it tocheckout.session.completed,customer.subscription.updated,customer.subscription.deleted, andinvoice.payment_failed. - Downgrades keep the config, freeze premium capabilities. A cancelled subscription drops the plan to Free and a failed payment marks it past-due (access kept through Stripe's dunning). Your configuration is never rewritten, but on the lower plan the worker enforces the plan live: it stops delivering Slack (Free has no Slack) and stops monitoring any sources beyond the plan's source limit (the first N by config order are kept; the rest show as over-limit on the health page, uncollected). Over-cap members are retained (a downgrade never removes a teammate). History retention is also clamped to the lower tier's cap — the one data-destructive downgrade effect: existing report/timeline/source-health rows beyond the new tier's window are reaped on the next prune (re-upgrading widens the window again but cannot restore already-deleted rows). None of this rewrites the blob — re-upgrading resumes delivery and monitoring with no reconfiguration. The same rule fires when a Team trial lapses to Free (there is no Stripe event for that — it is resolved live at delivery time), so a trial that quietly ends stops delivering Slack too.
Like sign-in, billing is not wired into the published Helm chart — it lands with the hosted cloud rollout; the self-hosted chart never sets it.
AWS identity (hosted)
aws:
# The IAM principal this InfraSigns deployment's own AWS credentials resolve to.
worker_principal_arn: arn:aws:iam::123456789012:role/infrasigns-worker
This is the service's own AWS identity, not a tenant's. It is the value an
organization must name as Principal.AWS in the trust policy of the role they
grant you, so the worker may call sts:AssumeRole against it — the far end of the
cross-account setup a CloudWatch source describes.
It is what makes hosted CloudWatch available at all. Leave it unset and a
signed-in organization sees the CloudWatch tile in the add-source wizard offered
disabled, with a sentence saying the service has no AWS identity of its own,
and the save endpoint refuses a cloudwatch source. That is deliberate: without a
principal there is nothing for a tenant's trust policy to trust, so the setup
cannot be completed, and a stored source that can never work would still consume
one of the organization's plan source slots and fail on every collection cycle.
The tile stays on screen rather than disappearing, so its absence cannot be read
as "InfraSigns does not support CloudWatch".
Before you set this
Setting the key turns the capability on for every organization, and the key is
only one half of a cross-account assume. The other half is yours and nothing
in InfraSigns can check it, so a deployment that sets the key and stops here looks
configured and collects nothing: every tenant completes the wizard, saves a
perfectly correct trust policy, and every collection returns AccessDenied.
- Hold credentials that resolve to the principal you name. The key is a declaration, not a probe — see below.
- Give that principal an identity policy allowing
sts:AssumeRoleon the tenant roles it will assume. A trust policy on the tenant's side grants nothing on yours; without this grant every collection fails withAccessDeniedno matter how correct the tenant's policy is. How to scope that grant is a real decision with a trade-off — an enumerated list of customer role ARNs does not survive self-service — and it is worked through in AWS CloudWatch, under Operator setup and Operator hardening, together with the hosting-account confused-deputy it interacts with. Read those two paragraphs before you write the policy. -
Keep the role. IAM resolves a role-ARN principal to that role's unique id when a tenant SAVES their trust policy, so deleting and recreating the worker role under the same name silently invalidates every policy already stored.
-
Self-hosted ignores it, but still validates it. A single-tenant deployment reads AWS through the process's own ambient credential chain (or a
role_arnper source), assumes no cross-account role on a tenant's behalf, and therefore needs no declaration: nothing there consults the value. The SHAPE check below is not skipped, though — it runs wherever a config FILE is loaded — so a malformed value fails startup on a self-hosted deployment exactly as it does on a hosted one. A well-formed one is silently inert, which is what the startup line below is for. - It says so at startup. When the key is set, the process logs one
INFOline naming the principal that took effect: whether hosted organizations can now add CloudWatch sources, or that this is a self-hosted deployment, which reads the value nowhere. Nothing is refused — the line exists so a validated-inert key is distinguishable from a working one, which is otherwise the same silence. - It is a declaration, not a proof. Setting it does not verify that this
deployment actually holds credentials for that principal, and nothing probes the
credential chain to find out. An explicit key was chosen over a probe precisely
so the page states the operator's intent rather than guessing from a credential
chain that can answer differently at page-render time than at collection time.
A wrong value fails on the TENANT's side, and early: the commonest typo — a
digit off in the account id — names a principal that does not exist, and IAM
refuses to save a trust policy naming one (
Invalid principal in policy). That follows from how IAM stores these: a role or user ARN in aPrincipalis transformed to that identity's unique id when the policy is saved, so an ARN it cannot resolve has nothing to store. The tenant therefore hits the error while creating their role, with nothing on screen pointing at the operator's key, and it is worth checking this value first when a hosted organization reports that AWS will not accept the policy the wizard gave them. That promise covers the WHOLE accept set, and only because the set is roles and users: IAM resolves both to a unique id, while it does not resolve an account to one — which is a large part of why an account principal is refused below. A value that names a principal which EXISTS but is not the one the worker runs as fails later instead, at collection, withAccessDenied. - Validated at load, structurally, and much narrower than AWS. Accepted: two
forms only —
arn:<partition>:iam::<12-digit account>:role/<name>and…:user/<name>, paths allowed (role/team/name), across every partition (aws,aws-us-gov,aws-cn, …), with no region field. The question this key answers is which identity does the worker run as, not what would a trust policy accept, so three classes of legal value are refused on purpose:- an AWS account — the bare 12-digit id AWS documents as shorthand, or
…:iam::<account>:root. It is the shortest answer and it is also the widest. In AWS's own words an account principal "delegates authority to the account", after which "an administrator in that account must then grant access to an identity" — so the blast radius is every principal in your hosting account that your own IAM lets callsts:AssumeRole, and the broad grant self-service signup needs (see Operator hardening, above) is exactly that permission. The external ID does not compensate; it is the organization id, which the tenant reads off their own screen. - an STS session ARN (
arn:aws:sts::…:assumed-role/<role>/<session>, and…:federated-user/<name>). This is the trap worth knowing about, because it is whataws sts get-caller-identityprints on the worker — the obvious answer to "which principal do I run as". It names ONE session rather than the identity behind it, so a policy naming it can work for a while and then stop without warning — session names are chosen by whoever assumes the role, and AWS's own guidance is to name the role rather than a session "wherever possible". Name the durable identity instead: the IAM role for an assumed-role session, and for a federated-user session the IAM user whose long-lived credentials calledGetFederationToken, since that session "is based on the original identity that was federated". - any other IAM resource kind, and a kind with no name after it (
role/, or a doubled separator likerole//worker).group/,policy/andinstance-profile/are not identities.saml-provider/andoidc-provider/are principals — underPrincipal.Federated, never under thePrincipal.AWSthis value fills — so they are refused here without that being a claim that a federation ARN is never a principal.
- an AWS account — the bare 12-digit id AWS documents as shorthand, or
The value may also contain only letters, digits and : / _ + = , . @ -; anything
else is refused, because the string is printed verbatim into the JSON trust
policy the wizard offers for copying.
- Operator-only. Like auth and billing it is process-global: an
organization's own config document may not carry an aws: section, and a blob
that does is rejected.
- Already-stored sources are untouched. Clearing the key does not retire an
organization's existing CloudWatch source — the worker keeps building a
collector for a source with a role_arn (and keeps skipping a role-less one),
so the tenant sees a failing source they can act on rather than one that
silently vanished. Only adding is gated.
Like sign-in and billing, it is not wired into the published Helm chart — that chart renders no hosted sections at all.
Q&A bot
bot:
telegram: # getUpdates long-poll
allowed_chat_ids: [123456789] # non-empty enables this transport (fail-closed allowlist)
# token: "" # falls back to notify.telegram.token when empty
slack: # Socket Mode; operators @-mention the bot
allowed_channel_ids: [C0123ABCD]
app_token: "xapp-..." # required
# bot_token: "" # falls back to notify.slack.token when empty
mcp_servers: # external MCP tool servers the agent consumes (operator surface)
- name: runbooks # [A-Za-z0-9_-], ≤40; prefixes every tool as <name>__<tool>
url: https://runbooks.internal/mcp # MCP HTTP (streamable) endpoint
# token: "${RUNBOOKS_TOKEN}" # optional bearer; https required off localhost
tools: [list_runbooks, get_runbook] # per-server allowlist of tool names to expose
An LLM agent that answers infrastructure questions in Telegram and/or Slack
using the same read-only tools as the MCP server. Each transport is enabled by
its own fail-closed allowlist. Requires a reasoning llm.provider
(openai/anthropic); one question costs up to 9 calls against the shared
llm.max_calls_per_day. See the Q&A bot page for the security model
and operational notes.
bot.mcp_servers lets the agent consume read-only tools from external MCP
servers (a runbook server, a cloud provider's endpoint), namespaced as
<name>__<tool> and gated by a per-server tools allowlist. Validated whenever
present, but only consumed when a Q&A transport is enabled. Like the rest of the
AI surface it runs in both modes and is an operator feature rather than a
self-hosted one: the servers are named in this process configuration and the agent
answers about this deployment's own organization, so a signed-in tenant can
neither add a server nor reach a tool one exposes. See
External MCP tool servers.
API authentication
POST /api/digest/trigger runs a digest cycle on demand — a billable LLM call
plus user-visible notifications — so it requires a bearer token and is
disabled until one is configured:
api:
token: '${API_TOKEN}' # min 16 characters; empty disables the trigger endpoint, /api/maintenance, /mcp and GET /sources
Requests must send Authorization: Bearer <token>. The bundled
.env.example ships a dev-only token so the compose quickstart works out of
the box — replace it for anything reachable beyond localhost. The
infrasigns digest trigger CLI picks the token up from $API_TOKEN (or
--token).
Every surface this token gates — the digest trigger, the maintenance-window API
and the MCP /mcp endpoint — runs against the process configuration and the
default organization, and since
#537 all three are
mounted whether or not sign-in is configured. The token is the whole gate:
none of them reads a session, a cookie or a membership, so a request without a
matching bearer is answered 401 and one with it is being trusted as the
operator. What they act on does not change with the mode either — always this
deployment's own organization, never a signed-in tenant's, whose digests run
automatically per organization.
On a deployment that serves other people that makes api.token an operator
credential in a room with tenants in it: anyone you give it to can run this
organization's digest and silence its alerts, so treat it the way you treat
membership of default.
The endpoints that stay unauthenticated are the platform-probe set —
/healthz, /readyz and /metrics: they expose operational health, not data,
and Kubernetes and load balancers need them open. Restrict them at the network
layer if required.
/sources used to be in that list and no longer is. It answers with every
source name, its up/down state, its last collector error and its scrape
timestamps, which is an inventory of the estate you monitor rather than a
health signal, so it now takes the same api.token bearer as the three routes
above and answers the same 401 whether the key is unset or the bearer is
wrong. If you scrape it, add the credential; if you had no api.token set, it
now answers 401 to everyone, and serve logs a line saying so at start-up.
Upgrading from a version without api.token: the trigger endpoint was
previously unauthenticated and is now disabled until a token is configured.
Add API_TOKEN=<16+ chars> to your existing .env (compose) or secret
(Kubernetes) and, for automation calling the endpoint directly, send
Authorization: Bearer <token>.
Maintenance windows
Planned work produces alerts you already know about. A maintenance window silences their delivery for a bounded period, without touching what gets recorded: incidents, health-check state and reports are written throughout, so the post-mortem still has everything that happened. Only the notification is withheld.
This is what an Alertmanager silence cannot do. That silence covers inbound alerts only, so a planned reboot still pages through the health checks, the deploy verification and the source's own daily digest.
# Silence one source for two hours
curl -X POST http://localhost:8080/api/maintenance \
-H "Authorization: Bearer $API_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"source":"prod-metrics","reason":"kernel upgrade","duration":"2h"}'
# What is muted right now, and how much it has swallowed
curl -H "Authorization: Bearer $API_TOKEN" http://localhost:8080/api/maintenance
# Done early
curl -X POST -H "Authorization: Bearer $API_TOKEN" \
http://localhost:8080/api/maintenance/<id>/end
Scope. A window covers either one configured source ("source":"prod-metrics")
or the whole organization (omit source). What each scope reaches:
| Notification | Covered by a source window | Covered by an org-wide window |
|---|---|---|
| That source's daily digest | yes | yes |
| Inbound alerts attributed to that source | yes | yes |
| A health check running against that source | yes | yes |
| A deploy verdict verified against that source | yes | yes |
| The weekly trends report | no — it spans every source | yes |
A deploy verdict is covered by a window on the source the verification queries
(deploys.source), not by one named after the deployed service — the two are
different namespaces and a window scopes to the first.
An inbound alert batch is covered only when every alert in it attributes to
the window's source (via that source's alert_labels). A batch mixing two
sources, or carrying an alert nobody claimed, is delivered — a window on one
source must not silence another's alerts.
A source with no
alert_labelsattributes nothing, so its inbound alerts are reachable only by an org-wide window. If you want a source window to cover the alerts your fleet sends, give that source analert_labelsmatcher; until then the digest and its health checks are covered but the alerts are not.
A window covers the notifications sent while it is open, one at a time — it carries no memory of an episode. Combined with the resolution rule below, that means an alert firing inside a window and clearing at any point delivers its "resolved" but never its "firing": the message is worth having, though it can read as a resolution for something you never saw fire.
A resolution is never withheld. A window suppresses the message that says something started being wrong; it never suppresses the one that says it stopped. That message is what closes a page — a PagerDuty resolve is the only event that clears a triggered incident — so swallowing it would leave an alert that fired before the window paging someone indefinitely, with nothing to retry it. The cost is that an alert which both fires and clears inside a window still delivers its "resolved", with no "firing" before it.
A manual digest trigger is silenced too. POST /api/digest/trigger runs the
cycle and archives the report, and its response is the usual {"status":"done"},
but the notification is withheld like any other — the window you opened applies
to the digest you asked for. End the window first if you want the delivery.
A window must name a source you have configured. A typo (prd for prod) is
refused at creation rather than becoming a window that silences nothing while
reading exactly like one that works.
Duration is required, between one minute and 30 days. There is no open-ended window: that is how a fleet goes dark permanently. Re-open the window if the work runs long.
An organization may hold about 50 windows at once. Past that, opening
another is refused ("end one before opening another") until one lapses or is
ended. It is a fixed bound, not a setting: the list surfaces show at most 100
windows, and a limit an operator could raise above that would let an active
window fall off every list and become impossible to see or end. "About" is
literal — the check and the insert are one statement but not a lock, so two
requests racing for the last slot both succeed; the margin to 100 is what makes
that overshoot harmless. How far over it can go is bounded by the database
connection pool, since each request holds a connection for the length of its
insert: one InfraSigns process opens at most 10, so the worst case is 60 and the
margin swallows it. That arithmetic is per process — run several replicas
against one database and it becomes 50 plus ten per replica, which is worth
knowing before you scale out. Nothing legitimate reaches this bound — real work needs
one window per concurrent maintenance — so the usual cause is a script reopening
a window it never sees the answer to. The API answers this refusal with 409
Conflict, not 400 — the posted request is fine as it stands, and the same
one succeeds the moment any window in the organization ends, which is a
conflict with the organization's state rather than with the request's shape.
A capped list can never hide a window that is still silencing something — as
long as the organization's active set stays under the ~50-window cap. The
list and dedup checks above (the Notifications page, GET /api/maintenance,
and /mute//muteall's own duplicate check) all read the same capped set, and
that set puts every window still running ahead of any that has already lapsed
or ended — so with the ~50-window cap comfortably under the 100-row list limit,
an active window can never be the one a cap drops, while that margin holds.
What a truncated read loses in that ordinary case is only older history: a
window that already lapsed or was ended more than 24 hours ago.
The margin does not hold for an organization that was already carrying more
active windows than the cap the moment the cap shipped — a script that opened
hundreds before any limit existed is exactly this product's own history. For
that organization, and only that organization, a capped list genuinely CAN
drop a window that is still silencing something, and no reader of the capped
list can enumerate or end it. Every surface that renders a claim about what is
covered detects this and says so instead of repeating the sentence above: the
Notifications page and the dashboard note both switch to an explicit "more
than this page can show" wording, /windows prints its own version, and
GET /api/maintenance carries a second field, active_truncated, alongside
truncated — a script must read that field to tell "only history is missing"
from "some of what is silencing you right now is missing too" (truncated
alone answered true for both before this field existed). /mute's and
/muteall's own duplicate check is the one reader of the capped set that does
not detect this: activeWindowFor (internal/bot/command.go) discards both
truncation flags and falls back to the same best-effort answer it already
gives for an ordinary truncated read, rather than refusing to answer for an
org in this state. The actual remedy is ending a window you can see: the store lists
the active group newest-started first, so ending one shrinks it by one and the
next-oldest, previously hidden active window becomes visible on the next read.
Finished windows are kept as long as your timeline is. They fall out of every
list surface 24 hours after they end and stop affecting anything at all the
moment they lapse; timeline.retention_days is what eventually deletes the rows
(at startup, alongside the source-events journal). The clock starts when the
window actually stopped silencing things, so one you ended after five minutes is
reaped N days after you ended it, not N days after the end time you originally
asked for. With the default 0 they are kept forever, which costs a few hundred
bytes each.
⚠️ What that deletion costs you, if you already set timeline.retention_days
for the source-events journal: this knob now reaps maintenance windows too, and
what goes with them is suppressed_count — the count of notifications each
window swallowed, which is the answer to "why was it quiet during the reboot".
Nothing else records it. The knob is shared deliberately, because both tables
are append-only records of what happened to this deployment and neither is worth
a retention number of its own, but if your post-mortems reach back further than
your timeline does, raise the value rather than discovering the gap during one.
Silence is visible. Each window counts the notifications it suppressed
(suppressed_count in the list response, a column in the UI table), and the
infrasigns_maintenance_suppressed_total counter carries the same fact for
alerting — read it beside infrasigns_notifications_sent_total so "quiet
because muted" never looks like "quiet because the pipeline broke". While any
window is open the dashboard carries a note saying so, because a quiet dashboard
and a muted one are otherwise the same picture.
Three surfaces, one set of rules. The Notifications page carries a
maintenance section in both modes — open a window from a form, see the active
and recently-lapsed ones, end one early — and it is what a hosted organization
uses, since the API below acts on a different organization than theirs. The
API is an operator scripting surface: it is gated by api.token and acts on
the process's default organization, and since
#537 it is mounted in both
modes — a window opened through it silences this deployment's own organization,
which is what makes it safe to serve next to tenants. The Q&A bot takes
/mute, /muteall, /unmute and /windows as typed
commands — the surface an operator is already standing at
when the page arrives — and is an operator surface for the same reason as the
API, its allowlist of chat ids being process configuration no tenant can set.
All three call the same validation and the same gate, so a window opened any
way behaves identically; the only difference is what each records in
created_by (api; web / web:<member email> when a signed-in member opened
it; bot:telegram:<chat>:<user> or bot:slack:<channel>:<user> from chat). The
UI offers a fixed set of durations (30m to 24h) rather than free text; the API
keeps the full one-minute-to-30-day range, and the bot caps a chat-opened window
at 24h — free text plus a one-keystroke gap between 3d and 30d is a typo
found by the outage it hides. Two more rules the bot narrows on purpose: the
org-wide window has its own verb (/muteall, so it is never one typo of a source
name away from /mute), and its authorization is the chat allowlist, so every
member of an allowlisted channel can open and end windows.
See docs/web.md for the UI's authorization posture: a window is
member-level, not owner-only, and in self-hosted mode /app is unauthenticated.
HTTP endpoints
| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | /healthz |
none | liveness: status + uptime, always 200 |
| GET | /readyz |
none | readiness. Nothing here gates the pod since #507: it answers 200 whenever the process answers at all, so a 200 is not evidence that any dependency is reachable. The diagnosis is the top-level word — ok (every entry in the body reads ok) or warning (something does not) — and the machine-readable signal is infrasigns_readiness, which is what replaced the status code. A third value, degraded (503), exists in the code and is unreachable in the shipped binary — no call site registers a group as gating, so your own manifests cannot produce it either; it is kept as the seam a multi-replica deployment would flip back. All three groups — sources, notify and subsystems — are advisory, so anything that cannot be reached is diagnosed in the body without taking the pod off the network. sources/notify are live probes (dial the source / notifier). The sources set is the config file's own, deliberately — see Where a source lives — so a source added through the web UI is not one this endpoint can report on. Under notify, a channel routed to no feed reports ok without dialing, since it cannot deliver and probing it would spend a dial on a channel that sends nothing; a section that is present but builds no channel has no notify entry at all rather than a failing one, and infrasigns_notify_channel_deliverable is where that is reported. subsystems (database, scheduler, bot-telegram, bot-slack, provisioning) report whether each enabled component started and its worker is running — except database, which is a live 2-second Ping of the connection pool and is the one check here that fails for an ordinary runtime reason (the worker entries beside it are start-markers instead: they report a worker that never started or has exited, both of them boot or shutdown facts rather than running-pod conditions), and provisioning, which reports no worker at all and is present only when the start-up provisioning pass could not apply this file's sources. That entry is non-gating: it reads disconnected, never ok, and never evicts the pod — the causes that are about this deployment's stored document are repaired on a page this pod serves, so gating over them would put that page behind a port-forward, and the rest (a lost write race, a failed write) are settled elsewhere entirely. Its reason and remedy are in the boot ERROR line, not in this body. The bot transports report the same disconnected — advisory, non-gating — when their live chat connection is lost (see bot readiness; mirrored by the infrasigns_bot_connected gauge). The MCP mount reports nothing here and never will: the entry it used to carry could not fail — so it gated nothing — and it appeared only when api.token was set, which made its presence in this unauthenticated body a read of whether you had configured that key (#552). Nothing replaced it; infrasigns_mcp_requests_total is where MCP traffic is visible. The alert receiver reports nothing here either, and for the same two reasons (#558): its check could not fail — a receiver stops only at process exit, when /readyz has already gone — and the entry appeared only when webhook.token was set, so it gated nothing while telling every unauthenticated poller whether you had configured one. Nothing replaced it; infrasigns_webhook_requests_total is where receiver traffic is visible. Always 200 while the process answers (ok or warning). |
| GET | /metrics |
none | Prometheus scrape (metrics catalog) |
| GET | /sources |
Bearer api.token |
latest persisted per-source health — source names, up/down, last error, scrape timestamps. Unauthenticated until #558; it is an inventory of your estate rather than a health signal, so it joined the api.token family and answers 401 with exactly the body a wrong token gets, disabled or not |
| POST | /api/digest/trigger |
Bearer api.token |
manual digest run (?source=, ?wait=true); disabled without a token, and then it answers 401 with exactly the body a wrong token gets — it names no key, because the route is reachable by anyone and the name of an unconfigured feature is not theirs to learn; serve names it in a start-up line instead. Mounted in both modes and always acts on this deployment's own organization, so the token is the whole gate |
| POST | /api/maintenance |
Bearer api.token |
open a maintenance window (source, reason, duration); disabled without a token, and then it answers 401 with exactly the body a wrong token gets — it names no key, because the route is reachable by anyone and the name of an unconfigured feature is not theirs to learn; serve names it in a start-up line instead. Mounted in both modes and always acts on this deployment's own organization, so the token is the whole gate; 409 (not 400) when the organization is already at its ~50-window cap — the request is fine, the organization's state is not |
| GET | /api/maintenance |
Bearer api.token |
list active and recently-lapsed windows, with how many notifications each suppressed |
| POST | /api/maintenance/{id}/end |
Bearer api.token |
end a window early; idempotent |
| POST | /webhook/alerts |
Bearer webhook.token |
inbound alert receiver (details); disabled without a token, and then it answers 401 with exactly the body a wrong token gets — the same posture as the /api/* rows above and /mcp below, and for the same reason: the route is reachable by anyone and the name of an unconfigured feature is not theirs to learn. It used to be absent without a token, which told a stranger the answer the 401 withholds (#558) |
| POST | /webhook/deploys |
Bearer webhook.token |
deploy events for post-deploy verification (details); disabled unless deploys.enabled, and then it answers 401 with exactly the body a wrong token gets, for the reason the row above gives. It used to be absent unless enabled (#558) |
| GET/POST/DELETE | /mcp |
Bearer api.token |
MCP server (details); disabled without a token, and then it answers 401 with exactly the body a wrong token gets — the same posture as the two rows above, and for the same reason: the route is reachable by anyone and the name of an unconfigured feature is not theirs to learn. It used to be absent without a token, which told a stranger the answer the 401 withholds; serve names the key in a start-up line instead. Mounted in both modes and every tool reads this deployment's own organization, so the token is the whole gate |
| GET | / |
none | the root. Self-hosted: redirects to /app/default. Cloud sign-in mode: a visitor with a live session is redirected the same way, and a visitor with none gets the public landing page. Exact-match route — any other unrouted path stays a 404, and POST / is a 405. Present only when ui.enabled: true |
| GET | /pricing |
none | public plan comparison: the limits each tier carries, read from the same table the quota gates enforce. No prices, no checkout. Route absent unless ui.public_site: true and a billing client is wired — the key alone does not mount it, since on a deployment that sells nothing every tier applies to nobody reading it. The billing client ships only in the ee build, so on the public image this route is absent regardless of configuration; present only when ui.enabled: true |
| POST | /access-request |
none | the landing form's write: records one email address as an ask for an invitation, and grants nothing. The only anonymous write in this table that is not an /app route, so the cross-origin check below is the whole of its browser-forgery defence — there is no session cookie behind it. It runs behind a single token bucket for the whole process (60 submissions an hour, burst 10, a constant with no config key) and a refusal answers 429 with Retry-After; every outcome is counted on infrasigns_access_request_submissions_total (metrics catalog). Route absent unless ui.public_site: true, since without a public site there is no landing page to carry the form; present only when ui.enabled: true |
| GET | /app* |
none | embedded web UI (project-slugged: /app/default/...); present only when ui.enabled: true |
| POST | /app/sources/verify |
none | wizard probe: read-only GETs against the entered URL (Prometheus/Loki) or the entered endpoint list (healthcheck, capped at 10), carrying the source's own credential; aggregate reply only. Refuses cloudwatch/digitalocean/hetzner, an ambient-credential sigv4, and — on a self-hosted deployment — oauth2 (details) |
| POST | /app/sources/preview |
none | wizard preview analysis: bounded 24h range fetch + deterministic engine findings, carrying the source's credential; no LLM |
| POST | /app/sources/config |
none | wizard config generation (pure computation, no network) |
| POST | /app/{project}/reports/{id}/feedback |
none | records digest feedback (vote + note) from the web UI — an unauthenticated write; restrict /app at the network layer like the other UI routes |
| POST | /app/{project}/incidents/{id}/resolve |
none (cloud: any member of the organization) | closes one still-firing episode by hand (details) — the same unauthenticated write in self-hosted; a sender that is still repeating the alert re-opens the episode on its next send |
| POST | /app/{project}/maintenance |
none (cloud: any member of the organization) | opens a maintenance window from the UI — the highest-consequence unauthenticated write in this table: self-hosted it needs no credential and can withhold every page, digest and check alert for up to 24 hours. Gate /app at the network layer before exposing it to anything you do not control — the cross-origin check below closes the drive-by half of this, not the reachability half |
| POST | /app/{project}/maintenance/{id}/end |
none (cloud: any member of the organization) | ends one window early; idempotent, and an id this organization cannot see is answered like a success |
| GET | /assets/* |
none | embedded UI static assets; present only when ui.enabled: true |
The unauthenticated endpoints expose operational health, not data — restrict them at the network layer if required (see API authentication).
Every POST /app/... route above, and POST /access-request with it, refuses a
cross-origin browser request —
Sec-Fetch-Site, falling back to Origin vs Host — and answers 403 before
the handler runs, in both self-hosted and hosted mode
(details). There is nothing to configure. It is not
authentication: a caller that sends no browser headers at all (curl, a script)
is still allowed through, so the network-layer restriction above is unchanged.
The /api/* and /webhook/* endpoints are unaffected — they carry a bearer
token and are not browser-reachable surfaces.
Metrics collected
InfraSigns queries these node_exporter metrics — the same golden-signal set
the add-source wizard suggests for node jobs, so preview findings match what
the runtime produces. Each carries warn/crit thresholds for the
deterministic analysis engine:
# cpu_usage_percent — warn 80, crit 95
(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100
# memory_usage_percent (per instance) — warn 85, crit 95
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
# disk_usage_percent (worst non-tmpfs/overlay mount per instance) — warn 80, crit 90
max by (instance) ((1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100)
If node_exporter is not present, metric queries return empty results (logged as warnings) and the digest is generated from active alerts only.