Skip to content

Q&A bot

Ask your infrastructure questions in chat and get answers grounded in real data: "is anything firing right now?", "what's happening with disk on prod?", "what did this morning's digest say?". An LLM agent answers by calling the same read-only tools the MCP server exposes — source health, digests, the reports archive, live alerts, configured metric queries, and arbitrary read-only PromQL and LogQL log queries. The agent can also consume read-only tools from external MCP servers you configure (a runbook server, a cloud provider's MCP endpoint).

Two transports are supported and can run at once: Telegram (getUpdates long-poll) and Slack (Socket Mode). Each is enabled independently by its own fail-closed allowlist; both share one reasoning provider and one daily budget.

Enable — Telegram

bot:
  telegram:
    # Optional: falls back to notify.telegram.token when empty — the common
    # case is one bot identity doing both delivery and Q&A.
    token: ""
    # The transport is enabled iff this list is non-empty. Messages from any
    # other chat are dropped without a reply.
    allowed_chat_ids: [123456789]

llm:
  provider: openai   # a reasoning provider is required — see below

Getting a bot token and your chat ID works exactly like for the Telegram notification channel. Reusing the delivery bot's token is fine: replies and digest deliveries do not conflict.

In the Helm chart, list the chat IDs under config.bot.telegram.allowed_chat_ids; the token falls back to TELEGRAM_TOKEN from existingSecret, or provide BOT_TELEGRAM_TOKEN there for a separate bot identity.

Enable — Slack

bot:
  slack:
    # App-level token (xapp-…) for the Socket Mode WebSocket. Required.
    app_token: "xapp-..."
    # Bot token (xoxb-…) for replies. Falls back to notify.slack.token when
    # empty — reusing the Web API delivery token is fine.
    bot_token: ""
    # The transport is enabled iff this list is non-empty. Mentions in any
    # other channel are dropped without a reply.
    allowed_channel_ids: [C0123ABCD]

llm:
  provider: openai

Slack uses Socket Mode, so the pod dials out to Slack over a WebSocket — there is no inbound endpoint to expose, matching the self-hosted posture of the Telegram long-poll and the MCP stdio server. Create a Slack app, enable Socket Mode, add an app-level token with the connections:write scope (xapp-…, → app_token) and a bot token (xoxb-…, → bot_token) with app_mentions:read and chat:write, then subscribe to the app_mention event. Invite the bot to each channel and copy its channel ID (the C…/G… id, not the name) into allowed_channel_ids.

Operators talk to the Slack bot by @-mentioning it in an allowlisted channel; the answer lands as an in-thread reply. Direct messages are not part of v1 — the allowlist is by channel. In the Helm chart, list the channel IDs under config.bot.slack.allowed_channel_ids; provide BOT_SLACK_APP_TOKEN (required) in existingSecret, and BOT_SLACK_TOKEN (falls back to SLACK_TOKEN).

Reasoning provider

The bot requires a reasoning provider (openai or anthropic). With llm.provider: none (or mock) the config is rejected at validation: free-form questions cannot be answered deterministically, and a template answer would be a fake one.

How it answers

Each question runs an agent loop: the model inspects the question, calls tools (up to 8 rounds), and composes a reply. Telegram replies are capped at Telegram's 4096-character limit and formatted via Telegram HTML (bold, italic, inline code, code blocks); if Telegram rejects a formatted payload, the reply is resent as plain text — a formatting problem degrades to an unformatted reply, not a lost one. Slack replies are capped at 4000 characters and use Slack mrkdwn (*bold*, _italic_, `code`, fenced blocks).

  • Budget: every reasoning turn takes one unit from llm.max_calls_per_day, so one question costs up to 9 calls (see the call-volume table). The cap is shared across both transports. When the daily budget is exhausted, the bot refuses honestly in chat — it never synthesizes an answer without the model.
  • Read-only: the bot's tool set excludes trigger_digest (and any future tool with side effects). Asking it to "send the digest now" cannot make it happen.
  • One at a time: each transport processes questions sequentially; a burst queues (on Telegram's side, or in a bounded in-memory queue for Slack) and is answered in order.
  • Time limits: a question is bounded by a 3-minute wall clock and per-tool timeouts; hitting a limit produces an honest "try a narrower question" reply.

External MCP tool servers

The agent can consume tools from external MCP servers you run — a runbook lookup server, a cloud provider's MCP endpoint — so it can answer with context InfraSigns itself doesn't hold. Discovered tools join the agent's catalog namespaced as <server>__<tool> (so they never collide with native tools) and are offered to the model alongside the built-in ones. This is a self-hosted-only feature: like the MCP server and the Q&A bots, it is suppressed in the hosted service.

bot:
  telegram:
    allowed_chat_ids: [123456789]   # a Q&A transport must be enabled to consume them
  mcp_servers:
    - name: runbooks                # [A-Za-z0-9_-], ≤40; prefixes every tool
      url: https://runbooks.internal/mcp   # MCP HTTP (streamable) endpoint
      token: "${RUNBOOKS_TOKEN}"    # optional bearer; https required off localhost
      tools: [list_runbooks, get_runbook]  # per-server allowlist (see below)
  • Per-server allowlist is the authority. Only the tools you name in tools are ever exposed to the agent — a server that advertises more is ignored for the rest. This is the trust decision, not the server's own metadata.
  • Read-only, defense in depth. A tool the server affirmatively marks as a writer (readOnlyHint: false) is rejected even if allowlisted — the agent takes no actions. A tool with no read-only annotation is trusted to your allowlist. The server's hints can only veto, never grant.
  • Connected at startup. Servers are dialed and their tools discovered when the daemon starts; a server that is down contributes no tools and logs a warning — the daemon keeps running with its native tools and the reachable servers. Sessions persist and reconnect for the process lifetime.
  • Same untrusted-data handling for results. An external tool's result is the most untrusted class there is: it is framed as untrusted data, control-stripped, and length-capped exactly like every native tool result (the injection posture below).
  • Trust the server for its tool metadata, not just its results. A tool's name, description, and input schema arrive off the wire and are sent to the model as catalog text (a description is authoritative capability text, a stronger position than a result labelled untrusted). InfraSigns control-strips and length-caps the description and rejects a non-object/oversized schema, but it cannot neutralize a plain-language instruction a malicious server puts in a description. Point mcp_servers only at servers you trust to describe their own tools honestly — the allowlist gates which tool names run, not what their descriptions say.
  • Not re-exported. External tools join the agent only; they are never advertised on InfraSigns' own MCP server (no proxying a third party's tools through our endpoint).

Security model

  • Fail-closed allowlist: allowed_chat_ids / allowed_channel_ids is the trust boundary — both for who can read your infrastructure data and for who can burn LLM budget. Messages from non-allowlisted chats/channels are dropped silently (no reply, so the bot's existence is not confirmed to strangers) and counted in the unauthorized metric status. A flood from an allowlisted chat is an accepted residual, bounded by the daily budget cap.
  • Prompt injection (#81 posture): tool results carry external data (label values, alert annotations) and are framed as untrusted data, control-stripped, and length-capped; the operator question is length-capped and never spliced into the system prompt.

Operational notes

  • Run a single instance. On Telegram this is a hard requirement: Telegram allows only one getUpdates consumer per token, so a second replica — or a dev instance on the prod token — makes both flap with 409 Conflict. Slack Socket Mode is more forgiving (it permits several concurrent connections and load-balances events across them), but a second replica would run its own uncoordinated daily budget, so llm.max_calls_per_day would only hold per process. Either way, keep it to one instance (the Helm chart is single-replica already) and use separate tokens for local experiments. Using the same token for notification delivery is not a conflict — only two inbound listeners are.
  • Restarts: on Telegram a question that was in flight (or arrived while the daemon was down) is re-delivered on the next start — updates are kept for 24 hours — with the flip side that questions answered since the last poll cycle may be answered a second time after a restart (confirmation is one poll behind), re-spending their budget cost. On Slack an in-flight question is not re-delivered (the mention was already acknowledged); the operator re-asks, and a best-effort "restarting" reply says so.
  • Metrics: infrasigns_bot_questions_total{transport,status} — transport is telegram or slack; status is answered, refused_budget, error, or unauthorized. infrasigns_bot_connected{transport} is 1 while the transport's live connection is up and 0 once it has been lost — the alerting surface for a zombie bot. Alert with a hold (infrasigns_bot_connected == 0 for 5m) so a brief reconnect doesn't page. The gauge tracks the connection lifecycle, not whether the bot is answering (see the residual cases below). The provider calls themselves land on the shared infrasigns_llm_requests_total{path="bot"} and infrasigns_llm_request_duration_seconds{path="bot"} — one sample per conversation turn, so a question that takes N tool-calling steps records N, which is what it spends from the daily budget.
  • Readiness: each running transport appears in /readyz under subsystems.bot-telegram / subsystems.bot-slack. The value is ok while the listener goroutine is running (a structural, gating check — it never fails on a transient hiccup), or disconnected when the transport's live connection has been lost: three consecutive Telegram poll failures (e.g. a persistent 409 from a second consumer on the same token) or three consecutive failed Slack Socket Mode dials — and immediately when a token is rejected (invalid_auth) or the connection gives up entirely. The disconnected value is advisory and non-gating — the top-level status stays ok and the pod keeps serving webhooks/UI/metrics, because a dead chat listener must not evict the pod from its Service endpoints. Only a structurally dead worker (which happens on shutdown) reports an error: and gates. Alert on the infrasigns_bot_connected gauge for a machine signal.
  • What the connection signal does not catch: it tracks the connection lifecycle, so connected (gauge 1, /readyz ok) means "the transport is connected", not "the bot is answering". It can't see (a) a connection that fails intermittently but never three times in a row — one success resets the streak (both transports debounce on consecutive failures); or (b) a Slack socket that is connected but has silently stopped delivering events, or whose ack writes fail (Socket Mode's own keepalive eventually recycles such a socket). For those, watch the infrasigns_bot_questions_total rate alongside the connection gauge.