AI as a New Dimension of Monitoring — an MCP Server for Prometheus

I put an AI agent directly in front of live telemetry. Instead of staring at dashboards and recalling PromQL syntax, I ask in plain language: “Is this growing map a memory leak?” — and the agent builds the query itself, hits Prometheus, and explains the result.

The problem: monitoring stops at the dashboard

Classic monitoring works like this: Prometheus collects numbers, Grafana draws charts — and that’s where it ends. From there on it’s a human’s job. Someone has to open the dashboard, read the curves, recall the right PromQL (rate()? histogram_quantile()? sum by (instance)?), and connect the facts: is this rising heap a real problem, or just the normal GC sawtooth?

That “last mile” — from data to conclusion — is still manual. And that’s exactly what I wanted to automate.

The idea: an AI agent in front of live telemetry

The missing layer is provided by an MCP (Model Context Protocol) server for Prometheus. MCP is an open protocol that lets any AI assistant (Claude Code, an IDE agent, a custom client) reach for external tools and data. Here, that “resource” is a live Prometheus.

It’s a small, self-contained playground made of two repositories:

• The monitored stack — a sample application plus the full metrics pipeline. This is where the data the MCP reads actually comes from. It’s an example app whose only job is to deliver metrics, not a real product: 👉 github.com/marcinzygmunt-pl/prometheus-stack
• The MCP server — the bridge between the AI agent and Prometheus: 👉 github.com/marcinzygmunt-pl/prometheus-mcp

The monitored stack bundles:

• a traffic-generating app in Java that loads itself with synthetic traffic (HTTP calls, in-memory map “churn,” real pressure on the JVM/GC) — so there’s always something to observe;
• Prometheus + node-exporter collecting app and host metrics (CPU, memory, disk, network, JVM heap, GC, and custom loadgen_* business metrics);
• the MCP server, which on startup reads every metric name Prometheus knows and registers one MCP tool per metric — plus escape hatches for raw PromQL.

The core: one tool per metric

The most interesting design decision: instead of exposing a single generic “query Prometheus” tool, the server dynamically scans the available metrics on startup and creates a separate tool for each one (loadgen_map_size, jvm_memory_used_bytes, process_cpu_usage, node_cpu_seconds_total, …).

Each such tool takes sensible arguments:

• matchers — a label selector, e.g. area="heap" or job="load-generator";
• rate_window — e.g. 5m, which wraps the metric in rate(metric{...}[5m]) (for counters);
• start + end + step — a range query instead of an instant one;
• time — the evaluation timestamp.

This gives the agent a map of the available telemetry right in the tool descriptions — it doesn’t have to guess which metrics exist. And when more is needed (aggregations, histogram_quantile()), promql_query and promql_query_range are there as raw-PromQL escape hatches, plus refresh_metric_tools, which re-scans Prometheus and updates the tool set.

A note on tool count. Prometheus can expose hundreds of metric names. Too broad a tool set bloats the MCP client’s context, so the default allow-list of metrics (PROM_METRICS_INCLUDE) is narrowed, and PROM_MAX_TOOLS hard-caps it. You tune this with include/exclude regexes.

What it looks like in practice

I ask in plain language:

“How did CPU load look over the last 15 minutes?” “Analyze the heap.” “Is the rising loadgen_map_size causing a memory leak?”

The agent picks the right metric tool, builds the PromQL, runs the query, reads the result, and explains it — correlating heap, post-GC “live data,” and the growing map to tell a real leak apart from the normal GC sawtooth. No staring at a dashboard, no recalling query syntax.

Example questions to try

Once the stack is running, here are questions that map cleanly onto the metrics it exposes:

Host / machine

• “Show me CPU stats” → node_cpu_seconds_total
• “What’s the load average?” → node_load1 / node_load5 / node_load15
• “How much memory is free right now?” → node_memory_MemAvailable_bytes
• “Plot disk I/O over the last hour” → node_disk_read_bytes_total / node_disk_written_bytes_total
• “What’s the network throughput?” → node_network_receive_bytes_total / node_network_transmit_bytes_total

JVM application

• “Show me JVM heap usage” → jvm_memory_used_bytes
• “Are there any long GC pauses?” → jvm_gc_pause_seconds
• “How many threads is the app running?” → jvm_threads_live_threads
• “What’s the HTTP request rate?” → rate(http_server_requests_seconds_count[1m])

Load generator

• “How many calls per second is loadgen making?” → rate(loadgen_http_calls_total[1m])
• “What’s the p95 latency of the load generator?” → loadgen_http_call_duration_seconds{quantile="0.95"} (it’s a summary with pre-computed quantiles — there are no histogram buckets, so histogram_quantile() returns nothing)
• “What’s loadgen’s process CPU usage?” → process_cpu_usage{job="load-generator"} (a 0–1 fraction of available CPU)
• “Is loadgen_map_size leaking memory?” → correlate loadgen_map_size with jvm_memory_used_bytes

A couple of phrasing tips that produce better answers: say “right now” vs “over the last hour” to choose instant vs range queries, and mention a window (“last 5 min”) for rate-based metrics. You don’t need to say “Prometheus” or get the metric name right — the agent figures it out.

One honest caveat: the MCP only surfaces what Prometheus actually scrapes, and the shape of a metric matters. The load generator’s latency, for example, is a summary (pre-computed quantile="0.5/0.95/0.99"), not a histogram — so histogram_quantile() returns nothing and you read the quantile label directly. A good setup tells you that plainly instead of inventing a number: every value comes straight from Prometheus, or you’re told why it isn’t there.

The stack

• Java 25 + Micronaut 5 (the micronaut-mcp-server-java-sdk module over the official MCP Java SDK 1.0.0)
• streamable HTTP transport on /mcp
• Prometheus + node-exporter
• Docker — the whole thing stands next to the pgo/prometheus stack on a shared monitoring network

Wiring it into Claude Code is a single command:

claude mcp add --transport http prometheus http://localhost:8765/mcp

Why it matters

This is a reference rig for AI-native observability. The sample stack is deliberately exposed so MCP clients can connect and experiment — but you’d point the same pattern at any production Prometheus by changing a single PROMETHEUS_BASE_URL environment variable.

Monitoring stops being something you have to read. It becomes something you can simply ask.

---

Sample stack (the monitored app + Prometheus that feed the MCP): github.com/marcinzygmunt-pl/prometheus-stack MCP server code: github.com/marcinzygmunt-pl/prometheus-mcp