{"id":311,"date":"2026-06-03T05:38:47","date_gmt":"2026-06-03T05:38:47","guid":{"rendered":"https:\/\/marcinzygmunt.pl\/blog\/?p=311"},"modified":"2026-06-21T17:38:44","modified_gmt":"2026-06-21T17:38:44","slug":"ai-as-a-new-dimension-of-monitoring-an-mcp-server-for-prometheus","status":"publish","type":"post","link":"https:\/\/marcinzygmunt.pl\/blog\/ai-as-a-new-dimension-of-monitoring-an-mcp-server-for-prometheus\/","title":{"rendered":"AI as a New Dimension of Monitoring \u2014 an MCP Server for Prometheus"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">I put an AI agent directly in front of live telemetry. Instead of staring at dashboards and recalling PromQL syntax, I ask in plain language:&nbsp;<em>&#8220;Is this growing map a memory leak?&#8221;<\/em>&nbsp;\u2014 and the agent builds the query itself, hits Prometheus, and explains the result.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">The problem: monitoring stops at the dashboard<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Classic monitoring works like this: Prometheus collects numbers, Grafana draws charts \u2014 and that&#8217;s where it ends. From there on it&#8217;s a human&#8217;s job. Someone has to open the dashboard, read the curves, recall the right PromQL (<code>rate()<\/code>?&nbsp;<code>histogram_quantile()<\/code>?&nbsp;<code>sum by (instance)<\/code>?), and connect the facts: is this rising heap a real problem, or just the normal GC sawtooth?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That &#8220;last mile&#8221; \u2014 from data to conclusion \u2014 is still manual. And that&#8217;s exactly what I wanted to automate.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The idea: an AI agent in front of live telemetry<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The missing layer is provided by an&nbsp;<strong>MCP (Model Context Protocol) server for Prometheus<\/strong>. MCP is an open protocol that lets any AI assistant (Claude Code, an IDE agent, a custom client) reach for external tools and data. Here, that &#8220;resource&#8221; is a live Prometheus.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It&#8217;s a small, self-contained playground made of&nbsp;<strong>two repositories<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The monitored stack<\/strong>\u00a0\u2014 a sample application plus the full metrics pipeline. This is where the data the MCP reads actually comes from. It&#8217;s an example app whose\u00a0<em>only<\/em>\u00a0job is to deliver metrics, not a real product:\u00a0<strong>\ud83d\udc49\u00a0<a href=\"https:\/\/github.com\/marcinzygmunt-pl\/prometheus-stack\">github.com\/marcinzygmunt-pl\/prometheus-stack<\/a><\/strong><\/li>\n\n\n\n<li><strong>The MCP server<\/strong>\u00a0\u2014 the bridge between the AI agent and Prometheus:\u00a0<strong>\ud83d\udc49\u00a0<a href=\"https:\/\/github.com\/marcinzygmunt-pl\/prometheus-mcp\">github.com\/marcinzygmunt-pl\/prometheus-mcp<\/a><\/strong><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The monitored stack bundles:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>a\u00a0<strong>traffic-generating app<\/strong>\u00a0in Java that loads itself with synthetic traffic (HTTP calls, in-memory map &#8220;churn,&#8221; real pressure on the JVM\/GC) \u2014 so there&#8217;s always something to observe;<\/li>\n\n\n\n<li><strong>Prometheus + node-exporter<\/strong>\u00a0collecting app and host metrics (CPU, memory, disk, network, JVM heap, GC, and custom\u00a0<code>loadgen_*<\/code>\u00a0business metrics);<\/li>\n\n\n\n<li>the\u00a0<strong>MCP server<\/strong>, which on startup reads every metric name Prometheus knows and\u00a0<strong>registers one MCP tool per metric<\/strong>\u00a0\u2014 plus escape hatches for raw PromQL.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">The core: one tool per metric<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The most interesting design decision: instead of exposing a single generic &#8220;query Prometheus&#8221; tool, the server&nbsp;<strong>dynamically scans the available metrics on startup and creates a separate tool for each one<\/strong>&nbsp;(<code>loadgen_map_size<\/code>,&nbsp;<code>jvm_memory_used_bytes<\/code>,&nbsp;<code>process_cpu_usage<\/code>,&nbsp;<code>node_cpu_seconds_total<\/code>, \u2026).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Each such tool takes sensible arguments:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>matchers<\/code>\u00a0\u2014 a label selector, e.g.\u00a0<code>area=\"heap\"<\/code>\u00a0or\u00a0<code>job=\"load-generator\"<\/code>;<\/li>\n\n\n\n<li><code>rate_window<\/code>\u00a0\u2014 e.g.\u00a0<code>5m<\/code>, which wraps the metric in\u00a0<code>rate(metric{...}[5m])<\/code>\u00a0(for counters);<\/li>\n\n\n\n<li><code>start<\/code>\u00a0+\u00a0<code>end<\/code>\u00a0+\u00a0<code>step<\/code>\u00a0\u2014 a range query instead of an instant one;<\/li>\n\n\n\n<li><code>time<\/code>\u00a0\u2014 the evaluation timestamp.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This gives the agent a&nbsp;<strong>map of the available telemetry right in the tool descriptions<\/strong>&nbsp;\u2014 it doesn&#8217;t have to guess which metrics exist. And when more is needed (aggregations,&nbsp;<code>histogram_quantile()<\/code>),&nbsp;<code>promql_query<\/code>&nbsp;and&nbsp;<code>promql_query_range<\/code>&nbsp;are there as raw-PromQL escape hatches, plus&nbsp;<code>refresh_metric_tools<\/code>, which re-scans Prometheus and updates the tool set.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><strong>A note on tool count.<\/strong>&nbsp;Prometheus can expose hundreds of metric names. Too broad a tool set bloats the MCP client&#8217;s context, so the default allow-list of metrics (<code>PROM_METRICS_INCLUDE<\/code>) is narrowed, and&nbsp;<code>PROM_MAX_TOOLS<\/code>&nbsp;hard-caps it. You tune this with include\/exclude regexes.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">What it looks like in practice<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I ask in plain language:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><em>&#8220;How did CPU load look over the last 15 minutes?&#8221;<\/em>&nbsp;<em>&#8220;Analyze the heap.&#8221;<\/em>&nbsp;<em>&#8220;Is the rising&nbsp;<code>loadgen_map_size<\/code>&nbsp;causing a memory leak?&#8221;<\/em><\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\">The agent picks the right metric tool, builds the PromQL, runs the query, reads the result, and&nbsp;<strong>explains it<\/strong>&nbsp;\u2014 correlating heap, post-GC &#8220;live data,&#8221; and the growing map to tell a real leak apart from the normal GC sawtooth. No staring at a dashboard, no recalling query syntax.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Example questions to try<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Once the stack is running, here are questions that map cleanly onto the metrics it exposes:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Host \/ machine<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&#8220;Show me CPU stats&#8221; \u2192\u00a0<code>node_cpu_seconds_total<\/code><\/li>\n\n\n\n<li>&#8220;What&#8217;s the load average?&#8221; \u2192\u00a0<code>node_load1<\/code>\u00a0\/\u00a0<code>node_load5<\/code>\u00a0\/\u00a0<code>node_load15<\/code><\/li>\n\n\n\n<li>&#8220;How much memory is free right now?&#8221; \u2192\u00a0<code>node_memory_MemAvailable_bytes<\/code><\/li>\n\n\n\n<li>&#8220;Plot disk I\/O over the last hour&#8221; \u2192\u00a0<code>node_disk_read_bytes_total<\/code>\u00a0\/\u00a0<code>node_disk_written_bytes_total<\/code><\/li>\n\n\n\n<li>&#8220;What&#8217;s the network throughput?&#8221; \u2192\u00a0<code>node_network_receive_bytes_total<\/code>\u00a0\/\u00a0<code>node_network_transmit_bytes_total<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>JVM application<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&#8220;Show me JVM heap usage&#8221; \u2192\u00a0<code>jvm_memory_used_bytes<\/code><\/li>\n\n\n\n<li>&#8220;Are there any long GC pauses?&#8221; \u2192\u00a0<code>jvm_gc_pause_seconds<\/code><\/li>\n\n\n\n<li>&#8220;How many threads is the app running?&#8221; \u2192\u00a0<code>jvm_threads_live_threads<\/code><\/li>\n\n\n\n<li>&#8220;What&#8217;s the HTTP request rate?&#8221; \u2192\u00a0<code>rate(http_server_requests_seconds_count[1m])<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Load generator<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>&#8220;How many calls per second is loadgen making?&#8221; \u2192\u00a0<code>rate(loadgen_http_calls_total[1m])<\/code><\/li>\n\n\n\n<li>&#8220;What&#8217;s the p95 latency of the load generator?&#8221; \u2192\u00a0<code>loadgen_http_call_duration_seconds{quantile=\"0.95\"}<\/code>\u00a0(it&#8217;s a\u00a0<strong>summary<\/strong>\u00a0with pre-computed quantiles \u2014 there are no histogram buckets, so\u00a0<code>histogram_quantile()<\/code>\u00a0returns nothing)<\/li>\n\n\n\n<li>&#8220;What&#8217;s loadgen&#8217;s process CPU usage?&#8221; \u2192\u00a0<code>process_cpu_usage{job=\"load-generator\"}<\/code>\u00a0(a 0\u20131 fraction of available CPU)<\/li>\n\n\n\n<li>&#8220;Is\u00a0<code>loadgen_map_size<\/code>\u00a0leaking memory?&#8221; \u2192 correlate\u00a0<code>loadgen_map_size<\/code>\u00a0with\u00a0<code>jvm_memory_used_bytes<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">A couple of phrasing tips that produce better answers: say&nbsp;<strong>&#8220;right now&#8221;<\/strong>&nbsp;vs&nbsp;<strong>&#8220;over the last hour&#8221;<\/strong>&nbsp;to choose instant vs range queries, and mention a&nbsp;<strong>window<\/strong>&nbsp;(&#8220;last 5 min&#8221;) for rate-based metrics. You don&#8217;t need to say &#8220;Prometheus&#8221; or get the metric name right \u2014 the agent figures it out.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><strong>One honest caveat:<\/strong>&nbsp;the MCP only surfaces what Prometheus actually scrapes, and the&nbsp;<em>shape<\/em>&nbsp;of a metric matters. The load generator&#8217;s latency, for example, is a&nbsp;<strong>summary<\/strong>&nbsp;(pre-computed&nbsp;<code>quantile=\"0.5\/0.95\/0.99\"<\/code>), not a histogram \u2014 so&nbsp;<code>histogram_quantile()<\/code>&nbsp;returns nothing and you read the quantile label directly. A good setup tells you that plainly instead of inventing a number: every value comes straight from Prometheus, or you&#8217;re told why it isn&#8217;t there.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">The stack<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Java 25<\/strong>\u00a0+\u00a0<strong>Micronaut 5<\/strong>\u00a0(the\u00a0<code>micronaut-mcp-server-java-sdk<\/code>\u00a0module over the official MCP Java SDK 1.0.0)<\/li>\n\n\n\n<li><strong>streamable HTTP<\/strong>\u00a0transport on\u00a0<code>\/mcp<\/code><\/li>\n\n\n\n<li><strong>Prometheus<\/strong>\u00a0+\u00a0<strong>node-exporter<\/strong><\/li>\n\n\n\n<li><strong>Docker<\/strong>\u00a0\u2014 the whole thing stands next to the\u00a0<code>pgo\/prometheus<\/code>\u00a0stack on a shared\u00a0<code>monitoring<\/code>\u00a0network<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Wiring it into Claude Code is a single command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>claude mcp add --transport http prometheus http:\/\/localhost:8765\/mcp\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Why it matters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is a&nbsp;<strong>reference rig for AI-native observability<\/strong>. The sample stack is deliberately exposed so MCP clients can connect and experiment \u2014 but you&#8217;d point the same pattern at&nbsp;<em>any<\/em>&nbsp;production Prometheus by changing a single&nbsp;<code>PROMETHEUS_BASE_URL<\/code>&nbsp;environment variable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Monitoring stops being something you have to&nbsp;<em>read<\/em>. It becomes something you can simply&nbsp;<em>ask<\/em>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Sample stack (the monitored app + Prometheus that feed the MCP):&nbsp;<a href=\"https:\/\/github.com\/marcinzygmunt-pl\/prometheus-stack\">github.com\/marcinzygmunt-pl\/prometheus-stack<\/a><\/em>&nbsp;<em>MCP server code:&nbsp;<a href=\"https:\/\/github.com\/marcinzygmunt-pl\/prometheus-mcp\">github.com\/marcinzygmunt-pl\/prometheus-mcp<\/a><\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I put an AI agent directly in front of live telemetry. Instead of staring at dashboards and recalling PromQL syntax, I ask in plain language:&nbsp;&#8220;Is this growing map\u2026<\/p>\n","protected":false},"author":1,"featured_media":355,"comment_status":"open","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[13,72,15,71,74,73],"class_list":["post-311","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-ai","tag-devops","tag-java","tag-mcp","tag-observability","tag-prometheus"],"jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/marcinzygmunt.pl\/blog\/wp-content\/uploads\/2026\/06\/pb_ZFmGvN45AvIGjB_pur_zevfYA9V.png","_links":{"self":[{"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/posts\/311","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/comments?post=311"}],"version-history":[{"count":2,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/posts\/311\/revisions"}],"predecessor-version":[{"id":313,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/posts\/311\/revisions\/313"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/media\/355"}],"wp:attachment":[{"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/media?parent=311"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/categories?post=311"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/marcinzygmunt.pl\/blog\/wp-json\/wp\/v2\/tags?post=311"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}