NATS JetStream vs Kafka — lightweight messaging for microservices and AI agents
Lightweight messaging for microservices and AI agents — a Kafka alternative, with a working Java demo.
NATS JetStream is a lightweight persistence layer on top of the NATS broker — a Kafka alternative that, for many use cases, is simpler and ships with features Kafka doesn’t give you out of the box. I built a demo in Java 25 + Micronaut: the same code publishes and consumes through both systems, measures latency, and shows JetStream’s features live. The code is public — docker compose up and it runs.
Below: the JetStream model vs Kafka’s log, latency measurements, five runnable scenarios (deduplication, replay, work queue, KV…), and use cases where NATS fits ideally — from edge and microservices to task queues and AI-agent communication.
Kafka in two sentences (you know it anyway)
Kafka is a distributed log — an append-only record split into partitions, read by consumer groups that track their own offset. It shines where you need huge throughput, durability, and a rich ecosystem (Connect, Streams, ksqlDB). Since version 4 it runs natively in KRaft mode.
The price of that power is operational complexity and a model where a lot is done around the broker: deduplication, task queues, replaying from a specific point. It’s doable — just not always elegant.
NATS and JetStream — a light core with strong features
NATS is a lightweight pub/sub broker: a single binary, sub-second startup, hierarchical subjects (orders.eu.*). On its own it delivers at-most-once — fast, but without durability.
JetStream is a persistence layer built into the same server and the same client. It adds streams, durable consumers, and guarantees — without standing up separate infrastructure. And this is where it gets interesting, because JetStream has a few things that take gymnastics in Kafka:
- Per-message deduplication — set the
Nats-Msg-Idheader and the server drops repeats within a dedup window. Idempotent publish “for free”, durable across client sessions. In Kafka, producer idempotence is limited to a single session. - Client-side replay — the consumer picks its own start point:
DeliverPolicyByStartTime(from a moment) orByStartSequence(from a number). Many independent consumers replay the same history without groups or resetting offsets. - Per-message acknowledgment —
ack()(at-least-once),ackSync()(double-ack, confirmed by the server),nak(), redelivery afterAckWait. Kafka commits a group’s offset, not an individual message. - WorkQueue — native queue semantics: a message goes to a single consumer and disappears after it’s acknowledged. In Kafka you emulate this with partitions and groups.
- KV and Object store — key-value and object stores built on JetStream, in the same client. The Kafka client has no equivalent.
On top of that comes operational simplicity: one lightweight server (great for edge and leaf-node clusters too), a simple mental model.
Demo: one API, two backends
To keep the comparison fair, I built an app where the same code publishes and consumes through both systems. Stack:
- Java 25, Micronaut 5, the
micronaut-kafkaandmicronaut-natsmodules - Kafka (KRaft) and NATS with JetStream in containers
- HdrHistogram for latency percentiles
At its heart is a shared MessagePublisher port with two implementations (Kafka, JetStream) and a shared sink both consumers report to. That makes the feeder and the measurement backend-agnostic — you switch with DEMO_BACKEND=kafka|jetstream|both.
public interface MessagePublisher {
Backend backend();
void publish(String id, String payload, long sentAtEpochNanos); // synchronous (demo)
void publishAsync(String id, String payload, long sentAtEpochNanos); // non-blocking (benchmark)
}
Kafka is handled declaratively (@KafkaClient / @KafkaListener), and JetStream via the raw jnats client with the newer, simpler API (Simplified Consumer) — deliberately, because the declarative, reactive client deferred publishing and skewed the measurement.
What you see live
./gradlew shadowJar
docker compose up --build
The feeder publishes a batch of messages to Kafka and JetStream, and the consumer (a separate container) prints what it received from both — with latency computed off a shared clock. A round-trip through both systems in a single terminal window.
Latency benchmark
The benchmark mode sends at a steady rate with a warm-up phase, asynchronously (so the sender keeps pace and doesn’t skew the result), measures latency against the scheduled send time (coordinated-omission-aware), and prints percentiles:
docker compose run --rm -e DEMO_MODE=benchmark -e DEMO_BACKEND=jetstream consumer
docker compose run --rm -e DEMO_MODE=benchmark -e DEMO_BACKEND=kafka consumer
On a single node, with equal guarantees (Kafka acks=all + idempotence vs JetStream with a publish ack), at 500 msg/s, mean of 3 runs:
| percentile | Kafka | JetStream | JS faster |
|---|---|---|---|
| p50 | 1.85 ms | 0.74 ms | 2.5× |
| p95 | 4.87 ms | 1.41 ms | 3.5× |
| p99 | 10.59 ms | 2.94 ms | 3.6× |
| p99.9 | 19.38 ms | 6.61 ms | 2.9× |
| max | 21.48 ms | 8.49 ms | 2.5× |
JetStream is faster at every percentile (~2.5–3.6×), and the tail is the most telling part: where Kafka blows out at p99.9/max (~19–21 ms), JetStream stays tight (~7–8 ms) — its worst case (max 8.5 ms) beats Kafka’s p99 (10.6 ms).
Two caveats, to be fair: this is one node (RF=1 — it doesn’t show the cost of replication), and the numbers are illustrative and heavily configuration-dependent. A fun finding from building the demo: when I published synchronously, Kafka’s latency “jumped” to tens of milliseconds — but that was sender backlog, not Kafka’s fault. After switching to async publishing, everything dropped back to sane values. A common lesson from benchmarks: first you measure your harness, only then the system.
The key point: JetStream’s real edge isn’t raw latency — it’s the features and the simplicity. And that’s what the next scenarios show.
JetStream feature scenarios
Each one is a self-contained, runnable example that prints an observable effect:
docker compose run --rm -e DEMO_MODE=dedup -e DEMO_BACKEND=jetstream consumer
docker compose run --rm -e DEMO_MODE=guarantees -e DEMO_BACKEND=jetstream consumer
docker compose run --rm -e DEMO_MODE=replay -e DEMO_BACKEND=jetstream consumer
docker compose run --rm -e DEMO_MODE=workqueue -e DEMO_BACKEND=jetstream consumer
docker compose run --rm -e DEMO_MODE=kv -e DEMO_BACKEND=jetstream consumer
- dedup — two publishes with the same
Nats-Msg-Id; the second comes back asduplicate=true, and the stream keeps one message. - guarantees — we receive a message and deliberately don’t ack it → after
AckWaitit arrives again (redelivery); afterackSyncdeliveries stop. - replay — we publish 5 events, then two independent consumers replay history from the 3rd event: one by sequence, the other by time — without resetting offsets.
- workqueue — 6 jobs, two competing workers; each job handled exactly once, the queue drains after ack.
- kv — put/get/update with revision numbers.
What it looks like in code
The best “for free” example — deduplication is a single messageId:
PublishOptions opts = PublishOptions.builder().messageId("order-42").build();
js.publish(subject, body, opts);
PublishAck ack = js.publish(subject, body, opts); // same Nats-Msg-Id
// ack.isDuplicate() == true — the stream keeps a single message
And replay is the consumer choosing its start point — no offset resets:
ConsumerConfiguration.builder().deliverPolicy(ByStartTime).startTime(cutoff).build();
// or: .deliverPolicy(ByStartSequence).startSequence(seq)
The rest (guarantees, WorkQueue, KV) is in the repo — the code is deliberately readable so you can paste it straight into your own project.
Where JetStream really shines
AI agent systems — the killer use case
Building a multi-agent system? This is JetStream’s strongest use case today. Three pillars: fast messaging between agents (lots of small control messages — routing, tool calls, status updates), the WorkQueue pattern (a pool of agents pulls tasks from a queue, each handled exactly once), and dedup by ID — an agent can safely retry a command, and Nats-Msg-Id guarantees the tool won’t run the same task twice. All of it under one roof:
- pub/sub over subjects (
agent.planner.*,agent.tools.search) — natural routing of tasks and events between agents, - deduplication — an agent can safely retry a command;
Nats-Msg-Idguarantees the tool runs it once (idempotent commands), - per-message ack + redelivery — a task handed to a worker agent won’t get lost; if it isn’t acked, it comes back,
- KV store as shared memory/state — context, configuration, the agents’ “scratchpad”,
- replay — a new (or restarted) agent replays the conversation/event history from a chosen point to catch up on context,
- WorkQueue — a pool of worker agents pulls tasks from a queue, each handled once.
And all of this on a lightweight server that’s easy to run next to your app — without pulling in a heavy streaming platform just so agents can talk to each other.
Other good fits
- Edge / IoT — a light broker, leaf nodes, working over flaky connectivity.
- Task queues — native WorkQueue instead of emulating it with partitions.
- Event sourcing / audit — replay by time or sequence as a built-in feature.
- Microservices without heavy infrastructure — pub/sub, request-reply, and KV in one.
- Live config and feature flags — a KV store with change watching.
Summary — what to pick
Reach for Kafka when you need extreme throughput, a rich streaming ecosystem (Connect/Streams), data-warehouse integrations, and you have a team that already knows it.
Consider NATS JetStream when you care about operational simplicity and need features in the core: deduplication, per-message ack, replay, queues, KV — or you’re building something lightweight: agent communication, edge, microservices. It often turns out you get “for free” what you’d have to bolt on in Kafka.
🔗 More: NATS · JetStream docs · Java client (jnats)
Best of all, just run the demo and see the difference for yourself 👇