Lower mean
request latency
54.4 s→13.0 s
END-TO-ENDTHE WORKLOAD KEEPS CHANGING
Agents evolve.
Infra should.
Agents reason, code, act, and start new work.
Their infrastructure should move with them.
FREEDOM TO ADAPT
Evolving
Infra.
Same hardware. More freedom.
Infrastructure that adapts as agents work.
Unlock the degrees of freedom.
CHANGING WORKLOADSTHE SAME HARDWARE
01 / SEE THE DIFFERENCESAME MODEL. SAME WORKLOAD. SAME GPUs.
Less waiting.
More work done.
126 coding agents. 36 GB300 GPUs.
One hour of serving, side by side.
EXPERIMENT REPLAYGLM-5.2 · CLAUDE CODE · 60× PLAYBACK
Dynamo Evolving InfraReal traces. One shared comparison window.
Lower mean
Lower P95
time to first token
228.4 s→19.6 s
TAIL LATENCYAs many agent
sessions completed
52→117
IN ONE HOURCompared with Dynamo in the demo runs: GLM-5.2, 126 concurrent Claude Code agents, and the same 36 GB300 GPUs. Scored over one hour after a 30-minute warmup. E2E measures individual LLM requests.
BUILD FOR WHAT COMES NEXT
Your agents are evolving.
Your infrastructure can, too.
Open-source code coming soon
02 / THE IDEAGIVE THE SYSTEM ROOM TO MOVE.
Freedom is the
first step.
A system can only adapt along the dimensions it can change.
When compute roles, KV placement, and request routing are tightly coupled, adjusting one disrupts the others. Evolving Infra separates compute allocation from KV cache management, opening the space for runtime adaptation.
COMPUTE
Free to reassign
Free to reassign
PPP ↔ DDD
ADAPTDECOUPLE
CONTEXT
Kept accessiblePRESERVE
Kept accessiblePRESERVE
Unlock
the freedom.
Symmetric engines let every node serve prefill or decode without a restart. A shared host-memory KV pool keeps reusable context accessible when compute roles change.
DECOUPLED INFRASTRUCTUREUnderstand
the work.
The planner learns prefill and decode costs online, accounting for cached context. It reallocates nodes when a workload shift persists, without offline profiling.
WORKLOAD-AWARE PLANNINGMove compute.
Keep context.
The router balances load and fetches KV while requests wait. A branch-aware directory follows context changes, retaining useful KV and evicting unreferenced copies first.
AGENT-AWARE KV MANAGEMENTA See the architecture
One router assigns prefill and decode work to symmetric engines. The host memory of all nodes forms a shared KV pool, and a cluster-wide directory tracks the active branches that reference each copy.
B Explore the evidence
When the agent mix shifts,
the system follows.
Across a three-hour shift from math to computer-use agents, Evolving Infra reaches 2.1× Dynamo’s throughput and 28% more than the best static split.
Across coding, math,
and computer use.
In the paper’s stationary benchmarks, mean request latency improves by up to 4.1× and P95 TTFT by up to 14× over Dynamo. These are separate benchmark runs from the demo above.
Putnam · GLM-5.2
Claude Code · GLM-5.2
OSWorld · Kimi K2.6
Gray: Dynamo. Red: Evolving Infra. Upper panels show mean request E2E, lower panels P95 TTFT. The planner can trade TTFT for decode throughput: Putnam has higher P95 TTFT at 224 and 288 agents while mean request latency improves.
THE RESEARCH BEHIND THE SYSTEM
Built for
evolving workloads.
Evolving Infra: Decoupling Compute Allocation and KV Cache Management for Agent-Adaptive LLM Serving
1 Georgia Tech2 UIUC3 UC Berkeley4 NVIDIA5 OpenAI6 MIT
* Equal contribution. Technical report and BibTeX coming soon.