Every enterprise running LLMs is operating something that increasingly looks like an in-memory database they didn't know they built.

A prompt arrives. The system converts it into an internal representation, creates temporary state, keeps that state close to compute, schedules competing requests around scarce memory, reuses what it can, and repeatedly accesses that state to generate the response.

I spent part of my early career building MySQL and working deep in database systems. Looking at modern inference through that lens, many of these problems feel surprisingly familiar.

But the more interesting parallel isn't what inference has already borrowed from databases. It's what it hasn't borrowed yet.

Databases didn't become efficient simply by building better caches and execution kernels. They built optimizers that understood the workload, available state, hardware and cost, and chose the execution plan.

Inference is heading toward the same architecture.

Inference is obviously not a database.

The algorithms and hardware are different. But inference serving has become a memory-management, caching, scheduling, locality and execution-engine problem.

Database performance revolved around managing the buffer pool. Inference economics revolves around managing KV state.

1From prompt to response, translated for a database engineer

LLM inference Database kernel analogue What's actually analogous
Tokenization Parser / lexical analysis Converts external input into an internal representation the execution engine understands. SQL becomes tokens/AST; text becomes model token IDs.
Prefill Plan initialization / working-set construction Performs substantial upfront work and materializes the state that subsequent execution repeatedly consumes.
Attention Repeated access over the working set Each decode step accesses prior state. The analogy is the systems problem of repeatedly scanning/accessing a growing working set—not database lookup semantics.
FlashAttention Buffer-aware / cache-aware execution Reorganizes execution around the memory hierarchy to minimize expensive data movement, analogous to cache-conscious database operators.
KV Cache Buffer pool Keeps expensive-to-reconstruct working state close to execution rather than recomputing it. Capacity, residency, eviction, and locality become first-class concerns.
Decode Iterator / execution loop Repeatedly executes against retained state to produce the next unit of output. Not relational iterator semantics, but a similar execution-engine loop.

The database kernel hiding inside inference

DATABASE KERNELINFERENCE ENGINE
Virtual memory / buffer pagingPagedAttention
Buffer-pool residencyKV-cache residency
Materialized result reusePrefix caching
Workload schedulingContinuous batching
Compression / compact representationQuantization
Speculative executionSpeculative decoding
Operator/workload specializationPrefill/decode disaggregation
Locality-aware placementKV/prefix-aware routing
Cost-based query optimizationInference execution optimizationEmerging
EXPLAINEXPLAIN INFERENCENot yet
Same class of systems problems, different hardware. GPU HBM is the new buffer pool.

These aren't isolated tricks. Together they show inference becoming an execution engine, managing memory, locality, concurrency, reuse and specialization under tight hardware limits.

The kernel problem is familiar: keep useful state close, move as little as possible, and keep expensive execution units busy.

2What 40 years of database engineering can still teach inference

Inference has already rediscovered much of the kernel. There is still much more to borrow.

Cache policy

Workload-aware cache management

A reusable system prompt has different value from a one-time document scan. RAG, chat and agent loops have different access patterns.

Cache management should understand predicted reuse, not simply recency.

Memory tiers

A real KV memory hierarchy

KV state is already moving beyond HBM into host memory and external tiers.

HBMDRAMCXLNVMeDistributed storage

Each tier has different latency and cost characteristics. Sometimes retrieving cold KV state may cost more than rebuilding it.

Once KV state spans a hierarchy, caching stops being only a cache-management problem. It becomes a planning problem.

Retrieve the state?Move the request to the state?Recompute it?Use a different execution path?

Those are execution-plan decisions.

Routing

Locality-aware scheduling

KV-aware and prefix-aware routing already exists in NVIDIA Dynamo, llm-d and SGLang's router.

The hard part is the trade-off.

Route to stateThe GPU already holding 100K tokens of useful context. Risk a hotspot.
Route to idleAn idle GPU. Pay to rebuild.

This is the classic locality-versus-skew problem distributed databases have fought for years.

Invalidation

Context has dependencies

KV state isn't a conventional mutable cache. Change an earlier token and downstream state becomes invalid.

Agentic workloads make this harder because context branches, summarizes, edits and reorders continuously.

Unlike a database page, you can't simply patch the corresponding KV entry. Context reuse becomes a dependency-management problem, not merely a caching problem.

What each side can learn from the other

There's an irony in how inference infrastructure evolved.

In only a few years, the industry independently arrived at paging for KV memory, continuous scheduling, cache reuse, locality-aware routing, compression, workload specialization and increasingly sophisticated memory hierarchies. Many are new implementations of systems problems database kernels have wrestled with for decades.

Had some of today's inference problems been framed early to database-kernel engineers, we might have recognized parts of the solution space much sooner.

But the learning now goes both ways. Inference engineers have been forced by GPU constraints and economics to push continuous scheduling, reduced precision, speculative execution, accelerator-aware data movement and disaggregation of compute-bound and memory-bound work far more aggressively.

Database systems can borrow some of that thinking back.

The opportunity isn't to turn inference into a database. It's to bring two systems disciplines together that are increasingly solving the same class of problems from opposite directions.

INFERENCEDATABASE KERNELS
Continuous batchingFiner-grained dynamic scheduling
QuantizationPrecision as an execution decision
Speculative decodingAggressive speculate-and-verify execution
Prefill/decode disaggregationSeparate compute-bound and bandwidth-bound execution
GPU-aware executionOptimize explicitly around accelerator memory movement

3Where is the query optimizer for inference?

This may be the biggest opportunity.

This may also be where the two worlds finally converge.

Databases bring decades of cost-based optimization. Inference brings a new execution environment: heterogeneous accelerators, dynamic state, uncertain output lengths, precision choices, speculation, and radically different economics around compute and memory movement. The next inference optimizer shouldn't simply copy a database optimizer. It should combine what both disciplines have learned.

Databases evolved from simply executing queries to choosing the best physical execution plan.

DatabaseQuery
↓
Statistics
↓
Cost-based optimizer
↓
Physical execution plan
↓
Execution
Inference todayRequest
↓
Router / Scheduler
↓
Model
↓
GPU

No equivalent end-to-end optimizer yet

Smart routers started by answering which model or endpoint should handle a request. Then came policy, quality, latency, availability and cost.

The natural next step is bigger: how should this request physically execute?

Once you look at inference this way, model routing is only one decision inside a much larger physical execution plan.

The optimizer chooses jointly across model, precision, hardware, KV locality, cache tier, prefill/decode placement, batch class and speculation.

RouterWHERE?Model · endpoint · provider · region
SchedulerWHEN?Queue · batching · priority
OptimizerHOW?Model · precision · hardware · KV placement · cache tier · prefill/decode · speculation
Admission controlSHOULD IT RUN?Admit · defer · downgrade · reroute · reject
One runtime

Understands the request, state, hardware, SLA and economics, then chooses the physical execution plan.

From router to runtime: four decisions, each answering a different question, converge into one execution plan.

Today these decisions are spread across routers, schedulers, serving engines, cache managers and infrastructure policy. The architectural opportunity is to make them one planning problem.

To make those decisions, inference needs its equivalent of database statistics: prompt length, expected output length, prefix reuse probability, KV footprint, cache residency, GPU pressure, SLA, speculative acceptance rate, tenant priority and cost.

The optimizer should decide what enters the context too

Physical execution isn't only about choosing a model, GPU, precision or cache tier. It also includes deciding what information should be moved into the model's working context in the first place.

TodayRetrieve broadly
↓
Move data into context
↓
Let the model find what matters
Database instinctFilter at the source
↓
Project only what's needed
↓
Move the minimum required data
↓
Execute

Context stuffing resembles a full scan followed by late filtering. Database systems learned to push predicates toward the data and move only what execution actually needs.

AI runtimes should increasingly push filtering toward the data instead of pulling data toward the model.

That principle applies across retrieval, databases, search, MCP tools and enterprise APIs. If a source system can filter, aggregate or project before returning data, the runtime may avoid moving unnecessary information into an expensive context window.

That makes context construction another physical-plan decision:

What state is needed?Where should filtering happen?What should be retrieved?What should enter context?What should be computed instead?

The optimizer shouldn't only decide where inference runs. It should decide how much information needs to reach inference at all.

EXPLAIN INFERENCE

Databases didn't just optimize execution. They made the optimizer inspectable.

EXPLAIN tells an engineer why the system chose a particular plan. If inference becomes dynamically planned, enterprises will need the same thing.

HYPOTHETICAL — illustrative execution plan, not a benchmark
EXPLAIN INFERENCE  "Summarize Q3 escalations for account 4471"

Prefix cache   18,400 / 21,000 prompt tokens reused
               HBM, gpu-07

Prefill        2,600 tokens, chunked
               co-located with decode

Model          mid-tier
               FP8 weights, FP8 KV

Placement      gpu-07 (KV-local)

Alternative    gpu-12 (idle)
               +1.1s KV rebuild, rejected

Decode         ~400 output tokens predicted

Speculation    enabled
               estimated acceptance 0.7

SLA class      interactive
               TTFT < 500 ms

Est. cost      $0.0041
               vs. $0.019 without cache reuse

This is observability at the execution-plan level.

Why this model? Why this GPU? Why this precision? Why reuse this state? Why recompute instead of retrieve? Why speculate? Why did this request cost what it cost?

The optimizer could decide not to infer

The most important optimization may be not running inference.

A cached result, deterministic lookup, database query, search operation, smaller model or conventional computation may satisfy the request more cheaply and reliably.

A sufficiently capable runtime could choose among those paths rather than assuming every request must become an LLM call.

Once the runtime can make that decision, it isn't merely routing anymore.

It is optimizing AI execution.

4Inference companies may be hiring from too narrow a talent pool

Memory allocation. Buffer management. Cache replacement. Data locality. Scheduling. Admission control. Workload isolation. Compression. Statistics. Cost-based optimization. Prefetching. Distributed state. Resource governance.

These aren't new disciplines. They are database-kernel disciplines.

Inference companies obviously need ML researchers, CUDA engineers and distributed-systems experts. But engineers who built Oracle, MySQL, PostgreSQL, SQL Server, DB2, Teradata, SAP HANA and modern analytical engines spent decades solving this class of problem.

For database engineers looking at AI infrastructure from the outside: the hardware is new. Many of the systems problems aren't.

The bottom line

The inference industry has already rediscovered paging, caching, batching, compression, locality and workload specialization.

The next step is to rediscover the optimizer.

The next-generation runtime won't simply route a request to a model. It will understand the request, available state, hardware, SLA and economics, then construct the best physical execution plan across them.

  • Sometimes that means a different GPU.
  • Sometimes a different model.
  • Sometimes retrieving cached state.
  • Sometimes recomputing it.
  • Sometimes waiting.
  • And sometimes not calling an LLM at all.

That's why the next generation of inference infrastructure starts looking less like an API router and more like a database execution engine.

We spent forty years learning how to optimize execution around scarce compute, memory and data movement. Now we're rebuilding that intelligence around GPUs.

We've seen this class of systems problem before. Now we're solving it again, at GPU speed — and at GPU economics.

The database analogy doesn't stop at inference

Everything so far has been about the execution engine: memory, locality, scheduling, state reuse and optimization.

But once models become long-running agents that maintain memory, call tools and change real-world state, a different set of database problems appears.

Transactions. Concurrency. Durability. Recovery. Security. Workload management.

Inference is rediscovering the database execution engine. Agentic AI is beginning to rediscover the transaction manager.