Inference Runtime
C++20 · Concurrency · AI Infrastructure
Adaptive AI Inference Runtime
Schedule, batch, route, and serve inference under bounded resources.
A C++20 single-host inference control plane with bounded admission, workload-aware scheduling, residency-aware routing, worker-local dynamic batching, model residency and eviction, plus optional real llama.cpp GGUF execution on Apple Metal.
01 — The Runtime Problem
The Runtime Problem
Once inference requests compete for bounded queue capacity, workers, batch opportunities, and resident model resources, “run the model” becomes a scheduling and resource-management problem.
ThroughputvsWaiting
Load balancevsModel residency
Memory pressurevsReload cost
Useful workvsOverload / cancellation
Requests
Runtime Policies
Workers / Models / Backend
02 — Request Lifecycle
Request Lifecycle
One asynchronous HTTP/programmatic path feeds a deterministic control plane: admit, schedule, route, batch, resolve residency, execute, and return a single terminal result.
Request states
Received
Queued
Running
First terminal transition wins.
- Shared request lifetime with mutex/CV state coordination
- One routing thread · one execution thread per Worker
- Asynchronous HTTP layer over the control plane
03 — Scheduling & Routing
Scheduling & Routing
Scheduling chooses which request is eligible next. Routing chooses which Worker receives it. They are separate policy layers.
Adaptive here means policy-driven runtime decisions, not a learned scheduler.
Request Scheduling
FIFO
Deterministic oldest-request baseline.
Workload Aware
Prefer the largest currently queued same-model group, then oldest group → lexical model ID → oldest request in group.
A bypass_count threshold promotes requests skipped too many times.
The scheduler returns one request. It does not build the batch.
Worker Routing
Round Robin
Baseline cyclic feasible-worker routing.
Least Loaded
Minimize queue_depth + active_count.
Residency Aware
Tier 0 — model already resident · Tier 1 — fits free estimated memory · Tier 2 — eviction required. Within the best tier: lowest load → lowest WorkerId.
Residency-aware selection
| Worker | Model resident | Fits free | Evict? | Load | Tier |
|---|---|---|---|---|---|
| Worker 0 | YES | — | NO | 3 | 0 |
| Worker 1 | NO | YES | NO | 0 | 1 |
| Worker 2 | NO | NO | YES | 0 | 2 |
SELECT → Worker 0
Residency is preferred before raw load tier. This is not GPU routing.
04 — Dynamic Batching
Dynamic Batching
Requests arrive individually. A Worker may wait briefly for compatible same-model requests, then finalize the batch before inference begins.
Worker-local dynamic static batching
This is not continuous batching.
Contiguous same-model wait
A1 → A2 → A3 → B1
Execute [A1 A2 A3]; B1 waits for the next batch
No non-contiguous reordering
A1 B1 A2 A3
[A1] · [B1] · [A2 A3]
Not [A1 A2 A3]
Concurrency 2
Baseline
4.20 req/s
p95 736.4 ms
mean batch 1.00
Adaptive batching
3.15 req/s
p95 1181.9 ms
mean batch 2.00
A real batch formed, but end-to-end performance got worse.
Concurrency 4
Baseline
2.17 req/s
p95 2849.9 ms
mean batch 1.00
Adaptive batching
4.35 req/s
p95 2105.4 ms
mean batch 4.00
Serving output-token throughput: 67.84 → 136.35 tokens/s (+101.0%)
Batch formation alone does not guarantee a performance win.
At concurrency 4, batching doubled median HTTP serving throughput. At concurrency 2, the same strategy made both throughput and p95 worse.
The result is workload-dependent, so the benchmark keeps both.
05 — Model Residency & Eviction
Model Residency & Eviction
Each Worker owns its ModelManager, logical resident set, estimated memory budget, and backend. Routing alone is not enough under memory pressure.
Memory accounting is estimate-based — not real Apple GPU / VRAM telemetry.
Model request
Already resident?
Fits estimated free budget?
Eviction policy
Unload victim(s)
Reserve
Load requested model
With LlamaCppBackend, load/unload corresponds to real GGUF context/model resources. Capacity decisions remain estimate-based.
LRU
- oldest last_used
- model ID tie-break
Cost Aware
Final lexicographic ordering — not a floating-point keep_score formula.
- lower estimated_load_cost
- lower use_count
- larger estimated_memory_bytes
- older last_used
- lexical model_id
06 — Real llama.cpp Execution
Real llama.cpp Execution
The control plane depends on an abstract IModelBackend, so deterministic synthetic testing and real GGUF inference share the same scheduler/router/worker architecture.
The default/control-plane test backend is synthetic. The optional llama.cpp backend performs real model inference.
IModelBackend
SyntheticModelBackend
LlamaCppBackend
llama.cpp
Qwen3.5-0.8B-Q4_0
Metal
CPU
25 / 25
Layers offloaded
Accepted Metal validation
07 — Reliability Under Pressure
Reliability Under Pressure
Bounded queues deliberately reject excess work rather than allowing unbounded growth or collapsing transport responsiveness.
Overload result
Bounded overload rejection — intentional backpressure under saturation.
3
HTTP 200
61
HTTP 503
0
Transport failures
503 rate: 95.3%
08 — Engineering Proof
Engineering Proof
152
CTest entries
148 passed · 4 skipped without AIRUNTIME_TEST_GGUF
19
Benchmark tool tests
133 / 133
ASan + UBSan
132 / 132
TSan
Excludes sandboxed HTTP integration runner
CI runs default + llama CPU jobs. The sanitizer matrix is local validation, not CI-gated.
Benchmark evidence SHA
dfc00d73033af62f1888bac014d5586086c5b30d
llama.cpp pin
9cc33944f9b7a44243618d5522adae357d7fdc90
Real model
Qwen3.5-0.8B-Q4_0