ACTIVE Systems

LLM Admission Gateway — Reserve, Commit, Expire

A rate limiter for inference traffic, where cost is tokens rather than requests and the dominant part of that cost is unknown at admission time. Three-phase protocol on Redis + Lua, with lease reclamation and SLO-aware shedding — and a benchmark that reports the capacity it can defend, not the one it can survive.

VIEW_SOURCE() LIVE_DEMO()

status ACTIVE

type Systems

started 2026-07-01

stack TypeScript Node.js Redis Lua PostgreSQL Docker Prometheus

// DESCRIPTION

A conventional API rate limiter charges every request one unit and decides once, at arrival. Neither half of that holds for LLM inference.

Why inference needs a different limiter

Cost is tokens, and it varies by three orders of magnitude. A one-line completion and a 200k-token context are one request each. A limiter counting requests lets a handful of large prompts saturate a GPU fleet while the counter still reads nearly empty.
Most of that cost is unknown at admission time. Prompt tokens can be counted; output tokens are only known when generation stops. The gateway must commit to a decision before it can know what it is deciding about.
Requests hold a GPU for seconds. So concurrency is a separate constraint from rate, and when the predicted wait already exceeds the caller's deadline, admitting is strictly worse than shedding — it occupies a slot for the whole wait and then delivers a response nobody is still waiting for.

The protocol, and why the third phase is not optional

reserve   hold budget for a request whose output length is not yet known
commit    reconcile against what generation actually consumed
expire    reclaim holds whose owner never came back

Without the lease in phase three, every crashed client permanently burns a slice of its tenant's budget. The tenant's effective quota decays monotonically toward zero with no error anyone can point at — because every individual decision was correct. That failure shape is the reason the protocol has three phases instead of two.

One atomic script per phase

Splitting refill / compare / deduct across round trips would let two gateway replicas each observe the same remaining budget and both admit, over-admitting by the number of replicas. Redis runs a script to completion before serving any other command, so each decision is serialised on the shard. Expiry is swept on the admission path in bounded batches rather than by a background job — a sweeper is another process to deploy, monitor and elect a leader for, and it returns budget on its own schedule instead of when someone needs it.

Rejections carry distinct status codes

Collapsing everything into 429 tells a client to retry when retrying is hopeless, and tells a proxy that a saturated backend is a client-side quota problem. Budget exhaustion is 429 with Retry-After; a full queue is 503 (server capacity, not client quota); an infeasible deadline is 503; and a request larger than the tenant's entire budget is 400 with no Retry-After, because retrying can never work.

Measured, with the spread reported

One replica, open-loop, full reserve+commit per request, five sweeps:

offered cycles/s	p99 (ms)	p99 range over 5 runs	spread
250	0.70	0.65–0.71	1.1×
1000	0.73	0.71–0.89	1.3×
2000	1.48	1.44–6.05	4.2×
2500	6.91	2.87–29.01	10.1×

The capacity claim is 1,000 cycles/s, not 3,000 — even though the service serves 2,999 of 3,000 offered. Above ~2,000 the tail moves more between two runs of the same binary than it does across the whole 250→2,000 range, so the measurement has stopped describing the service and started describing a shared machine. The spread column exists so that judgement is the reader's and not buried.

Adaptive reservation: 3.07× the burst from the same budget

Holding the caller's declared ceiling can never overrun, but strands budget against tokens usually never generated. Holding an estimate from recent output admits 57.5% of a 400-request burst against 18.8%, at an overrun cost of 1.83% of tokens used. The mechanism is measured on the real gateway; the magnitude is conditional on an assumed output-length distribution, and the README says so rather than quoting the ratio bare.

// HIGHLIGHTS

Three-phase reserve / commit / expire protocol — the lease is what stops quota decaying to zero through crashed clients, silently and with every decision correct
One Lua script per phase: splitting the steps would over-admit by the replica count
Five shed reasons on four distinct status codes; unsatisfiable deliberately carries no Retry-After
p99 under 1 ms to 1,000 admission cycles/s with the tail repeating to 1.3× across five runs
Capacity reported where the spread closes (1,000), not where the service still survives (3,000)
Adaptive reservation admits 3.07× of a burst versus worst-case, for 1.83% overrun

< BACK_TO_PROJECTS()