HOME / BLOG / When Should a Pipeline Pay …
MARKDOWN

> When Should a Pipeline Pay for an LLM Call?

|
~2min READ
streaming ECML PKDD optimal stopping LLM uncertainty systems
// Pairing a cheap fast model with an expensive LLM is now a standard pattern, and the trigger is almost always a hand-tuned threshold. Treated as a sequential stopping problem instead, the classical trigger families all turn out to be the same object.

Streaming inference pipelines increasingly pair a lightweight fast model with an LLM that supplies semantic understanding at substantial cost. Almost every such system has to answer the same question:

When is this observation worth an LLM call?

In practice that is a threshold somebody tuned. The question has had very little formal treatment, which is odd given that it is the knob controlling the entire cost profile of the system.

Casting it as sequential stopping

Treat it as a risk-based sequential stopping problem. A trigger policy π fires when a risk functional R(H_t) over the observation history crosses a threshold θ. That framing is enough to prove six things:

  1. A minimum inter-event time bound — the policy cannot chatter.
  2. Threshold policies are optimal, via smooth pasting.
  3. Approximate SPRT guarantees when the parameters are estimated rather than known.
  4. O(√(T log T)) regret on stationary streams, degrading gracefully to O(√((C_T+1)·T log T)) under C_T changepoints.
  5. O(1/√T) convergence for online gradient descent on adaptive thresholds.
  6. A calibration-to-miss-rate transfer inequality — what your calibration error costs you in missed events.

The families collapse

The part worth the paper: event-triggered control, optimal stopping, SPRT, CUSUM and Bayesian triggers are not five competing approaches to pick between. Each is a special case of one risk functional under a particular choice of R.

That changes the engineering question from "which trigger family should I use" to "what is my risk functional, and what does that choice imply". The second question has an answer you can derive.

Real LLM calls, not simulated costs

Evaluated on turbofan degradation data (CMAPSS) with actual LLM invocations. This matters more than it sounds: a simulated cost model lets you assume away the failure modes that dominate in practice — the call that returns something ungrounded, the call that is slow enough to invalidate the decision it was supporting.

So the assumptions get verified rather than assumed, the risk-function design gets ablated, six baselines are compared (including a RouteLLM-style router and contextual bandits), and both cost sensitivity and LLM failure modes are analysed rather than excluded.

Lesson: when the expensive component is a language model, a cost model that does not call it is measuring a different system.

Where this sits

This is the same boundary problem that shows up in the Titans engine from the other direction. There, six orders of magnitude between a nanosecond fast path and a millisecond model forces the model out of the request path entirely, and the interesting question becomes what a stale advisory is still worth. Here the model is in the path and the question is when to pay for it. Both are the same trade priced differently.

Uncertainty-Aware Sequential Decision Rules for Event-Triggered LLM Invocation in Streaming Systems — ECML PKDD 2026, Research Track. arXiv:2607.13048 · code · Zenodo.

BACK_TO_BLOG()