Cost and latency

A restaurant doesn't cut costs by buying worse ingredients for every dish. It sends simple orders to the line cook and complex ones to the head chef, pre-chops what every dish shares, doesn't plate food nobody eats, cooks things in parallel, does bulk prep overnight when it's cheaper, and watches the one kitchen station whose bills suddenly spike.

Every lever in this lesson is one of those moves. The measure that matters is cost per successful task, not cost per call: a cheap attempt that fails still has to be paid for, and so does fixing it.

All prices in this module are illustrative constants chosen for round arithmetic (a "large" model at $5 per million input tokens and $25 per million output tokens; a "small" one at $1 and $5). They show the shape of the trade-offs; check your provider's current price list for real numbers.

Everyday picture A taxi that charges one rate for the distance to your pickup and a higher rate for the ride itself. Models charge per token (a word piece, about 4 characters of English): one rate for tokens you send (input) and a higher rate for tokens the model writes (output), because writing happens one token at a time (see primer.ml.inference).

Worked example A large-model call with 10,000 input tokens and 500 output tokens: 10,000 × $5 / 1,000,000 = $0.05 for input, plus 500 × $25 / 1,000,000 = $0.0125 for output, total $0.0625. If 8,000 of those input tokens are a stable prefix served from the prompt cache (reused work from an earlier identical beginning, billed here at 10% of the input price), input becomes 8,000 × $0.50/M + 2,000 × $5/M = $0.014, and the call costs $0.0265, 58% less.

n_in, c, n_out = 10_000, 8_000, 500
p_in, p_out, rho = 5, 25, 0.1
cost = (c * rho * p_in + (n_in - c) * p_in + n_out * p_out) / 10**6
round(cost, 4)  # → 0.0265

Two facts fall out: output tokens cost several times more than input (ask for concise answers), and a cached prefix is nearly free (put stable content first; see primer.agents.context). Caches usually charge a small premium the first time a prefix is written; this module ignores it for simplicity.

In code: request_cost is the formula above, reading each model's Price from PRICES.

Everyday picture A hospital triage nurse: sprained ankles go to the nurse practitioner, chest pains to the cardiologist. Nobody sends every patient to the most expensive specialist.

Worked example The sample workload is 10 tasks: 7 simple (classify a ticket, extract a date) and 3 complex (plan a migration). All on the large model it costs $0.671. Routing the 7 simple ones to the small model, which is 5x cheaper per token, brings it to $0.314: 53% saved, with the hard tasks still on the strong model.

flowchart LR
  T[Incoming task] --> R{Router<br/>rules or a small classifier}
  R -->|classify, extract,<br/>format, short| S[Small, fast model]
  R -->|plan, analyze,<br/>multi-step reasoning| L[Large model]
  S --> O[Result]
  L --> O

In code: route is the rule-based router; workload_cost prices a list of WorkItem tasks with any levers switched on; routing_savings compares all-large against routed on SAMPLE_WORKLOAD.

Everyday picture A receptionist who's been asked "what's the wifi password?" a hundred times just answers from memory. That's an exact response cache: the same question (ignoring case and spaces) returns the stored answer with no model call. A semantic cache goes further and answers similar questions from memory, using embeddings (vectors where closeness means similar meaning, see primer.ml.embeddings) to decide "similar". That's where the receptionist starts giving the vacation-policy answer to someone asking about sick leave.

Worked example With the toy embedder in primer.common.embedder:

The two near-misses score higher than a genuine paraphrase. No single threshold separates them. Two cheap guards help: identifiers and numbers must match exactly (ERR-4012 ≠ ERR-4013), and negation must match ("cancel" ≠ "don't cancel"). They can't catch "sick" vs "vacation", which share a topic and differ in a single word.

flowchart TD
  Q[New question] --> E{Exact match<br/>in response cache?}
  E -->|yes| A1[Return stored answer]
  E -->|no| V[Embed and find the<br/>nearest stored question]
  V --> T{Similarity above<br/>threshold?}
  T -->|no| M[Call the model]
  T -->|yes| G{IDs, numbers and<br/>negation identical?<br/>entry not expired?}
  G -->|no| M
  G -->|yes| A2[Return stored answer]
  M --> S[Store the new answer]

In code: ResponseCache is the exact cache. SemanticCache is the semantic path: SemanticCache.lookup skips expired entries and entries whose identifiers or negation differ, then returns the nearest survivor, and SemanticCache.get applies the threshold. semantic_cache_sweep draws the figure from the labelled pairs in CACHE_PAIRS.

Everyday picture Don't photocopy the whole binder when the colleague needs one page. Compress tool results to the fields the next step needs (primer.agents.context.compress_tool_output), send the top few reranked chunks instead of dozens, remove repeated boilerplate from system prompts, and ask for concise output, because output is the expensive direction.

In code: workload_cost models trimming as cutting each task's tool output to 500 tokens, and concise output as cutting long answers by a third.

Everyday picture Boil the pasta while the sauce simmers. Cooking them one after the other takes the sum of the times; together, the time of the slowest.

Worked example Three independent lookups of 100 ms each: sequentially about 300 ms; concurrently about 100 ms.

sequenceDiagram
  participant A as Agent
  participant T1 as Weather API
  participant T2 as Calendar API
  participant T3 as CRM API
  Note over A,T3: Sequential: about 300 ms
  A->>T1: call
  T1-->>A: result
  A->>T2: call
  T2-->>A: result
  A->>T3: call
  T3-->>A: result
  Note over A,T3: Parallel: about 100 ms
  par
    A->>T1: call
  and
    A->>T2: call
  and
    A->>T3: call
  end
  T1-->>A: result
  T2-->>A: result
  T3-->>A: result

In code: run_sequential awaits each call before starting the next; run_parallel starts them all with Python's asyncio gather and waits once.

Streaming (showing tokens as they're generated) doesn't reduce total time either, but users see progress immediately, which changes how fast the system feels.

Everyday picture Sending the laundry out to be done overnight at half price instead of waiting at the express counter. Providers offer batch APIs: submit many requests, get results within hours (often within 24), typically at about half the price. Use them for anything no one is waiting on: nightly evals, backfilling document processing, bulk classification.

In code: batch_cost prices a list of requests at the batch discount.

Everyday picture A prepaid card with a hard limit, and a bank that texts you when a purchase looks nothing like your usual spending.

A task budget caps steps and tokens per task, so a confused agent stops instead of looping all night. A tenant spend cap (a tenant is one customer organisation on a shared platform) stops one customer's runaway usage from becoming a surprise invoice. An anomaly alert fires when a task uses far more tokens than usual:

import statistics
history = [1000, 1100, 900, 1050, 950, 1000]
mu = statistics.mean(history)
# divides by 6 - 1, as above
sigma = statistics.stdev(history)
mu, round(sigma, 1)  # → (1000, 70.7)
z = 3
# the alert line
round(mu + z * sigma)  # → 1212
# alert?
10_000 > mu + z * sigma  # → True

In code: TaskBudget.charge counts each step's tokens and raises BudgetExceeded at either limit; TenantSpend.record adds a finished task's cost to its tenant's total and returns a cap alert or an anomaly alert (the formula above).

Everyday picture A cheap printer that jams on 40% of pages isn't cheap once you count the wasted paper and the time spent clearing jams.

def per_success(c, p):
    # retry until it works
    return c / p
def per_task(c, p, h=2.00):
    # a person fixes each failure
    return c + (1 - p) * h
round(per_success(0.002, 0.60), 4), round(per_success(0.010, 0.95), 4)  # → (0.0033, 0.0105)
round(per_task(0.002, 0.60), 3), round(per_task(0.010, 0.95), 3)  # → (0.802, 0.11)
# small vs. large, with cleanup
round(per_task(0.002, 0.60) / per_task(0.010, 0.95))  # → 7

In code: cost_per_success_with_retries is and cost_per_success_with_cleanup is .

Order the levers so the ones that can't hurt quality come first:

1. Prompt caching: reorder the prompt so the stable prefix is reused. No behaviour change.
2. Trim tokens: compress tool results and send fewer chunks.
3. Concise output: ask for shorter answers where length adds nothing.
4. Route easy tasks to the small model: the first lever that can change quality, so it's validated with evals before and after.
5. Batch the non-interactive share.

In code: five_x_plan switches the levers on one at a time, in this order, and reports the cost and cumulative reduction after each.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.