Cost-per-task as a first-class metric and circuit breaker (not cost-per-token)
CORE INSIGHT
Cost-per-token is the wrong denominator now. What matters is cost per completed task. A "cheaper" model that burns 3x the tokens to finish the same job costs the same or more. Example: Claude 7.5 uses 30M tokens to complete a task at ~$500; a model 3x cheaper per token that still ends up at ~$500 for the same task is not actually cheaper. Sticker price per token is close to a vanity metric. Real efficiency = tokens-to-done x price-per-token.
This is more sophisticated than the industry's (and Alpha's earlier) "route you to the cheaper model" pitch. Model choice is only good if it lowers cost-per-completed-task including the retry tax.
WHAT TO ADD
- New metric + threshold: cost-per-task, alongside the existing per-agent monthly budget guardrail.
- Threshold breach = signal to go "back to the drawing board" and find where cost is bleeding (usually retries or runaway tool loops).
TASK BOUNDARY (the denominator)
- A task = one logical unit of work an agent was asked to complete; may span many LLM calls, retries, and tool calls across a session.
- Alpha already groups by session/workflow — reuse that scaffolding. Need either a customer signal for "this is one task" or infer from the run boundary. Getting this wrong makes the metric noisy.
KILLER USE: HONEST MODEL COMPARISON
- With cost-per-task, compare models on the real axis: run the same task through Model A vs Model B and show cost-per-completed-task side by side, including the retry tax. Sometimes the "expensive" model wins because it one-shots while the cheap one flails and retries. Only possible because Alpha is in-path seeing tokens-to-done. Strong demo + routing signal.
PROACTIVE VS REACTIVE (sequence them)
- Reactive first (easy, honest): task exceeds dollar threshold -> notify + surface the timeline so they see where it bled (retries, tool loops).
- Proactive next (harder, more valuable): kill/throttle the task mid-flight when projected cost crosses the line, before it finishes burning. A real cost-runaway circuit breaker — maps to the cost-runaway failure mode leaders are worried about.
RETRIES TIE IT TOGETHER
- Retries are likely the single biggest hidden cost-per-task inflator. Alpha already surfaces retry depth. Story: high cost-per-task -> drill in -> see retry storm -> fix via guardrail, prompt change, or model swap. That's the concrete "drawing board" loop.
CAUTION
- Don't auto-optimize too aggressively at first. Auto-killing a task or auto-swapping a model can break workflows in ways costing more than tokens saved. Recommend the fix, human approves, earn trust, then automate. Same human-in-the-loop-first principle as replay auto-optimization.
STRATEGIC FIT
- Ties to the leaders' narrative: Jensen (inference inflection, token spend exploding), the ROI-can't-be-proven problem (enterprises spending $9-19M/yr, costs buried in compute/storage lines), Nadella (paying twice / ownership). Alpha makes agent cost visible, cheaper, and owned — cost-per-task is the unit that makes "visible + cheaper" honest.
productclaude-connector · 26 Jul 2026
Replay & Simulation design — proxy-native cache-keyed replay via third API key
Design for replay/simulation without owning the agent framework's orchestration loop. Alpha stays a proxy; the framework (LangGraph/CrewAI/etc.) keeps driving the agent logic.
MECHANISM
- A third API key per agent, dedicated to simulation/replay (alongside the existing production and test keys). Agents are already the primitive and everything is attributable per key.
- Caller passes the simulation key plus a trace ID to replay.
- Alpha acts as a keyed cache over the recorded trace: when an incoming LLM request matches what was recorded for that trace/step, Alpha returns the previously generated LLM output instead of calling the model. On a mismatch, it falls through to a live LLM call and records the new output into a fresh replay trace.
- Divergence cascades naturally: change one input (e.g. flip a guardrail off) and that step's request no longer matches, so it goes live; its new output changes the next step's input, so that goes live too, and so on down the chain. No need to detect "everything changed."
- Replay = the special case where nothing diverged and every LLM call is a cache hit. Simulation = the case where one thing was changed and the run goes live from the divergence point forward. Same machinery yields both.
MATCHING KEY
- Key on the actual incoming user request only — not timestamps, not serialization noise. Deliberately simple to avoid false cache misses.
- Extend the key with step position in the trace sequence, so LLM call N serves call N's recorded output (a multi-step agent can have several calls where the user input is unchanged but intermediate context differs). Ordering likely already available from the agent timeline.
SCOPE / HONEST CAVEAT TO BUYERS
- Tool calls execute locally in the agent and can return different data each time — out of Alpha's scope. So even a no-change replay is not guaranteed byte-identical.
- Position it as: "deterministic on everything the model saw through us, live on everything we don't control." Do NOT sell it as perfect reproduction.
WHY THIS BEATS THE DURABLE-EXECUTION ROUTE
- Full deterministic replay would require owning the orchestration loop (Temporal/Restate/DBOS style). Alpha deliberately chose not to be a framework. This cache-keyed approach keeps Alpha entirely out of the framework's way while still delivering replay + simulation.
- Demos well: flip a guardrail, watch the run diverge live from that point — a memorable buyer moment.
- Neutral-in-path position: framework owners aren't neutral; neutral players aren't in the loop. Alpha is both neutral and in the request path.
RELATED CONTEXT
- Maps directly to the "own your traces / ownership is the alpha" thesis and the industry conversation (Nadella agent harness, enterprise control/ownership layer).
- Enterprise/regulated tier: true deterministic replay is a grow-into story, not needed for the 50–500 ICP near-term.
Arena must change from its current form to a guided cost-revelation flow. Step 1 — BASELINE: user pastes/connects their current prompt + model; Arena shows current cost per call, projected monthly cost at their volume, and output quality score. Step 2 — OPTIMIZE: Arena applies prompt optimization and other pillar improvements (context trimming, caching hints) and shows the same output quality with the cost difference highlighted. Step 3 — ROUTE: Arena routes to a cheaper model with full context preserved, shows side-by-side quality comparison proving same quality, and the total cost saving in large type ('You would save $X,XXX/month'). The aha is cumulative: each step stacks savings on a running meter. End state: one screen showing before/after monthly cost + quality parity + a 'get this on your real traffic' CTA into the $250/mo tier. This IS the PLG conversion engine — the free tool must scare and delight in under 10 minutes.
Arena must change from its current form to a guided cost-revelation flow. Step 1 — BASELINE: user pastes/connects their current prompt + model; Arena shows current cost per call, projected monthly cost at their volume, and output quality score. Step 2 — OPTIMIZE: Arena applies prompt optimization and other pillar improvements (context trimming, caching hints) and shows the same output quality with the cost difference highlighted. Step 3 — ROUTE: Arena routes to a cheaper model with full context preserved, shows side-by-side quality comparison proving same quality, and the total cost saving in large type ('You would save $X,XXX/month'). The aha is cumulative: each step stacks savings on a running meter. End state: one screen showing before/after monthly cost + quality parity + a 'get this on your real traffic' CTA into the $250/mo tier. This IS the PLG conversion engine — the free tool must scare and delight in under 10 minutes.