Inference Cost
Inference cost measures the expense of running a trained model for a defined workload, with its unit and accounting boundary stated explicitly.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/inference-cost/
01What it means
Inference cost is the expense attributable to model execution, expressed per request, token, time interval, or other declared unit. Managed services may meter input, output, cached input, or provisioned capacity differently; self-hosted systems incur compute and operating costs. Cost per completed product task is a broader measure that also accounts for repeated calls and supporting work. Neither figure is meaningful without a consistent boundary.[1][2]
- Metered execution
- Input, output, cache, compute, or provisioned capacity
- Supporting work
- Retrieval, tools, retries, review, and operations if included
- Workload boundary
- Task type, period, volume, and currency
- Acceptable completion
- Quality threshold that prevents cheap failures looking efficient
Input, output, cached usage, repeated calls, reserved capacity, hosting, review, and retrieval can change the effective cost materially.
02Why it matters
A cheap model call can support an expensive feature when retries, long context, tool use, or failed tasks multiply consumption. Relating spend to acceptable outcomes helps teams evaluate usage limits, pricing, model selection, and the affordability of automation.
03When to use it
Measure inference cost when budgeting a model-based feature or comparing deployment options. Use task-level cost for product decisions and request-level measures for diagnosis. Do not treat current supplier rates as permanent or compare systems with different quality requirements.
04How it works
- Define the workload, period, currency, cost boundary, and criteria for an acceptable completed task.
- Record metered usage and effective rates, including repeated calls, caching rules, and any reserved capacity.
- Allocate hosting and operating costs consistently when comparing self-hosted and managed options.
- Calculate both execution cost and broader cost per acceptable task, separating review and retrieval costs where included.
- Segment by task length and customer behavior, stress-test heavy usage, and evaluate optimizations against quality and latency.
05A practical example
In an illustrative support workflow, 1,000 attempts consume $40 in model execution and $20 in retrieval and tool services. If 800 meet the acceptance criteria, model cost is $0.04 per attempt, while the declared $60 system boundary yields $0.075 per acceptable result. Human review would need to be added before estimating the full service contribution.
06Where it breaks
Allocation depends on workload and utilization, especially with shared or reserved capacity. Quality thresholds change the denominator. Average costs conceal expensive tails, and reduced token use may increase failure or review. This measure alone does not establish profitability.
Common mistakes
- Comparing token prices while ignoring token volume and repeated calls.
- Dividing only by successful calls while omitting the cost of failed attempts.
- Treating provider charges as the complete cost of serving a customer.
- Claiming savings without rechecking quality, latency, and review effort.
07Sources
- Cost optimizationAmazon Web Services · Official documentationCost optimization
- Agent Platform PricingGoogle Cloud · Official documentationAgent Platform Pricing