Operations
Inference
Running a model to get an answer — the thing you pay for, as opposed to training it.
Training a frontier model costs hundreds of millions and is done by the provider. Inference is what happens each time you use it, and it is what appears on your bill.
The practical consequence: AI features have a variable cost per use, unlike most software where serving one more user is nearly free. A feature that is cheap in a pilot can be expensive at scale.
This is why cost belongs in the design from the start — routing simple work to cheap models and reserving expensive ones for where accuracy pays for itself.
Related