Operations
Latency
How long the system takes to respond — often the difference between a feature people use and one they abandon.
Models are slow relative to ordinary software. A complex request can take many seconds, and an agent taking several steps can take much longer.
What helps:
- Streaming the answer as it is generated, so something appears immediately.
- Smaller models where the task allows.
- Being honest in the interface about longer-running work rather than showing a spinner and hoping.
Latency is a product decision as much as a technical one. Users tolerate waiting when they can see progress and understand why.
Related