Inference

Inference is the stage where a trained model takes new input and produces an output, as opposed to training, where its weights are updated. Every chatbot reply, generated image, and recommendation is an inference pass. The cost profile differs sharply from training: training is a one-time capital expense, while inference is a recurring bill that scales with usage. For a large language model, inference has two phases. Prefill reads the prompt in parallel and is compute-bound; decoding produces one token at a time and is limited by memory bandwidth, which is why output tokens usually cost more than input tokens. Providers reduce that cost with batching, KV-cache reuse, quantization, and speculative decoding. Groq and Cerebras built custom chips aimed at fast decoding, and vLLM and TensorRT-LLM pursue the same goal on GPUs. Prompt caching, offered by Anthropic and OpenAI, skips recomputation of a repeated prefix. A common mistake is judging a deployment only by tokens per second. Latency to the first token, throughput under concurrent load, and cost per request often matter more to a product than raw speed on a single request.