Streaming

Streaming, in the context of language models, means delivering a response incrementally as it is generated instead of waiting for the full text. The model produces one token at a time regardless, so streaming simply forwards each piece to the client as it appears. Nothing gets faster in total, but the time until the user sees something drops from several seconds to a fraction of one, and that perceived latency is what people judge. It also lets a user stop a response heading in the wrong direction, saving output tokens. Server-Sent Events is the usual transport because it is one-directional and works over plain HTTP, while WebSockets are used when the client also needs to send data mid-stream; the major SDKs expose streaming as an option on every completion call. Streaming complicates the parts around it. Errors can arrive after a partial answer is already on screen, structured output cannot be validated until the stream ends, moderation must run on fragments or after the fact, and proxies or CDNs sometimes buffer responses and defeat the effect entirely. Token usage totals also become available only once the stream closes.