AI Newsway
Back to glossary

A token is the smallest unit of text a language model reads and writes. Models do not see characters or words directly. A tokenizer first splits input into subword pieces drawn from a fixed vocabulary, usually built with byte-pair encoding, and each piece maps to an integer the model embeds as a vector. Common English words often become a single token, while rare words, code, and non-Latin scripts split into several. Korean and Japanese text therefore consume more tokens per character than English, which affects both cost and how much fits in a context window. Tokens are the unit of billing for the OpenAI, Anthropic, and Google APIs, priced separately for input and output, and they are the unit in which a context window is measured. They also govern speed, since generation happens one token at a time. Different model families use different tokenizers, so the same paragraph yields different counts on GPT, Claude, and Gemini. Estimating token count from word count is the usual mistake. For non-English text or JSON payloads the real number can be several times higher, which is why teams measure with the provider's tokenizer before setting limits.