Glossary

Token

TOH-ken

The small chunk of text an AI model reads and writes, usually a word or part of a word.

A token is the unit of text an LLM works with. It’s usually a whole word, part of a word or a punctuation mark. As a rough rule, one token is about 4 characters of English text.

How it works

Before a model reads your prompt, a tokenizer splits it into tokens. unbelievable might become un, believ and able. The model then generates its answer one token at a time.

Why it matters

  • Pricing: AI APIs charge per input and output token.
  • Limits: the context window is measured in tokens.
  • Speed: longer answers take longer to generate.

Code and non-English text usually take more tokens per word.