Glossary
Token
TOH-ken
The small chunk of text an AI model reads and writes, usually a word or part of a word.
A token is the unit of text an LLM works with. It’s usually a whole word, part of a word or a punctuation mark. As a rough rule, one token is about 4 characters of English text.
How it works
Before a model reads your prompt, a tokenizer splits it into tokens. unbelievable might become un, believ and able. The model then generates its answer one token at a time.
Why it matters
- Pricing: AI APIs charge per input and output token.
- Limits: the context window is measured in tokens.
- Speed: longer answers take longer to generate.
Code and non-English text usually take more tokens per word.