Token
The unit of text a model processes. Roughly a word fragment, and the thing context windows, pricing and limits are all counted in.
In depth
What it really means
Models do not read characters or words. They read tokens, which are common character sequences learned from the training data. Frequent words are one token. Rare words, technical terms and made-up product names split into several.
The practical implication is small but real: an invented product name spelled unusually costs more tokens and represents less cleanly than a normal word. It is one small argument against creative spelling in brand names.
How it works
- Text is split into tokens using a learned vocabulary.
- Each token maps to an ID the model can process.
- Tokens fill the context window up to its limit.
- The model predicts one token at a time when generating.
English text: 1 token is roughly 4 characters, or about 0.75 words
So 1,000 words is roughly 1,300 tokens. A 2,000-word article is roughly 2,600. Code, non-English text and unusual spellings tokenize less efficiently and use more tokens per word.
Pros & cons
Pros
- Knowing token counts lets you estimate what fits in a retrieval context.
- It makes API costs predictable when you build tooling.
- It explains why concise pages behave better in retrieval.
Cons
- It is an abstraction most marketers never need to touch directly.
- Token counts vary between models, so estimates are approximate.
- Over-focusing on it distracts from content that actually matters.
Common mistakes
- Confusing tokens with words when estimating limits, which undercounts by about a third.
- Assuming token limits apply per page rather than across the whole context.
- Creative brand spellings that tokenize badly and represent inconsistently.
Best practices
- Estimate token count as words divided by 0.75 when sizing content for retrieval.
- Keep pages concise so more of the relevant part fits in a retrieval budget.
- Prefer conventional spelling for product names.
- Use a tokenizer tool when you need an exact count for API work.
FAQs
What is a token in AI?
The unit of text a model processes, roughly a word fragment. Context windows, API pricing and model limits are all counted in tokens.
How many tokens is a word?
In English, about 1.3 tokens per word, so 1,000 words is roughly 1,300 tokens. Code and non-English text use more.
Why do tokens matter for content?
They determine how much of your page fits into a retrieval context. Concise pages get more of their relevant content into the window.
Does my brand name affect tokenization?
Slightly. Unusual spellings split into more tokens and represent less consistently. It is a minor argument for conventional spelling.
Keep reading
Related on LymLyt
Beyond LymLyt
Further reading
Want this working on your site?
We build the content behind the term, ranked in search and cited by AI.
Book a 30-min call →