Large language models concept

Tokens and tokenization

Reference

AI tokens are the pieces of text that a language model processes. A token might represent a character, part of a word, a whole word, or punctuation; it is not a fixed unit like a word or syllable. Tokenization is the process that divides text into tokens and maps them to identifiers the model can use. The exact boundaries depend on the tokenizer, so the same text can have different token counts.

Reviewed 2026-10-09

A token is a text piece, not a word

An AI token is a piece of text a model processes. It can be a character, part of a word, a whole word, or punctuation. A tokenizer is the tool that divides text into these pieces; tokenization is that dividing process. The resulting pieces are associated with token IDs, which are numbers used to represent them. Tokens are therefore a model’s text units, not a standard measure of words.

Many tokenizers use pieces smaller than words. A common word may stay whole, while a less common or longer word may split into several familiar pieces. Spaces and capitalization can affect the boundaries too. So “tokens vs words” is not a one-to-one comparison: a sentence with ten words might have more or fewer than ten tokens. Counts also depend on which tokenizer and encoding are used.

How text becomes tokens and back

First, a tokenizer may normalize the text, for example by mapping equivalent character forms to a standard form. It may then split text into initial units before its tokenization model finds the final pieces. Some systems use subword pieces: frequent words can remain intact, while rare words can be divided into smaller pieces. The exact steps and boundaries depend on the tokenizer.

Next, the tokenizer maps each piece to an integer token ID, producing a sequence the model can process. When the model generates text, it produces output tokens; a decoder turns those pieces back into readable text. In short: text goes in, a tokenizer makes pieces and IDs, the model processes or generates a sequence, and decoding turns generated pieces back into text.

Why token counts can mislead

A token count is not a word count. The same text can produce different counts with different tokenizers or encodings, and languages can have different relationships between words and tokens. A rough English estimate, such as about three-quarters of a word per token, is only a rule of thumb; it cannot tell you the exact count for a particular sentence.

Small text changes can alter token boundaries. A leading space or a capital letter may change how a word is divided, and unusual spellings may split into several pieces. Also, counting only the visible text may miss tokens used for message structure, tools, images, or files in a full request. For an exact count, check the tokenizer or request format that applies to the system you are using.

Check your understanding

Try predicting how a tokenizer will divide “tokenization,” “the,” and the same word with a leading space. Then inspect each string with a tokenizer tool and compare the pieces and counts. The point is not to memorize one answer: a tokenizer’s boundaries depend on its encoding, so check the specific tokenizer when an exact count matters.

To test the idea, compare a word count with the token count for a short sentence. Ask yourself whether each token is a whole word, whether spaces affect the split, and whether the count would necessarily stay the same with a different tokenizer. If you can explain why those counts need not match, you understand the central difference between words and tokens.

Questions people ask

Are tokens the same as words?

No. A token can be a whole word, part of a word, a character, or punctuation. One word may use several tokens, and some tokens include spaces.

Why does the same text have different token counts?

Token boundaries and IDs depend on the tokenizer and encoding. Spelling, spaces, capitalization, and language can also affect how text is divided.

Is there a reliable number of words per token?

There is no exact conversion. A rough English estimate is about three-quarters of a word per token, but it varies with the text and tokenizer.

How can I see how a sentence is tokenized?

Use a tokenizer tool to inspect the pieces for your text. For an exact count in a particular system, check its tokenizer and whether the count includes request structure or attached content.

Learn more

Tokens | OpenAI API documentation
A concise explanation of token pieces, token counts, and a tokenizer tool for inspecting text.
Summary of the tokenizers
An illustrated account of subword tokenization and how rare words can be split into known pieces.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing
A primary paper introducing SentencePiece as a language-independent subword tokenizer and detokenizer.

What changed

Sources

Part of Large language models: From text to tokens and back How a model learns and continues patterns