Tokenizers
Start with tokens and text
A free beginner-friendly lesson on how tokenizers break text into pieces and map those pieces to model inputs.
Start with: Tokenizers
RedLectern topic
Large language models
How language models represent inputs · learn patterns · generate language · fail.
For: Everyone · builders.
After this page you can: Explain tokens and context and recognize common LLM limitations.
Page reviewed
Part of: Artificial Intelligence › Foundation models
About Large language models
A large language model (LLM) processes language as tokens: pieces of text that a tokenizer maps to numerical identifiers. During training, the model adjusts numerical parameters to capture patterns in language. When generating, it uses the preceding context to estimate a continuation, produces tokens, and converts them back into text. This is why an LLM can continue a sentence, follow an example, or draft a response without storing a separate rule for every request. Start with what an LLM is, then see how tokens and tokenization work.
The context window limits how much input and generated output a particular model configuration can handle at once. A longer window can make more material available, but it does not guarantee that every detail will be used correctly. LLMs produce plausible continuations, which can include errors, missing information, or unsupported claims. Treat fluent wording as something to evaluate, not as evidence by itself. The context window guide and hallucination guide explain these limits. Builders can use the inference and reasoning topic to explore what happens during generation.
Start here
Talking About Large Language Models
Follow the language-model idea
Read a short paper that explains LLMs through token-sequence prediction and pattern completion.
Start with: Talking About Large Language Models
Learn the foundations
1.From text to tokens and backA tokenizer turns text into token identifiers the model can process.
A tokenizer turns text into token identifiers the model can process. A token may be a whole word, part of a word, or another piece of text; token boundaries depend on the tokenizer. The model's learned parameters shape how it processes those identifiers. During generation, it estimates a likely next token from the preceding context, then repeats that step to form a sequence. The generated token sequence is converted back into readable text. A context window sets a model and configuration's supported limit for input and output, so a conversation or document may use part of that capacity before a reply is generated.
Keep the terms distinct: parameters are the learned numerical values; weights are commonly used for those values; a checkpoint is a saved snapshot that can be loaded. A stated knowledge cutoff is about training-data recency, not a guarantee that every earlier fact is present. Explore tokens, context windows, and parameters versus weights.
- Tokenizers 15 min
A step-by-step introduction to converting text into numerical inputs.
- Model weights, parameters, and checkpoints 10 min
Clarifies how weights, parameters, and saved checkpoints relate.
Large language model
A model trained at substantial scale to process and generate language.
Token
A unit encoding model input or output, often a piece of a word.
Tokenizer
A procedure that converts text into a model's token identifiers.
Context window
The maximum context length supported under a stated model and configuration.
Parameters
Adjustable numerical values learned during training.
Weights
Learned coefficients, often used loosely to refer to all model parameters.
Checkpoint
A saved model or training state that can be loaded later.
Next-token prediction
Estimating the next token from preceding context.
Hallucination
Generated content presented as factual despite being false or unsupported.
Knowledge cutoff
A stated boundary for training-data recency, not a guarantee of complete knowledge.
Try it
A tokenizer splits “unhelpful” into “un”, “help”, and “ful”. How many tokens are shown, and why might another tokenizer give a different count?
Three tokens are shown. A different tokenizer can use different vocabulary pieces and splitting rules, so the same visible word may produce a different token sequence.
Tokens And TokenizationContext Window ExplainedParameters Vs Weights
2.How a model learns and continues patternsLanguage modeling treats text as sequences of tokens.
Language modeling treats text as sequences of tokens. During training, a model adjusts its parameters to make useful predictions about token sequences; a common objective is estimating what token comes next given the preceding context. At use time, the model generates a continuation one token at a time. Because it has learned patterns across examples, it can continue text in familiar forms or adapt to a pattern shown in the current prompt. That ability is pattern completion, not a guarantee that the output is correct or that the model has checked its claims against the world.
Try the same simple pattern twice, changing the final example, and observe how the expected continuation changes. For the underlying architecture, see the original Transformer paper; for a conceptual discussion of what prediction does and does not establish, read Talking About Large Language Models.
- Attention Is All You Need 30 min
The paper that proposed the Transformer architecture based solely on attention mechanisms.
Try it
Complete this pattern with one plausible answer: “red, blue, red, blue, red, ___”. Then explain what the model would be using to choose a continuation.
A plausible answer is “blue”. The continuation follows the repeating pattern in the preceding tokens; a different prompt or pattern could support a different continuation.
3.Where fluent output can failA fluent continuation can still be false, unsupported, or incomplete.
A fluent continuation can still be false, unsupported, or incomplete. Language models can produce statements that sound credible while containing factual errors, so wording alone is not evidence that a claim is true. A model's available context and stated training-data recency also limit what information it can use, and neither tells you whether a particular answer is reliable. When accuracy matters, check specific claims against suitable sources and look for evidence that directly supports them.
Try a verification habit: pick one factual sentence in a generated answer, identify what evidence would establish it, and check that evidence independently. Stanford HAI's AI Index discusses factual inaccuracies and hallucinations as an ongoing concern. Read why AI hallucinates for more on plausible but unsupported output.
- Talking About Large Language Models 30 min
Discusses why generated reasoning-like text can be unreliable and why output needs scrutiny.
Try it
A generated answer states, “The town library opened in 1912,” but gives no source. What would you check before repeating the claim?
Find a reliable source about the library, such as its official history or a local archive record, and confirm that it directly supports the opening year. Until then, treat the date as unverified.
Explore subtopics
Latest
No recent updates are available.
Books
- Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models
Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models
Daniel Jurafsky and James H. Martin · 2026
The authors’ 2026 online draft covers tokens, language models, embeddings, transformers, and pretraining. The authors make the manuscript available on their site for readers to use.
Build a Large Language Model (From Scratch)
Sebastian Raschka · 2024
Follow a code-level path through a GPT-style model and its training stages. The book expects intermediate Python skills and some machine-learning knowledge.
Hands-On Large Language Models: Language Understanding and Generation
Jay Alammar and Maarten Grootendorst · 2024
Use its visual explanations and examples to build an intuitive picture of tokens, embeddings, and transformer behavior. A university course lists it among its textbooks.
Natural Language Processing with Transformers, Revised Edition
Lewis Tunstall, Leandro von Werra, and Thomas Wolf · 2022
Use its example notebooks to explore transformer anatomy, text generation, and training transformers from scratch. A university NLP course lists it as a recommended textbook.