Foundation models

“Foundation model” describes a model trained on broad data that can be adapted to a range of tasks. It is a broader category than language models: examples and research areas also include vision, speech, proteins, molecules, and multimodal models. The term describes a model’s role in later systems, not a guarantee of quality or general ability.

A base model is the pretrained starting point before a particular instruction or task adaptation. Post-training covers later training stages; fine-tuning is one way to adapt an already trained model, and instruction tuning adapts behavior toward following instructions. A reasoning model is described by its use of additional computation for multi-step problem solving. “Small language model” is a relative size label; the term itself does not specify a universal cutoff.

• Transparency Note: Phi Silica on Non-Copilot+ PCs 10 min

  Provides an official example of how a maker defines the term small language model.

Foundation model

A broadly pretrained model adaptable to multiple downstream tasks.

Pretraining

Initial training that develops broadly reusable model capabilities.

Fine-tuning

Additional training that adapts an already trained model.

Post-training

Training after a base model is created to change behavior or capabilities.

Base model

A pretrained model before a particular instruction or task adaptation.

Instruction-tuned model

A model additionally trained to respond to instructions.

Reasoning model

A model designed or trained to devote computation to multi-step problem solving.

Small language model

A relatively small language model; the term does not specify a universal size cutoff.

Base Vs Instruct Models

Pretraining is an initial stage that learns from broad data; for language models, a common setup uses an objective such as predicting text or filling in missing text. Fine-tuning continues training on a smaller, task-related dataset to adapt the pretrained model. Instruction tuning is a kind of fine-tuning in which varied instructions are used to adapt the model toward following instructions across tasks.

Adaptation is not the same as building an application. Training changes the model; an application may instead place a model in a workflow with a particular interface, instructions, data, or tools. To evaluate a claim about a system, ask whether it describes the pretrained model, the later training, or the complete application. A model’s ability to transfer to a task does not by itself establish that it will perform reliably in every use.

• Introducing FLAN: More generalizable Language Models with Instruction Fine-Tuning 15 min

  Describes instruction tuning as fine-tuning on varied instructions and discusses generalizing to instructions not seen during training.

A broadly pretrained model can be adapted for different tasks, but broad training does not make every model interchangeable. Models may differ in their input and output types, training objectives, and downstream adaptations. The foundation model label can cover work in language, vision, speech, and other areas; a language model is only one part of that broader category.

Reusable models can support many applications, but defects in a shared foundation can carry into downstream systems. Evaluation therefore needs to match the actual task and system: a result for a model does not automatically establish the behavior of an application that adds instructions, tools, or other components. Treat claims about capabilities as bounded by the evidence and conditions that support them, and identify which layer is being evaluated.

• Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer 15 min

  Shows how one text-to-text framework can be applied to tasks including translation, summarization, question answering, and classification.