Pretraining

Choose the prediction target before choosing a corpus or budget. Autoregressive training predicts text in sequence; masked-language training predicts selected content from its context. These are distinct objectives, and experimental comparisons report different outcomes across architectures, objectives, and whether later task adaptation is included.

When planning from scratch, state the objective, the unit being predicted, and what evidence would count as useful transfer. Broad pretraining and later task adaptation are separate decisions: research across several domains and classification tasks found gains from a second, in-domain pretraining phase. Use a small example to check that the training signal matches the behavior you want to study.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  Read a concrete example of pretraining representations from unlabeled text using left and right context.

• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

  See a study of domain-adaptive and task-adaptive pretraining after broad-coverage pretraining.

A data plan should specify more than the corpus's total size: it should say which sources and domains contribute training tokens. The Pile paper describes one example corpus built from 22 diverse subsets and explicitly notes concerns for prospective users, making it a useful case study rather than a universal recipe.

For continued pretraining, state what new domain or task data is being added and what outcome will be evaluated. One study across biomedical and computer-science publications, news, and reviews found gains from a second in-domain phase on eight classification tasks. Treat such findings as evidence for a particular setting. Record the available compute budget, candidate model sizes and token counts, data mix, and evaluation plan so the allocation can be revised against measured results.

• Training Compute-Optimal Large Language Models

  Use its analysis of model size and token count under a given compute budget to frame allocation choices.

• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

  Review evidence on adding domain or task-specific pretraining after broad-coverage pretraining.

A scaling law is an empirical relationship between training resources and measured model performance. Scaling studies can help estimate how a measured loss changes as model size, dataset size, or training compute changes, and can inform how to allocate a fixed compute budget.

The allocation recommendation is evidence from a study, not a universal constant. One early study argued for relatively large models trained on less data; a later compute-optimal analysis recommended increasing training tokens alongside model size. When using either result, ask what was measured, what budget was fixed, and which training setup the conclusion describes.

• Training Compute-Optimal Large Language Models

  Compare a later study's allocation of model size and tokens for a given compute budget.

Scaling law

An empirical relationship between resources and measured model performance.