Neural networks
I'm new to deep learning
Start with the neural-network module, then trace one training update from prediction to parameter change.
Start with: Neural networks
RedLectern topic
Deep learning
Neural-network training · optimization · representations · practical limitations.
For: Technical learners
After this page you can: Explain and train a small neural network.
Page reviewed
Part of: Artificial Intelligence › Machine Learning
About Deep learning
Deep learning is machine learning with neural networks that use multiple processing layers. A network transforms inputs through layers of calculations; during training, its adjustable parameters are updated to reduce a loss function. Activation functions let layers represent nonlinear relationships, while backpropagation calculates gradients that an optimizer can use to adjust parameters.
To learn the subject, follow one small model from input to prediction: define a task and loss, run a forward pass, update parameters, and check performance on data held out from training. The training loss alone does not tell you whether the model will generalize: an overfit model can perform well on its training examples and poorly on new ones. This page introduces the training loop, learned representations, and practical checks. The direct child page, Neural-network architectures, compares network designs; How neural networks learn is the topic’s concept page.
Start here
Exercises: Neural networks
I want to practice training
Use the training exercises to connect loss curves and parameter updates to model behavior.
Start with: Exercises: Neural networks
Learn the foundations
1.How a neural network learnsA neural network is a composition of calculations: layers transform the current values, and activation functions can introduce nonlinear transformations betwee…
A neural network is a composition of calculations: layers transform the current values, and activation functions can introduce nonlinear transformations between stages. For a training example, the model makes a prediction and a loss function measures its discrepancy from the target. Backpropagation computes gradients of that loss with respect to the parameters; gradient descent uses those gradients to update the parameters. The learning rate controls the update scale, while batch size determines how many examples contribute to an update. An epoch is one pass over the defined training set.
Short practice: Train a small network on a simple labeled dataset. Record training and validation loss over several epochs. If training loss falls while validation loss rises, treat that as evidence to investigate overfitting rather than as proof that the model is improving on new examples.
- Training using backpropagation
Explains how gradients support training multilayer networks and describes vanishing and exploding gradients.
- Interpreting loss curves
Provides exercises for reading training behavior from loss curves.
Deep learning
Machine learning that uses neural networks with multiple processing layers.
Neural network
A model made of connected layers of adjustable calculations.
Layer
A stage of computation in a neural network.
Activation function
A function that transforms intermediate values, often adding nonlinearity.
Loss function
A mathematical function that measures prediction loss, which training typically aims to minimize.
Backpropagation
A method for calculating parameter gradients through a model’s chain of computations.
Gradient descent
An iterative method that uses gradients to update parameters in an effort to reduce an objective.
Learning rate
A setting that controls how strongly gradient descent adjusts parameters on each iteration.
Batch size
The number of examples processed in a batch for a training iteration.
Epoch
One traversal of the defined training set, with each example processed once.
Try it
A binary classifier predicts 0.8 for a positive example. If the target is 1, what should happen to the loss when the prediction moves closer to 1? Name one training setting that affects how parameters are updated.
For a suitable classification loss, the loss should decrease as the prediction approaches the target. The learning rate affects update strength; batch size determines how many examples contribute to an update.
2.What layers representA hidden layer transforms its inputs into intermediate values that later layers can use.
A hidden layer transforms its inputs into intermediate values that later layers can use. These learned representations can make a task easier for the final prediction layer by encoding useful patterns from the training objective. An embedding is one example: it represents an item as a vector, and a neural network can learn the embedding as part of training. The representation is shaped by the task and data used to train the model; it should not be assumed to capture every property that matters for another task.
Short practice: Choose a layer in a trained classifier and describe what its output vector represents in the context of the task. Compare two examples using that representation, then ask whether the similarity you observe is relevant to the intended prediction. Treat the description as a testable interpretation, not a direct explanation of the model’s reasoning.
- Embedding space and static embeddings
Introduces embeddings as vector representations and describes how items can relate in an embedding space.
- Deep Learning, chapter 15: Representation Learning
An open textbook chapter devoted to representation learning.
Try it
Take three examples from a classification task and compare their hidden-layer representations. What similarity do you observe, and what evidence would show whether it helps the target task?
Describe the observed similarity and compare it with the examples’ labels or task outcomes. A useful representation should be judged by its effect on the intended task, not by visual grouping alone.
3.What training results do and do not showA low training loss shows that the model fits the training examples under the chosen loss; it does not by itself show that predictions will work well on new da…
A low training loss shows that the model fits the training examples under the chosen loss; it does not by itself show that predictions will work well on new data. Overfitting occurs when a model performs well on training data but poorly on new examples. Comparing training and validation loss can help reveal this pattern, though it does not explain every cause. The training set may fail to represent real data, and a model may be too complex for the available evidence. Deep networks can also encounter optimization problems such as vanishing or exploding gradients, which can make learning slow or unstable.
Short practice: Sketch training and validation loss curves where training loss keeps falling but validation loss rises. State what this pattern suggests, then name one additional check you would make before changing the model.
- Interpreting loss curves
Uses exercises to distinguish common training and validation loss patterns.
- Training using backpropagation
Describes vanishing and exploding gradient problems in neural-network training.
Try it
Training loss falls across epochs while validation loss rises. What does this pattern suggest, and what is one check you would make before changing the network?
It suggests overfitting. Check that the training, validation, and test sets are suitable and comparable, and inspect the examples before deciding on a model change.
Explore subtopics
Latest
No recent updates are available.
Books
Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
Follows neural networks from feedforward models through training optimization and practical methodology. Its complete online text is free on the authors’ site.
- Understanding Deep Learning
Understanding Deep Learning
Simon J. D. Prince · 2023
Pairs mathematical explanations with a focused treatment of training and model evaluation. The author’s book site provides a free online version.
Dive into Deep Learning
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola · 2023
Works through neural-network concepts alongside implementations, including backpropagation, numerical stability, optimization, and generalization. The text is freely available online.