Mathematics for Machine Learning
Start with the math behind machine learning
Begin with the mathematical foundations, then use the worked examples to connect the notation to machine-learning methods.
Start with: Mathematics for Machine Learning
RedLectern topic
Math for AI
The linear algebra · probability · calculus · optimization needed to understand AI.
For: Technical beginners
After this page you can: Read basic model equations and follow training explanations.
Page reviewed
Part of: Artificial Intelligence
About Math for AI
Math for AI brings together a few tools that recur in machine-learning explanations: linear algebra, probability, calculus, and optimization. The open textbook Mathematics for Machine Learning organizes its mathematical foundations around these subjects, then connects them to machine-learning problems such as regression and classification. This page offers a practical route through the ideas so you can read equations and explain what a training step is doing.
Start by identifying what the symbols represent and checking the shapes of the quantities in an equation. In computing references, a one-dimensional array is often called a vector, a two-dimensional array a matrix, and an array with more dimensions a tensor. Probability distributions describe possible outcomes and their probabilities; entropy summarizes uncertainty in a distribution. For a function of several parameters, the gradient collects its partial derivatives. Gradient descent uses the negative gradient to take a step that lowers a cost function.
You do not need to master every branch of mathematics before reading a basic model equation. Build fluency with small examples: multiply a matrix by a vector, compare simple probability distributions, and calculate a derivative before following an update rule.
Start here
Gradient descent, how neural networks learn
See gradient descent in a neural network
Connect a cost function, its gradient, and a downhill parameter update in a visual explanation.
Start with: Gradient descent, how neural networks learn
Learn the foundations
1.Read vectors, matrices, and tensorsVectors, matrices, and tensors are common names for numerical arrays with different numbers of dimensions in computing.
Vectors, matrices, and tensors are common names for numerical arrays with different numbers of dimensions in computing. Their shapes tell you how many entries they contain along each dimension. When reading an equation, check that the dimensions fit the operation: for example, matrix multiplication depends on the dimensions of both factors. A vector can represent an input or a set of parameters; a matrix can represent weights or a transformation. Cosine similarity compares two vectors using their normalized dot product, which corresponds to the cosine of the angle between them. These terms are useful for reading model equations, but the array terminology is a computing convention and does not cover every mathematical meaning of vector, matrix, or tensor.
- 18.700 Linear Algebra
The course calendar covers vector spaces, matrices, inner products, and eigenvalues.
- Pairwise metrics, affinities and kernels: cosine similarity
The documentation defines cosine similarity as the L2-normalized dot product of vectors.
Vector
An ordered collection of numbers representing a quantity or item.
Matrix
A rectangular grid of numbers used to represent data or transformations.
Tensor
A numerical array that can have any number of dimensions.
Probability distribution
A description of how probability is assigned across possible outcomes.
Gradient
The direction and rate of a function's change across its inputs.
Entropy
A measure of uncertainty in a probability distribution.
Cosine similarity
A measure comparing the directions of two vectors.
Try it
Let A = [[2, 0], [0, 3]] and x = [1, 2]. Compute Ax. What is the cosine similarity between [1, 0] and [0, 1]?
Ax = [2, 6]. The vectors [1, 0] and [0, 1] are perpendicular, so their cosine similarity is 0.
2.Use probability to describe uncertaintyA probability distribution assigns likelihoods to possible outcomes.
A probability distribution assigns likelihoods to possible outcomes. For a simple discrete distribution, list the probabilities and check that they sum to one. Entropy summarizes uncertainty: for two possible outcomes, a fair split has higher entropy than a heavily biased split. This helps you interpret probability and entropy in equations without treating either as a guarantee about what will happen. Start with small distributions and ask how much probability is concentrated on one outcome. The distinction is useful when reading model explanations that describe a range of possible outputs, or compare a distribution before and after an update.
- Mathematics for Machine Learning
Its mathematical foundations include probability and distribution.
- Lecture Notes on Statistics and Information Theory
The notes define entropy for distributions and show that a uniform distribution has maximum entropy on a finite set.
Try it
For two outcomes, compare the entropy of the distributions (0.5, 0.5) and (0.9, 0.1). Which is higher?
The distribution (0.5, 0.5) has higher entropy because its probability is spread evenly across the two outcomes.
3.Follow gradients and training stepsA training explanation can describe a cost as a function of model parameters, then show how the parameters change to reduce it.
A training explanation can describe a cost as a function of model parameters, then show how the parameters change to reduce it. For a function of several inputs, the gradient is a vector of partial derivatives; it describes local change in each input direction. In gradient descent, the negative gradient gives a downhill direction for the cost. The chain rule helps explain how a change at one stage of a composed calculation affects later outputs, and its derivatives make up the gradient used in backpropagation. When reading an update rule, identify the cost being minimized, the parameters being changed, and the direction of the update. This is enough to follow the purpose of a basic training step without deriving every formula.
- 18.657 Mathematics of Machine Learning
The course describes a mathematically rigorous introduction to machine-learning methods and their analysis.
- What is backpropagation really doing?
The lesson explains how backpropagation computes a gradient used to adjust network parameters.
Try it
For f(w) = (w - 3)^2, find the gradient at w = 1 and say which direction a gradient-descent step moves.
The derivative is 2(w - 3), so at w = 1 the gradient is -4. Gradient descent moves in the negative-gradient direction, toward larger w.
Latest
No recent updates are available.
Books
Mathematics for Machine Learning
Marc Peter Deisenroth, A. Aldo Faisal, Cheng Soon Ong · 2020
The authors present mathematical foundations alongside machine-learning examples that use them. Their site keeps the book PDF freely available.
- Pattern Recognition and Machine Learning
Pattern Recognition and Machine Learning
Christopher M. Bishop · 2006
Develops probabilistic graphical models and a Bayesian perspective. Microsoft Research provides the full book as a PDF.
