Dive into Deep Learning
Start with the architecture map
Read the overview, then compare how dense, convolutional, recurrent, and attention-based networks organize computation.
Start with: Dive into Deep Learning
RedLectern topic
Neural-network architectures
Convolutional · recurrent · transformer · state-space · dense and expert architectures.
For: Technical learners · developers.
After this page you can: Compare architectures by computation and task requirements.
Page reviewed
Part of: Artificial Intelligence › Machine Learning › Deep learning
About Neural-network architectures
Neural-network architectures differ in the structure they assume and the way they compute. Dense networks connect each layer broadly; convolutional networks use local filters with shared parameters; recurrent networks carry information across sequence steps. Transformers use attention to relate representations, while state-space models update a compact state over time. Mixture-of-experts architectures route inputs through selected subnetworks.
These choices affect which computations can run in parallel, how information moves through the model, and what structure the architecture makes easier to learn. A useful comparison asks what the input looks like, whether its order or spatial layout matters, how far dependencies reach, whether outputs are produced step by step, and what compute or memory limits apply. No architecture is a universal fit: compare models on the actual task, data, and deployment constraints.
Start here
Attention Is All You Need
Compare sequence architectures
Study the original Transformer paper alongside a structured state-space model paper to see different sequence-computation designs.
Start with: Attention Is All You Need
Learn the foundations
1.Architecture building blocksA convolutional network applies learned filters over local input regions, sharing parameters across positions.
A convolutional network applies learned filters over local input regions, sharing parameters across positions. A recurrent network carries information from one sequence step to the next. A Transformer uses attention mechanisms; in self-attention, positions in the same sequence exchange information. Encoder and decoder name common roles for components that respectively represent inputs and produce outputs.
Dense and expert architectures differ in how they use their subnetworks: dense computation applies the main network broadly, while a mixture of experts uses a gating mechanism to select a sparse combination of experts for an input. State-space models represent sequence processing through an evolving internal state. These terms describe useful patterns, not rigid categories: architectures can combine building blocks.
- Convolutional Neural Networks for Visual Recognition
Explains local connectivity, shared parameters, and convolutional layers.
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Describes a gating network that selects a sparse combination of expert subnetworks for each input.
Convolutional neural network
A network that uses shared local filters to detect structured patterns.
Recurrent neural network
A network that carries an internal state between sequence steps.
Transformer
A neural-network architecture that uses attention to update representations.
Attention
A computation that weights information according to its relevance to another representation.
Self-attention
Attention that relates positions within the same input sequence.
Encoder
A model component that converts inputs into internal representations.
Decoder
A model component that produces outputs from internal representations or context.
Mixture of experts
An architecture that routes inputs through selected specialized subnetworks.
Dense model
A model that uses the same main network rather than selecting expert subnetworks.
State-space model
A sequence model that updates a compact internal state over time.
Try it
For a model that classifies an image, a model that predicts a time series one step at a time, and a model that relates tokens across a sequence, name a plausible architecture for each and one structural feature it uses.
An image classifier could use a CNN with local shared filters; a step-by-step sequence predictor could use an RNN with a carried hidden state; a sequence model relating token positions could use a Transformer with self-attention.
What Is A TransformerWhat Is AttentionWhat Is Mixture Of Experts
2.How computation movesArchitecture comparisons become clearer when you trace how information and computation move.
Architecture comparisons become clearer when you trace how information and computation move. A recurrent network passes information between adjacent sequence steps, so its computation is naturally described step by step. The original Transformer paper presents an attention-based design that dispenses with recurrence and convolution, and reports that its models were more parallelizable in its machine-translation experiments.
State-space sequence models use a state update to represent sequence processing; structured variants can also be computed with sequence-wide operations. Mixture-of-experts models add routing: a gating network chooses which expert subnetworks process an input. These patterns describe computational structure, not a guarantee of lower wall-clock time. Implementation, sequence length, hardware, batching, and the task all affect practical cost.
- Efficiently Modeling Long Sequences with Structured State Spaces
Introduces a structured state-space sequence model and discusses its computation for long sequences.
Try it
A sequence has many positions. Contrast the information path in a recurrent model with self-attention, and state one possible computational implication.
A recurrent model passes information through adjacent steps using a carried state; self-attention relates positions within the sequence. The recurrent dependency is sequential across steps, while attention can compute relations across positions in parallel during training.
3.Choose by task and structureStart by describing the input structure and output requirement.
Start by describing the input structure and output requirement. Spatially arranged inputs can make local filters a natural choice; sequential inputs require a way to use order and carry or compare information across positions. For example, recurrent networks are used to model sequence dynamics, while convolutional networks are designed around local structure and shared filters.
Then compare the required dependency range, whether inference must be incremental, available training data, and the practical compute and memory budget. Transformers, recurrent networks, and state-space models make different trade-offs in how they relate sequence positions and carry context. Mixture-of-experts designs also add routing and specialized subnetworks. Test candidate architectures on the same data split and task-specific measure; do not infer practical speed or quality from the architecture label alone.
- Recurrent Neural Networks
Connects recurrent networks to sequence inputs and outputs, including language, time series, and video examples.
- From Fully-Connected Layers to Convolutions
Explains how locality and shared weights shape convolutional layers for image inputs.
Try it
Choose a starting architecture for (1) local patterns in images, (2) stepwise sequence processing with a compact carried state, and (3) a task where each input position should directly relate to other positions. Explain each choice in one sentence.
A CNN fits local image patterns because it uses shared filters over local regions. An RNN fits stepwise sequence processing because it carries state between steps. Self-attention is a candidate when positions should directly relate across the input sequence.
What Is A TransformerWhat Is AttentionWhat Is Mixture Of Experts
Latest
No recent updates are available.
Books
- Deep Learning
Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
Use it as a conceptual reference for feedforward, convolutional, and recurrent sequence networks. The authors provide the complete online text for free.