Neural-network architectures concept

What is a mixture of experts?

Reference

What is mixture of experts? A mixture of experts (MoE) is a neural-network architecture made of expert subnetworks and a learned router, or gating network, that selects which experts contribute to processing an input. This is called conditional computation: the active parts can vary with the input. In a sparse MoE, only a subset of experts runs for a given input, so total parameter count and computation used for that input are different measures; routing also brings coordination demands.

Reviewed 2026-10-09

A network with expert subnetworks

A mixture of experts (MoE) is a neural-network architecture with multiple smaller neural networks, called experts, and a learned gating network, often called a router. The router selects a sparse combination of experts to process an input, and the selected experts’ outputs contribute to the result. The experts have their own parameters and are trained together as part of the larger network. “Expert” names a part of the architecture; it does not mean a human specialist or a separate, independently built model. This design is one form of conditional computation: which parts of the network run depends on the input.

A router selects which experts run

In a routed MoE layer, the router evaluates an input representation and selects one or more experts. In language models, routing is commonly done for each token, a piece of text processed by the model. The chosen expert networks transform that token’s representation, and the model combines their results, often using routing weights. The number of selected experts and the routing details vary by design. Because only selected experts are evaluated, sparse routing can avoid computing every expert for that input. Routing decisions and moving inputs or results between experts can also add coordination costs.

How it differs from a dense model

A dense model uses its full network for every input token. A sparse MoE instead routes each token through only a subset of its experts. That means an MoE may have more total parameters than it activates for one token; total parameters and active parameters describe different things. Sparse activation can reduce computation for an input, but it does not mean the model has little to store: all expert parameters still need to be held in memory. Routing, load balancing, and communication between devices can make deployment more complex.

What “experts” means—and does not mean

The name can suggest a team of separate models, each assigned a human-readable subject such as maths or translation. In an MoE, experts are neural-network submodules with their own parameters, while a learned router chooses among them. Researchers have observed experts specializing in patterns such as syntax or semantics, but this does not mean each expert has a simple, fixed label. An MoE can be the whole model or a component inside a larger network, which may also contain ordinary dense layers.

Questions people ask

How do MoE models work?

A learned router selects one or more expert subnetworks for an input, often for each token in a language model. The selected experts process it, and their outputs are combined, often using routing weights.

What is the difference between dense and MoE models?

A dense model uses its full network for each token. A sparse MoE activates only a subset of its experts for each token, though all its expert parameters still need to be held in memory.

Are MoE experts separate models with assigned jobs?

Experts are subnetworks within a larger model, with their own parameters. They may specialize during training, but that does not mean each has a fixed, human-readable subject or job.

Does an MoE use fewer parameters than a dense model?

Not necessarily. An MoE can have many total parameters while activating only a subset for a token. Total parameters and the parameters active for one token are different quantities.

Learn more

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
The paper introduces expert networks, a gating network, sparse selection, and joint training in an MoE layer.
Experts backends
The documentation describes token routing, selected experts, and combining their outputs with routing weights.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
The paper explains how MoE models select different parameters for different inputs and discusses sparse activation.

What changed

Sources

Part of Neural-network architectures: Architecture building blocks How computation moves