Neural-network architectures concept

What is a transformer?

Reference

How do transformers work? They use attention to let sequence positions draw on information from other positions. A transformer is a neural network architecture: a design for processing information, often sequences such as text. Its attention layers build representations from relationships among sequence elements. The original transformer used both an encoder, which processes input, and a decoder, which produces output; other designs can differ.

Reviewed 2026-10-09

A sequence architecture built around attention

A transformer is a neural network architecture for processing sequences, including text. Its central operation is attention: a calculation that weighs information from different positions to produce an output. In self-attention, the positions being related come from the same sequence. The original transformer paper describes a design based on attention mechanisms, without recurrence or convolution.

An architecture is a design for how a neural network processes information; attention is a mechanism used within that design. The original transformer paired an encoder, which processes an input sequence, with a decoder, which produces an output sequence. Those two parts are one design choice, not the definition of every transformer.

How attention connects sequence elements

A transformer represents each sequence element as a vector, or list of numbers. For each position, attention compares a query with keys from positions it can consider, then uses the resulting weights to combine their values. The query, keys, and values are learned representations used in this calculation: the query is compared with keys, while values contribute to the output.

In self-attention, queries, keys, and values come from the same input sequence. Multiple attention heads let a layer combine information in different ways. Attention alone does not encode sequence order, so the original transformer added positional information to its representations. Transformer layers also include other operations, such as feed-forward networks, which process each position.

One architecture among several

Transformers are one kind of neural network architecture. Recurrent networks process sequential information one item after another; the original transformer paper presented its attention-based design as an alternative to recurrent and convolutional networks for sequence transduction, such as translating one sequence into another. Convolutional networks are another architecture family.

The word transformer names the overall architecture; attention names one of its mechanisms. Transformer designs may use an encoder, a decoder, or both. The original paper used both, with self-attention in each and attention between the decoder and encoder. This page describes the architecture, not a particular trained model or software library.

What the name does and does not mean

A transformer is not the same thing as a chatbot or other finished AI system. It describes an architecture; a trained model is built using an architecture and learned parameters, and an application may use a model as one component. So “transformer model” usually means a model whose architecture is a transformer, while “attention” refers to a calculation that can be used in that architecture.

Self-attention and attention are related terms, but they are not interchangeable. Self-attention relates positions within one input sequence. In the original encoder-decoder design, another attention calculation lets decoder positions use information from the encoder’s output. The distinction is about which sequence supplies the queries and which supplies the keys and values.

Questions people ask

What is a transformer in AI?

It is a neural network architecture that uses attention to process information, often sequences such as text.

Is a transformer the same as attention?

No. A transformer is an architecture; attention is a mechanism used within it.

Do all transformers have both an encoder and a decoder?

No. The original transformer used both, but the term refers to the broader architecture, which can be designed with an encoder, a decoder, or both.

Can transformers process images as well as text?

Yes. Images can be represented as sequences of patches, and transformer attention can combine information across those patches.

Learn more

Attention Is All You Need
Read the original paper’s explanation of the encoder-decoder design and its attention mechanism.
Transformers, 6.390 Introduction to Machine Learning
Follow a course chapter explaining query, key, value, self-attention, and transformer layers.

What changed

Sources

Part of Neural-network architectures: Architecture building blocks How computation moves