Neural-network architectures concept

What is attention?

Reference

Attention in AI is a calculation that combines information from different input positions, weighting each contribution according to how its key matches a query. The values are the information being combined. In self-attention, queries, keys, and values come from the same sequence, so positions can draw on other positions in that sequence. Attention is a building block used in transformer architectures, not the whole architecture.

Reviewed 2026-10-09

What attention means

Attention maps a query and a set of key-value pairs to an output. A query is the vector used to compare against keys; a key is a vector used in that comparison; and a value is the information that can contribute to the output. These names describe parts of a mathematical calculation, not literal questions or labels. In neural networks, the query, key, and value vectors are formed from representations of the input.

The calculation gives each value a weight based on how compatible its key is with the query, then adds the weighted values together. A greater weight means that value contributes more to this particular output. For scaled dot-product attention, the calculation compares a query with keys using dot products, scales the scores, and applies softmax to turn them into weights. This is one common form of attention; the general idea is to combine values according to query-key compatibility.

How self-attention works

Self-attention applies attention within a sequence: the queries, keys, and values come from the same sequence of representations. For each position, the operation compares that position’s query with keys from available positions, then uses the resulting weights to combine their values. The output for a position can therefore carry information gathered from other positions. This is how a sequence model can make a representation depend on more than the current position alone.

A mask can limit which positions are available in a calculation. In the Transformer paper’s decoder, self-attention is masked so a position cannot attend to later positions; this lets the decoder predict in sequence without using future outputs. Other designs can allow different patterns of access. So when someone asks what self-attention does, the key idea is that positions in one sequence use attention to gather information from other permitted positions in that same sequence.

Where attention fits

Attention is an operation inside a neural network. A transformer is an architecture that uses attention along with other computations, including position-wise feed-forward layers. The original Transformer paper describes an encoder that turns an input sequence into representations and a decoder that uses those representations to produce an output sequence. Attention by itself is not a complete transformer, a complete language model, or a training method.

Self-attention connects positions within one sequence. Encoder-decoder attention, also called cross-attention, connects two sources: its queries come from the decoder, while its keys and values come from the encoder output. This lets the decoder use information from the encoded input. A recurrent neural network handles sequence information differently: it passes a hidden state from one step to the next. These terms describe different ways information can move through a model, rather than different names for the same component.

What attention weights do and do not show

The word “attention” can sound like human concentration, but in a neural network it names a calculation over vectors. The weights show how the values contribute to a particular attention output. They do not, by themselves, explain why the model made a prediction. In experiments across several natural-language-processing tasks, Jain and Wallace found that attention weights often did not align with feature-importance measures, and that different attention distributions could lead to equivalent predictions.

It is also easy to confuse attention with self-attention or with a transformer. Attention is the broader calculation; self-attention is one case where the query, keys, and values come from the same sequence. A transformer is a larger architecture that uses attention alongside other layers. Keeping these levels separate makes explanations clearer: ask what inputs an attention operation can access, what it combines, and what other computations surround it.

Questions people ask

How does self-attention work?

For each position, it compares that position’s query with keys from positions in the same sequence, then combines their values using the resulting weights.

What are queries, keys, and values in attention?

They are vectors used in the calculation: a query is compared with keys, and the associated values are combined according to the resulting weights.

Is attention the same as a transformer?

No. Attention is an operation; a transformer is an architecture that uses attention along with other computations.

What is the difference between self-attention and cross-attention?

Self-attention draws its queries, keys, and values from the same sequence. In encoder-decoder attention, queries come from the decoder and keys and values come from the encoder output.

Do attention weights explain a model’s answer?

Not by themselves. Research has found that attention weights can differ from feature-importance measures and that different attention distributions can produce equivalent predictions.

Learn more

Attention Is All You Need
The original paper describes the query-key-value calculation, self-attention, and how attention is used in the Transformer.
Attention is not Explanation
This research paper examines whether attention weights provide meaningful explanations for model predictions.

What changed

Sources

Part of Neural-network architectures: Architecture building blocks How computation moves