Transformer

The Transformer is a neural network architecture built around self-attention, introduced by Google researchers in a 2017 paper. Self-attention lets every position in a sequence weigh every other position directly, so the model captures long-range relationships without passing information step by step as recurrent networks did. Because those comparisons happen at once rather than in order, training parallelises across GPUs, which is what made scaling to very large models practical. A block combines multi-head attention with a feed-forward layer, residual connections, and normalisation, and positional encodings supply the word order that attention alone does not. Three variants exist: encoder-only models like BERT for understanding tasks, decoder-only models like GPT, Claude, and Llama for generation, and encoder-decoder models like T5 for translation. The design generalised beyond text to images with Vision Transformer, to audio, and to protein structure prediction in AlphaFold. Its weakness is that attention cost grows with the square of sequence length, which makes long contexts expensive; FlashAttention, sparse attention, and state-space alternatives such as Mamba all target that limit.