The transformer is a neural network architecture introduced by Google researchers in the 2017 paper “Attention Is All You Need.” It has since become the foundation of virtually all modern generative AI, including large language models such as GPT and Claude (the “T” in GPT stands for transformer), image generators and speech models.
The transformer’s breakthrough is the self-attention mechanism. When processing a sequence, self-attention lets every element (such as a word) weigh its relationship to every other element at once, no matter how far apart they sit. In the sentence “the animal didn’t cross the street because it was too tired,” attention is what allows the model to link “it” to “animal” rather than “street.”
Transformers replaced earlier sequence models such as recurrent neural networks (RNNs) and LSTMs for two main reasons:
- Parallel processing – RNNs read text one token at a time, which is slow. Transformers process an entire sequence simultaneously, making it practical to train on internet-scale data using GPUs.
- Long-range understanding – Because attention connects any two positions directly, transformers handle long documents and distant dependencies far better than RNNs, which tend to “forget” earlier context.
A transformer is built from stacked layers, each combining self-attention with a feedforward neural network. The original design had two halves: an encoder that reads input and a decoder that generates output. Modern LLMs such as the GPT family use a decoder-only variant optimized for text generation, while models like BERT use only the encoder for understanding tasks.
The architecture has spread well beyond language. Vision transformers (ViTs) apply attention to image patches for computer vision, and transformers now power speech recognition, protein-structure prediction (AlphaFold) and multimodal models that handle text, images and audio together.
