Transformers are a type of neural network used in neural machine translation, which mainly involves tasks that transform an input sequence to an output sequence. Such tasks include speech recognition and text-to-speech transformation, just to mention a few.
These kinds of tasks require memory—the upcoming sentence has to work with some context from the previous sentence. This is quite critical so as not to lose any important context between sentences.
Until recently, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have been used to tackle this challenge. The problem with these is that they aren’t able to keep up with context and content when sentences are too long.