For processing data like text or signals, the transformer is used in many neural network topologies. Language processing is where neural network transformers are most commonly used.
As an example, this type of neural network takes a string of vectors as an input sentence and changes it into a vector termed an encoding.
The transformer’s mechanism is a crucial component. As a result of the attention mechanism, the encoding of a token is affected by the importance of other tokens in the input.
In order to select how to translate a word, the transformer uses the mechanism to concentrate on specific words on both sides of the word at hand.
- Formerly used neural network designs such as RNN, LSTM, and GRU have been replaced with transformer neural networks in deep learning.
Transformer Neural Network model
- The transformer model in machine learning converts a sentence into two strings: vector embeddings and positional encodings.
Text is represented numerically via word vectors, which are numerical representations. For a neural network to process the words, they must be converted to embedding representation. Words are represented as vectors. In the positional encodings, the word’s position is represented as a vector.
After adding word embeddings and encodings, the product is sent via lines of encoders, then lines of decoders. Because the full input is supplied into the network concurrently, it differs from RNNs and LSTMs, which feed the complete input sequentially.
There are encodings that are created by each encoder by taking the input and turning it into another series of vectors. This process is reversed in decoding. The encoded-words are converted back to probabilities of different output words in decoding. Using the softmax function, the output probabilities can be transformed into another natural language sentence.
Every encoder and decoder has attention techniques built in to allow the processing of a single input word to incorporate relevant data from certain other words while hiding the words that do not.
Now that we explained the transformer model let’s see how it fares against RNN and LSTM.
Because ML systems are more fragile than you think. All based on our open-source core.
Install Open SourceTNN vs RNN
- Transformers and RNNs are fundamentally different in design
In the long run, an RNN keeps a concealed state vector in place. Each input word is sent through a neural network’s layers and changed to alter the state vector. If the state vector were to remember inputs from the far past at some point, it may potentially do so.
Normally, the model’s hidden state does not contain a lot of information about the initial inputs. It is easy for new inputs to overwrite an existing state, resulting in information loss. In other words, an RNN’s performance deteriorates as the phrase length increases. Known as the long-term reliance problem, this is a serious issue.
It is difficult to use computing like GPUs with RNNs since they process input sequences sequentially. By processing incoming words simultaneously on a GPU, the Transformer’s parallel-processing techniques allow for faster training.
TNN vs LSTM
This type of RNN has been particularly successful in solving a range of issues, including text classification and speech recognition among others.
The cell state is at the heart of the LSTM architecture. LSTM maintains this concealed state overtime when input tokens are received. Due to its recurrent nature, an LSTM gets inputs one at a time.
- While RNN design is used in the LSTM, the ability to change information in a hidden cell state is tightly controlled using structures called “gates”.
Standard LSTM designs include three gates: the ‘input’ or ‘input gate,’ the ‘output gate,’ and the ‘forget gate,’ respectively.
In part, LSTM’s sophisticated gated design tackles the problem of long-term dependency. To train and execute the LSTM, however, sequencing is required. There is no need for a transformer’s attention mechanism because dependencies can move from one direction to another. The LSTM’s recurrent nature also makes it difficult to leverage parallel computation, resulting in an extremely slow training time.
- Transformer networks have an advantage over LSTMs and RNNs in that they can process many words at the same time.
Numerous studies enhanced LSTM performance by adding attention mechanisms before the transformer architecture was developed. To enhance accuracy, researchers discovered that a recurrent neural network was no longer essential and that a simple attention mechanism might do so instead. So the transformer might be designed in a parallel fashion, allowing training on graphics processing units.