In this blogpost, let us look at the Self Attention mechanism introduced in the seminal paper Attention is All You Need from the ground up. In order to understand it we will take the example of a text input — just as the gods who bestowed us with transformers intended.

We want to process the sentence, “All men must serve”

Computers cannot process language directly as they only understand and process numbers. A simplistic way to do it is to create a vocabulary, and assign words to numbers. To not deter our attention away from attention, we will simply create a vocabulary that contains only the words in the sentence: (In practice, a comprehensive vocabulary would be used)

Words and their mappings:

"All," "men," "must," "serve" → mapped to numbers

sentence = 'All men must serve'

dc = {s : i for i, s in enumerate([word for word in sentence.split()])} 

{'All': 0, 'men': 1, 'must': 2, 'serve': 3}

Now, we can convert this text (which the transformer cannot understand) into a vector-integer representation (which the transformer can understand).

python
CopyEdit

tokenized_sentence = [dc[word] for word in sentence.split()]

[0, 1, 2, 3]