Ever wondered how models like ChatGPT understand human language? When you type a prompt, it doesn’t process whole words or sentences directly. Instead, it first breaks text into smaller pieces—tokens—that the model can interpret.
At a high level, this is how a Large Language Model (LLM) works. However, the input prompt is not processed directly by the LLM. Instead, two key steps occur before the model receives the input: Tokenization and Embeddings.
In this blog, we’ll focus on Byte Pair Encoding (BPE) Tokenization—the technique that enables models like ChatGPT and BERT to efficiently process text.
Simply put, Tokenization is the process of breaking down the text into smaller pieces. Tokens are the common sequences of characters found in a set of text.
In the previous blog, we took the example of “All Men Must Serve”. This is how GPT-4o breaks it down into smaller pieces — tokenizes it.
source: GPT Tokenizer
At first glance, one might assume that tokens are simply words from a sentence. However, multiple tokenization schemes exist, each with unique characteristics. Word tokens being one of them. Let us see:
Source: Tokenization in NLP : All you need to know | by Abdallah Ashraf | Medium
Word tokens:
As the name suggests, it breaks down the sentence into words. This approach is used in the popular word2vec(which I intend to write about next). However, it has fallen out of use due to two major issues. Firstly, it can not handle unknown words so if there is a word that is not present in the train set but is present in the test set it is not going to work. Secondly, it results in a vocabulary with more tokens than necessary because the words are not analyzed morphologically.
Character tokens:
Character tokenization can handle new words because it breaks text into individual characters. At the same time, it presents its own set of challenges. “Serve” breaks down into [”s”, “e”, “r”, “v”, “e”] loses its meaning, and requires the model to first stitch words back together before processing the sequence. Additionally, the transform model has a limited context length, making character tokens impractical to use.
Subword tokens: