Tokenisation Overview

Updated Started

The simplest tokeniser:

  1. Splits text into chunks in some rule-based way, e.g. a word-based tokeniser would split on spaces and punctuation.
text = "Hello, world!"
tokens = text.split()
print(tokens)
['Hello,', 'world!']

Python's split() method only splits on spaces by default. We can use regular expressions to cover spaces and punctuation that also delimit words.

How does the re.findall(r'\w+|[^\w\s]') regex work?

It has two parts separated by | (which means "or"):

  • \w+ matches one or more word characters (letters, digits, underscores).
  • [^\w\s] matches anything that's not a word character and not whitespace. So it matches punctuation.

The ^ inside the brackets [^...] means "not", so [^\w\s] literally means "not a word character, not whitespace".

Both split() and findall() break text into pieces, but they work differently:

  • split(): You tell it what to remove (the separators), and it gives you what's left.
  • findall(): You tell it what to keep (the pattern), and it gives you all matches.
import re
tokens = re.findall(r'\w+|[^\w\s]', text)
print(tokens)
['Hello', ',', 'world', '!']
  1. Maps those chunks to numeric values:
vocab = {token: idx for idx, token in enumerate(set(tokens))}
print(vocab)
{'Hello': 0, ',': 1, '!': 2, 'world': 3}
token_ids = [vocab[token] for token in tokens]
print(token_ids)
[0, 1, 3, 2]

If we try to tokenise text with a word not in our vocabulary, we'd get a KeyError. So we need our first example of a special token, <UNK>.

vocab['<UNK>'] = len(vocab)

Now we can safely encode any text by using vocab.get(token, vocab['<UNK>']) instead of vocab[token].

new_text = "Goodbye, world!"
new_tokens = re.findall(r'\w+|[^\w\s]', new_text)
new_ids = [vocab.get(token, vocab['<UNK>']) for token in new_tokens]
print(new_tokens)
print(new_ids)
['Goodbye', ',', 'world', '!']
[5, 1, 3, 2]

Vocabulary size

A word-level vocabulary has two problems. It needs an entry for every word form it will ever see ("run", "runs", "running"), and anything outside it collapses into <UNK>. A character-level vocabulary has neither problem, but sequences become very long and each token carries little meaning. Subword tokenisers sit in between: common words get a single token, rare words are built from pieces, and the vocabulary size is a knob we choose.

Subword tokenisers

  • Byte Pair Encoding: start from bytes and repeatedly merge the most frequent adjacent pair. Built from scratch, including GPT-style pre-tokenisation and special tokens.
  • SentencePiece: treats whitespace as an ordinary symbol so no language-specific pre-tokeniser is needed, and adds byte fallback, normalisation and the Unigram model.

Linked from

Revision log

  1. Split byte pair encoding and SentencePiece into their own notes, merging in the ai-notes notebooks

Tags: AI