Tokenization
AI · Beginner
Definition
Tokenization splits text into tokens (words or sub-words) that language models turn into numbers.
Token count drives LLM cost and context limits.
Diagram
Flowchart
Example: Word tokenizer
vocab = {}
ids = [vocab.setdefault(w, len(vocab)) for w in 'ai is fun ai'.split()]
print(ids)Output: [0, 1, 2, 0]
Key points
- BPE and WordPiece are sub-word methods
- ~4 characters per token in English
Interview questions
Why sub-word tokens?
They handle rare and new words without a giant vocabulary.