Why don't GPT-style tokenizers feed every English word as single letters?
Show answer & explanation
Answer: Sequences get too long
Words should stay intact — Keeping every word intact sounds tidy, but natural language has endless names, typos, compounds, and new terms. Pure word tokens create an open-vocabulary problem. Subword tokenization is the compromise: common pieces stay compact, while unfamiliar words can still be built from smaller parts.
Sequences get too long ✓ — A character-level input keeps every letter visible, but it makes the sequence much longer. BPE-style tokenizers compress frequent chunks, so the same context window can carry bigger pieces of text.
Characters are faster — Characters feel simpler to humans, but they are not automatically faster for a language model. They create many more processing steps and burn through the context window sooner. The alphabet is tiny; the problem is the long sequence created when every paragraph is spelled out letter by letter.
More Technology questions
- Why can Cloudflare's lava-lamp camera feed improve encryption even though the cryptographic software that consumes it is deterministic?
- If an attacker learns a pseudorandom generator's seed and algorithm after watching several outputs, why can the later outputs become reconstructable?
- If a phone game shuffle and a physical noise source both look messy, what makes only one useful for security against someone who knows the code?
- At parking-lot speed, why do quiet EVs need alert sounds before tire noise helps?
- Why does the Ferrari 296 cabin sound duct take sound before exhaust treatment?
- Why do sound engineers tune engine orders instead of just making a Ferrari-like exhaust louder?