One emoji can be several UTF-8 bytes. What can a byte-level tokenizer do?
Show answer & explanation
Answer: Start from byte pieces
Start from byte pieces ✓ — Emoji are Unicode symbols encoded as bytes, and byte-level tokenizers start from byte values rather than human-visible characters. Those bytes may then be merged or left as separate token pieces. A tiny picture can therefore be computationally larger than it looks on screen.
Read Unicode code points — Unicode code points are a real layer, so this is a plausible guess. But byte-level BPE starts from encoded byte values and then applies learned merges. The surprise is that a visible emoji can be handled through lower-level byte pieces, not only through the character you see.
Always keep one token — One visible character is not guaranteed to stay one token. Emoji sit in Unicode and may enter the tokenizer as multiple UTF-8 bytes. This is the same trap as letter counting: screen units and model units do not line up neatly.
More Technology questions
- Why can Cloudflare's lava-lamp camera feed improve encryption even though the cryptographic software that consumes it is deterministic?
- If an attacker learns a pseudorandom generator's seed and algorithm after watching several outputs, why can the later outputs become reconstructable?
- If a phone game shuffle and a physical noise source both look messy, what makes only one useful for security against someone who knows the code?
- At parking-lot speed, why do quiet EVs need alert sounds before tire noise helps?
- Why does the Ferrari 296 cabin sound duct take sound before exhaust treatment?
- Why do sound engineers tune engine orders instead of just making a Ferrari-like exhaust louder?