Why can a chatbot’s tokenizer—a program that breaks text into smaller pieces—still process the never-seen misspelling “strwberry”?

Answer: It splits the unfamiliar string into smaller known chunks, such as letter groups.

It removes the odd letters until what remains matches a familiar word. — Wrong—it is true that the splitter does not treat the odd letters as a normal whole word, but it does not discard them. Ignoring letters would change the string, not encode it. The splitter preserves the input by representing those letters with smaller known pieces.

It splits the unfamiliar string into smaller known chunks, such as letter groups. ✓ — Right—the tokenizer does not need the whole word to be known. It can break “strwberry” into smaller chunks—subword pieces, characters, or bytes—that are already in its vocabulary. That fallback lets it represent typos and names that never appeared as complete entries.

It checks the whole string against a fixed vocabulary of complete words. — Wrong—a dictionary check is how spellcheckers often work, but this splitter is not limited to whole known words. If it only accepted complete dictionary entries, it would fail on typos and new names. Instead, it can represent unknown strings as smaller known pieces.