The story
Think about a child learning to read. Before she reads whole books, she learns letters. Then short words. Then she starts to recognise common chunks like "ing", "the", "tion". She does not read letter by letter, and she does not need a separate memory for every word in the dictionary. She uses a mix of chunks.
An AI language model works in a very similar way. It cannot read like we do. It needs the text cut into pieces first. Those pieces are called tokens.
The one plain idea
A token is a small piece of text that the AI treats as one unit. It might be a whole word, part of a word, a single letter, a space, or a punctuation mark.
Everything an AI language model does, it does with tokens. It reads tokens. It writes tokens. It is limited by tokens. It is billed by tokens.
The kitchen-table picture
Imagine a box of Lego. You can build almost anything, but you never build from raw plastic. You build from bricks. Common shapes come as a single brick. Rare shapes have to be built from several small bricks.
Tokens are the bricks. Common words like "the" or "coffee" are usually one brick. A rare word like "unbelievably" may be several bricks: perhaps "un", "believ", "ably". The exact split depends on the model.
Worked example
Take the sentence: "I love shopping online."
One plausible split into tokens looks like this:
| Piece | I | love | shopping | online | . |
|---|---|---|---|---|---|
| Tokens | 1 | 1 | 1 | 1 | 1 |
That is about 5 tokens. Now a harder sentence: "Antidisestablishmentarianism is long." A tokenizer will probably split the long word into several pieces, so the sentence may come to 8 or 9 tokens. The exact numbers depend on the model, so treat both counts as illustrative.
A common rule of thumb for ordinary English: one token is roughly four characters, or about three quarters of a word. So 100 tokens is roughly 75 words. This is a rough average, not a law. Code, numbers, and other languages behave differently (see Chapter 8).
Two meanings of the word "token"
You may have heard "token" in crypto. Crypto tokens are digital assets on a blockchain. AI tokens are a different thing. They are not coins you own. They are pieces of text. Same word, different world. In this course, "token" always means the AI kind unless we say otherwise.
No-code exercise
Open any free AI chat or any public tokenizer demo page that a model provider offers. Paste in three things: a plain English sentence, a long rare word, and a sentence in another language you know. Write down how many tokens each one uses. Which surprised you?
Self-check
- In one sentence, what is a token?
- Is a token always a whole word? Give one example that shows it is not.
- Is an AI token the same as a crypto token?
- Roughly how many words are 100 tokens in English?
Answers. 1) A small piece of text the model treats as one unit. 2) No. A long or rare word is often split into several tokens. 3) No. They share a name only. 4) About 75.