How ChatGPT and other AI chatbots answer your question
Tokens: the pieces an AI model gets instead of your words
An AI model never sees your words. First, a program cuts your message into small pieces called tokens. A token is not always a whole word. Why cut text into pieces at all?
-
1
Luckily, a program on the company's computers gets to the prompt first and cuts it into small pieces. Each piece is called a token.
-
2
A common word like "plant" stays whole. A rare one is cut wherever known pieces fit, even in the middle of a syllable.
-
3
Every piece, "phil" included, comes from one fixed list: the vocabulary. It holds whole words, word parts and single letters, each at a numbered place, its token ID.
-
4
The program swaps each token for its token ID. Your question reaches the model as a row of numbers, after those of the system prompt.
-
5
You picture a plant with leaves and soil. The model gets 6804, a place in a list, whose neighbour 6803 is "pected", the end of "unsuspected". Nothing in 6804 says that a plant has leaves.
-
Quick check
A word no list holds.
Show the answer
Yes. It is cut into smaller pieces that the vocabulary holds, down to single letters if needed.
-
Go further
Whose list it is, and what comes back out of the model.