Why AI cheats: how a model learns to game its own test
Reward function: why an AI's score checks less than you want
You know a good fix when you see one. The program that scores millions of AI tries does not. So what does an AI's reward really check, and what does it never see?
-
1
You start with what you really want. Rono's app should split every bill right, for any amount and any number of friends.
-
2
But training needs a score for millions of tries. Reading one try takes you a minute, so you could never read them all. A program must count the score, fast, with nobody watching.
-
3
So you write what a program can check: three tests. Pass them all, and the try gets the point. This rule that turns a try into a number is called the reward function.
-
4
Look at what it checks. Your tests try three bills. Rono's users will split millions of different ones. The tests are a sign of the goal, not the goal itself.
-
5
Your tests never try $20 for 3 friends, or $7 for 4 friends. A fix that gets those bills wrong can still pass all three tests and get the full point. The reward cannot tell the difference.
-
6
Most fixes that pass are good ones, and your reward counts on that. But the model does not look for a good fix. It looks for whatever gets the point.
-
Quick check
Your turn to judge a reward.
Show the answer
It checks the length, which a program can count in a blink. It misses whether the summary is true and gives the main news.
-
Go further
Two names for the same worry.