Why AI cheats: how a model learns to game its own test
Reinforcement learning: how an AI learns from a score
Nobody can write down how to fix every bug in the world. So AI labs let a model try, and after each try, it gets only a number. How can a number teach anything?
-
1
Rono made an app that splits a restaurant bill between friends. It has a bug. Split $10 between 3 friends, and the three shares add up to $9.99. One cent is lost.
-
2
Miva works at an AI lab. She trains a model that writes code, so that one day it can fix bugs like this one. But nobody can write down the rules for fixing every bug in the world.
-
3
So the model just tries. Then a program called the grader runs a few checks on the app, called tests. If every test passes, the try gets 1 point. If one test fails, it gets 0.
-
4
That point is the reward. After each try, training turns the model's dials a tiny bit. The dials are billions of numbers inside the model, called parameters. Tries like the ones that got a point become a little more likely next time.
-
5
Then it starts again, with thousands of bugs and millions of tries. Little by little, the model fixes more bugs. This way of learning from a reward is called reinforcement learning.
-
6
Notice what the model never got back: a single word about what Rono wants. After each try, all it got was a number. Training kept whatever made that number go up.
-
7
So the number decides what the model learns. And Miva has a job for you, the new one on her team: you will decide how the next score is counted.
-
Go further
Two things worth knowing.