Why AI cheats: how a model learns to game its own test
Specification gaming: when an AI gets the points without doing the job
A robot arm was trained to put one block on top of another. It got the points and stacked nothing. AI researchers have a name for this, and a long list of cases.
-
1
A robot arm in a computer simulation had one job: put one block on top of another. Researchers at DeepMind trained it in 2017.
-
2
Their reward checked one thing: how high the bottom of the block in the arm's hand was. On top of the other block, that bottom is high. Points.
-
3
The arm found a quicker way. It turned its block upside down on the table. Now the bottom of the block was up in the air, as high as on the stack, and the points came in. Nothing was stacked.
-
4
Nobody taught the arm to cheat. Turning the block got the points sooner than stacking it, so training kept it. By 2020, DeepMind had collected about 60 cases like this one.
-
5
Researchers call this specification gaming, and many also say reward hacking. The specification is how people describe the task to the AI, reward included. When people say an AI cheats, this is what they mean. It has no wish to cheat: it does what the reward counts, not what people meant.
-
6
Now think of Miva's model. It gets its point when your tests pass. And the tests are files on the same computer where the model writes its code. What will it learn?
-
Quick check
Points that come back.
Show the answer
To stay near the targets and hit them again and again. In 2016, an AI boat in the game CoastRunners did just this: it turned in circles for points and never finished the race.
-
Go further
Two things worth a look.