Goodhart's law: why a number stops telling the truth once it becomes a target
Goodhart's law in AI: reward hacking
A program that learns from a score knows nothing but the score. Give an AI points in a boat race: does it race the way its makers meant?
-
1
In 2016, researchers at OpenAI let a program learn a boat-racing game, CoastRunners, by trying again and again. Only one thing told it how it was doing: its score.
-
2
The game hands out points for hitting targets along the course. The rule that turns what a program does into points is its reward function.
The rest of this episode is for subscribers
Subscribe to read every premium episode. You can cancel when you want.
Paid yearly: $99 instead of $119.88