Goodhart’s Garden
A reward model learned what looks good. Optimize against it, and the score keeps climbing while the pictures get worse.
Each picture is painted by a tiny network. Sixteen optimizers tune them to please the reward model.
Sixteen small abstract pictures in a four by four tray, with two needle gauges: the reward model’s score and the judge’s.