A normal gamer plays a game normally. A smarter gamer (a speedrunner) breaks the game to beat it faster. A normal AI program optimizes approximately what you want it to. A smarter AI program can think of more ways to “break” your goal. It figures out how to get its hard-coded goal, rather than your implied desire. And, as ML systems get smarter at “optimization” in general, we’d expect this problem to happen increasingly often. (Machine learning researchers)
A normal gamer plays a game normally. A smarter gamer (a speedrunner) breaks the game to beat it faster. A normal AI program optimizes approximately what you want it to. A smarter AI program can think of more ways to “break” your goal. It figures out how to get its hard-coded goal, rather than your implied desire. And, as ML systems get smarter at “optimization” in general, we’d expect this problem to happen increasingly often. (Machine learning researchers)