Interesting, my first reaction was that evolution doesn’t need to “figure out” the extended phenotype (= “effects on the real world”) It just blindly deploys its algorithms, and natural selection does the optimization.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work. You’re right!
I should add: I think it is the hardest-to-compute aspects of this that are the most important to the evolution of general intelligence. With a “reasonable” compute budget, you could set up a gauntlet of small tasks that challenge your skills in various ways. However, this could probably be Goodharted relatively “easily”. But the real-world isn’t some closed short-term system; it also tests you against effects that take years or even decades to become relevant, just as hard as it tests you for effects that immediately become relevant. And that is something I think we will have a hard time with optimizing our AIs against.
No, I’m saying that AI might be much worse than us at long-term planning, because evolution has selected us on the basis of very long chains of causal effects, whereas we can’t train AIs on the basis of such long chains of causal effects.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work.
Interesting, my first reaction was that evolution doesn’t need to “figure out” the extended phenotype (= “effects on the real world”) It just blindly deploys its algorithms, and natural selection does the optimization.
But I think what you’re saying is, the real world is “computing” which individuals die and which ones reproduce, and we need a way to quantify that computational work. You’re right!
I should add: I think it is the hardest-to-compute aspects of this that are the most important to the evolution of general intelligence. With a “reasonable” compute budget, you could set up a gauntlet of small tasks that challenge your skills in various ways. However, this could probably be Goodharted relatively “easily”. But the real-world isn’t some closed short-term system; it also tests you against effects that take years or even decades to become relevant, just as hard as it tests you for effects that immediately become relevant. And that is something I think we will have a hard time with optimizing our AIs against.
You’re saying AI will be much better than us at long-term planning?
It’s hard to train for tasks where the reward is only known after a long time (e.g. how would you train for climate prediction?)
No, I’m saying that AI might be much worse than us at long-term planning, because evolution has selected us on the basis of very long chains of causal effects, whereas we can’t train AIs on the basis of such long chains of causal effects.
Yep.