Brute forcing extended high-fidelity simulations of all the humans that have ever lived in an attempt to formulate CEV will probably be too expensive for any first-generation AGI.
Prediction 1: that will never be a possibility, period, not just for a “first-generation” anything, but all the way out to the omega point. Not if you want the simulation to have enough fidelity to be useful for any practical purpose, or even interesting.
It probably won’t even be possible, let alone cost effective, to do that for one person, since you’d have to faithfully simulate how the environment would interact with that person. Any set of ideas that relies on simulations like that is going to end up being useless.
I agree that this is probably true, but I wouldn’t put it at at > 90% probability for far-future AI. With computing power greater than Jupiter brains, it probably still wouldn’t be practical, but my point in thinking about it was that if it were possible to brute-force for first-generation AGI, then there’s a chance for more efficient ways.
Prediction 1: that will never be a possibility, period, not just for a “first-generation” anything, but all the way out to the omega point. Not if you want the simulation to have enough fidelity to be useful for any practical purpose, or even interesting.
It probably won’t even be possible, let alone cost effective, to do that for one person, since you’d have to faithfully simulate how the environment would interact with that person. Any set of ideas that relies on simulations like that is going to end up being useless.
I agree that this is probably true, but I wouldn’t put it at at > 90% probability for far-future AI. With computing power greater than Jupiter brains, it probably still wouldn’t be practical, but my point in thinking about it was that if it were possible to brute-force for first-generation AGI, then there’s a chance for more efficient ways.