CEV also has the problem that nothing short of a superintelligence could actually use it, so unless AI has a really hard takeoff you’re going to need something less complicated for your AI to use in the meantime.
Personally I’ve always thought EY places too much emphasis on solving the whole hard problem of ultimate AI morality all at once. It would be quite valuable to see more foundation-building work on moral systems for less extreme sots of AI, with an emphasis on avoiding bad failure modes rather than trying to get the best possible outcome. That’s the sort of research that could actually grow into an academic sub-discipline, and I’d expect it to generate insights that would help with attempts to solve the SI morality problem.
Of course, the last I heard EY was still predicting that dangerous levels of AI will come along in less time than it would take such a discipline to develop. The gradual approach could work if it takes 100 years to go from mechanical kittens to Skynet’s big brother, but not if it only takes 5.
Also note that a really hard fast takeoff is even more of a reason to shift emphasis away from distant uncomputable impracticable problems and focus on the vastly smaller set of actual practical choices that we can make now.
CEV also has the problem that nothing short of a superintelligence could actually use it, so unless AI has a really hard takeoff you’re going to need something less complicated for your AI to use in the meantime.
Personally I’ve always thought EY places too much emphasis on solving the whole hard problem of ultimate AI morality all at once. It would be quite valuable to see more foundation-building work on moral systems for less extreme sots of AI, with an emphasis on avoiding bad failure modes rather than trying to get the best possible outcome. That’s the sort of research that could actually grow into an academic sub-discipline, and I’d expect it to generate insights that would help with attempts to solve the SI morality problem.
Of course, the last I heard EY was still predicting that dangerous levels of AI will come along in less time than it would take such a discipline to develop. The gradual approach could work if it takes 100 years to go from mechanical kittens to Skynet’s big brother, but not if it only takes 5.
Agreed.
Also note that a really hard fast takeoff is even more of a reason to shift emphasis away from distant uncomputable impracticable problems and focus on the vastly smaller set of actual practical choices that we can make now.