Evolution lacks the foresight that humans have. It seems relatively plausible that if evolution had only a little more foresight, it would have been able to instill its objectives much more robustly. Consider that even with the very stupid objective function we were given (maximize inclusive genetic fitness) many humans still say they want children intrinsically. That’s not exactly the same thing as wanting to maximize genetic fitness, but it’s pretty close.
Plus, my interpretation of agentofuser’s question was that they weren’t asking whether unaligned AGI would produce something we’d consider good outright; merely that it would retain some elements of human value. I think this is far more plausible than it seems Eliezer does, for reasons that Paul Christiano talked about here.
Evolution lacks the foresight that humans have. It seems relatively plausible that if evolution had only a little more foresight, it would have been able to instill its objectives much more robustly. Consider that even with the very stupid objective function we were given (maximize inclusive genetic fitness) many humans still say they want children intrinsically. That’s not exactly the same thing as wanting to maximize genetic fitness, but it’s pretty close.
Plus, my interpretation of agentofuser’s question was that they weren’t asking whether unaligned AGI would produce something we’d consider good outright; merely that it would retain some elements of human value. I think this is far more plausible than it seems Eliezer does, for reasons that Paul Christiano talked about here.