Thanks a lot for this text, it is an excellent summary. I have a deep admiration for your work and your clarity and yet, I find myself updating towards”I will be able to read this same comment in 30 years time and say, yes, I am glad that EY was wrong.”
I don’t have doubts about the validity of the orthogonality principle or about instrumental convergence. My problem is that I find point number 2 utterly implausible. I think you are vastly underestimating the complexity of pulling off a plan that successfully kills all humans, and most of this points are based on the assumption that once that an AGI is built, it will become dangerous really quickly, before we can’t learn any useful insights in the meantime.
If we merely lose control of the future and virtually all resources but many of us aren’t killed in 30 years, would you consider Eliezer right or wrong?
Yeah, ‘AGI takes control of virtually all resources but leaves many humans alive for years’ seems like it clearly violates one or more parts of the EY-model (and the Rob-model, which looks a lot like my model of the EY-model).
An edge case that I wouldn’t assume violates the EY-model is ‘AGI kills all humans but then runs lots of human-ish simulations in order to test some hypotheses, e.g., about how hypothetical aliens it runs into might behave’. I’m not particularly expecting this because it strikes me as conjunctive and unnecessary, but it doesn’t fly in the face of anything I believe.
Thanks a lot for this text, it is an excellent summary. I have a deep admiration for your work and your clarity and yet, I find myself updating towards”I will be able to read this same comment in 30 years time and say, yes, I am glad that EY was wrong.”
I don’t have doubts about the validity of the orthogonality principle or about instrumental convergence. My problem is that I find point number 2 utterly implausible. I think you are vastly underestimating the complexity of pulling off a plan that successfully kills all humans, and most of this points are based on the assumption that once that an AGI is built, it will become dangerous really quickly, before we can’t learn any useful insights in the meantime.
If we merely lose control of the future and virtually all resources but many of us aren’t killed in 30 years, would you consider Eliezer right or wrong?
Wrong. He is being quite clear about what he means
Yeah, ‘AGI takes control of virtually all resources but leaves many humans alive for years’ seems like it clearly violates one or more parts of the EY-model (and the Rob-model, which looks a lot like my model of the EY-model).
An edge case that I wouldn’t assume violates the EY-model is ‘AGI kills all humans but then runs lots of human-ish simulations in order to test some hypotheses, e.g., about how hypothetical aliens it runs into might behave’. I’m not particularly expecting this because it strikes me as conjunctive and unnecessary, but it doesn’t fly in the face of anything I believe.