But there will probably be a lag of several years between prestigious people endorsing Superintelligence or general concern for the problem, and prestigious people endorsing MIRI’s particular technical agenda.
I suspect that it would go something like
Superintelligence is dangerous
Better make sure it’s safe
But Godel (/Loeb) says you can never be 100% sure
I disagree with (3) because the Lobian obstacle is just an obstacle to a certain kind of stable self-modification in a particular toy model, and can’t say anything about what kinds of safety guarantees you can have for superintelligences in general.
I disagree with (4) because MIRI hasn’t shown that there are ways to make a superintelligence 90% or more likely (in a subjective Bayesian sense) to be stably friendly, and I don’t expect us to have shown that in another 20 years, and plausibly not ever.
I suspect that it would go something like
Superintelligence is dangerous
Better make sure it’s safe
But Godel (/Loeb) says you can never be 100% sure
But MIRI has shown that you can be very very sure
I can’t read tone in text so… are statements #3 and #4 jokes? I mean, I straightforwardly disagree with them.
It’s much more likely that I misunderstand something basic about what MIRI does.
Okay, fair enough. To explain briefly:
I disagree with (3) because the Lobian obstacle is just an obstacle to a certain kind of stable self-modification in a particular toy model, and can’t say anything about what kinds of safety guarantees you can have for superintelligences in general.
I disagree with (4) because MIRI hasn’t shown that there are ways to make a superintelligence 90% or more likely (in a subjective Bayesian sense) to be stably friendly, and I don’t expect us to have shown that in another 20 years, and plausibly not ever.
Thanks! I guess I was unduly optimistic. Comes with being a hopeful but ultimately clueless bystander.