Sorry if this was unclear, but there’s a difference between plans which work conditioning on an impossibility, and trying to do the impossible. For example, building a proof that works only if P=NP is true is silly in ways that trying to prove P=NP is not. The second is trying to do the impossible, the first is what I was dismissive of.
Yes, doing those things in ways that a capable alignment research can’t find obvious failure modes for. (Which may not be enough, given that they.aren’t superintelligences—but is still a bar which no proposed plan comes close to passing.)
I think that basically everyone at MIRI, Yampolskiy, and a dozen other people all have related and strong views on this. You’re posting on Lesswrong, and I don’t want to be rude, but I don’t know why I’d need to explain this instead of asking you to read the relevant work.
I asked because I’m talking with you and I wanted to know *your* reasoning as to why a technical solution to the alignment of superintelligence is impossible. It seems to be “lots of people see lots of challenges and they are too many to overcome, take it up with them”.
But it’s just a hard problem, and the foundations are not utterly mysterious. Humanity understands quite a lot about the physical and computational nature of our reality by now.
Maybe it would be more constructive to ask how you envisage achieving the political impossible of stopping the worldwide AI race, since that’s something that you do advocate.
Sorry if this was unclear, but there’s a difference between plans which work conditioning on an impossibility, and trying to do the impossible. For example, building a proof that works only if P=NP is true is silly in ways that trying to prove P=NP is not. The second is trying to do the impossible, the first is what I was dismissive of.
So what’s the impossible thing—identifying an adequate set of values? instilling them in a superintelligence?
Yes, doing those things in ways that a capable alignment research can’t find obvious failure modes for. (Which may not be enough, given that they.aren’t superintelligences—but is still a bar which no proposed plan comes close to passing.)
Is there someone you regard as the authority on why it can’t be done? (Yudkowsky? Yampolskiy?)
Because what I see, are not problems that we know to be unsolvable, but rather problems that the human race is not seriously trying to solve.
I think that basically everyone at MIRI, Yampolskiy, and a dozen other people all have related and strong views on this. You’re posting on Lesswrong, and I don’t want to be rude, but I don’t know why I’d need to explain this instead of asking you to read the relevant work.
I asked because I’m talking with you and I wanted to know *your* reasoning as to why a technical solution to the alignment of superintelligence is impossible. It seems to be “lots of people see lots of challenges and they are too many to overcome, take it up with them”.
But it’s just a hard problem, and the foundations are not utterly mysterious. Humanity understands quite a lot about the physical and computational nature of our reality by now.
Maybe it would be more constructive to ask how you envisage achieving the political impossible of stopping the worldwide AI race, since that’s something that you do advocate.