As long as all mature superintelligences in our universe don’t necessarily have (end up with) the same values, and only some such values can be identified with our values or what our values should be, AI alignment seems as important as ever. You mention “complications” from obliqueness, but haven’t people like Eliezer recognized similar complications pretty early, with ideas such as CEV?
It seems to me that from a practical perspective, as far as what we should do, your view is much closer to Eliezer’s view than to Land’s view (which implies that alignment doesn’t matter and we should just push to increase capabilities/intelligence). Do you agree/disagree with this?
It occurs to me that maybe you mean something like “Our current (non-extrapolated) values are our real values, and maybe it’s impossible to build or become a superintelligence that shares our real values so we’ll have to choose between alignment and superintelligence.” Is this close to your position?
“as important as ever”: no, because our potential influence is lower, and the influence isn’t on things shaped like our values, there has to be a translation, and the translation is different from the original.
CEV: while it addresses “extrapolation” it seems broadly based on assuming the extrapolation is ontologically easy, and “our CEV” is an unproblematic object we can talk about (even though it’s not mathematically formalized, any formalization would be subject to doubt, and even if formalized, we need logical uncertainty over it, and logical induction has additional free parameters in the limit). I’m really trying to respond to orthogonality not CEV though.
from a practical perspective: notice that I am not behaving like Eliezer Yudkowsky. I am not saying the Orthogonality Thesis is true and important to ASI, I am instead saying intelligence/values are Oblique and probably nearly Diagonal (though it’s unclear what I mean by “nearly”). I am not saying a project of aligning superintelligence with human values is a priority. I am not taking research approaches that assume a Diagonal/Orthogonal factorization. I left MIRI partially because I didn’t like their security policies (and because I had longer AI timelines), I thought discussion of abstract research ideas was more important. I am not calling for a global AI shutdown so this project (which is in my view confused) can be completed. I am actually against AI regulation on the margin (I don’t have a full argument for this, it’s a political matter at this point).
I think practicality looks more like having near-term preferences related to modest intelligence increases (as with current humans vs humans with neural nets; how do neural nets benefit or harm you, practically? how can you use them to think better and improve your life?), and not expecting your preferences to extend into the distant future with many ontology changes, so don’t worry about grabbing hold of the whole future etc, think about how to reduce value drift while accepting intelligence increases on the margin. This is a bit like CEV except CEV is in a thought experiment instead of reality.
The “Models of ASI should start with realism” bit IS about practicalities, namely, I think focusing on first forecasting absent a strategy of what to do about the future is practical with respect to any possible influence on the far future; practically, I think your attempted jump to practicality (which might be related to philosophical pragmatism) is impractical in this context.
It occurs to me that maybe you mean something like “Our current (non-extrapolated) values are our real values, and maybe it’s impossible to build or become a superintelligence that shares our real values so we’ll have to choose between alignment and superintelligence.” Is this close to your position?
Close. Alignment of already-existing human values with superintelligence is impossible (I think) because of the arguments given. That doesn’t mean humans have no preferences indirectly relating to superintelligence (especially, we have preferences about modest intelligence increases, and there’s some iterative process).
My overall position can be summarized as being uncertain about a lot of things, and wanting (some legitimate/trustworthy group, i.e., not myself as I don’t trust myself with that much power) to “grab hold of the whole future” in order to preserve option value, in case grabbing hold of the whole future turns out to be important. (Or some other way of preserving option value, such as preserving the status quo / doing AI pause.) I have trouble seeing how anyone can justifiably conclude “so don’t worry about grabbing hold of the whole future” as that requires confidently ruling out various philosophical positions as false, which I don’t know how to do. Have you reflected a bunch and really think you’re justified in concluding this?
E.g. in Ontological Crisis in Humans I wrote “Maybe we can solve many ethical problems simultaneously by discovering some generic algorithm that can be used by an agent to transition from any ontology to another?” which would contradict your “not expecting your preferences to extend into the distant future with many ontology changes” and I don’t know how to rule this out. You wrote in the OP “Current solutions, such as those discussed in MIRI’s Ontological Crises paper, are unsatisfying. Having looked at this problem for a while, I’m not convinced there is a satisfactory solution within the constraints presented.” but to me this seems like very weak evidence for the problem being actually unsolvable.
As long as all mature superintelligences in our universe don’t necessarily have (end up with) the same values, and only some such values can be identified with our values or what our values should be, AI alignment seems as important as ever. You mention “complications” from obliqueness, but haven’t people like Eliezer recognized similar complications pretty early, with ideas such as CEV?
It seems to me that from a practical perspective, as far as what we should do, your view is much closer to Eliezer’s view than to Land’s view (which implies that alignment doesn’t matter and we should just push to increase capabilities/intelligence). Do you agree/disagree with this?
It occurs to me that maybe you mean something like “Our current (non-extrapolated) values are our real values, and maybe it’s impossible to build or become a superintelligence that shares our real values so we’ll have to choose between alignment and superintelligence.” Is this close to your position?
“as important as ever”: no, because our potential influence is lower, and the influence isn’t on things shaped like our values, there has to be a translation, and the translation is different from the original.
CEV: while it addresses “extrapolation” it seems broadly based on assuming the extrapolation is ontologically easy, and “our CEV” is an unproblematic object we can talk about (even though it’s not mathematically formalized, any formalization would be subject to doubt, and even if formalized, we need logical uncertainty over it, and logical induction has additional free parameters in the limit). I’m really trying to respond to orthogonality not CEV though.
from a practical perspective: notice that I am not behaving like Eliezer Yudkowsky. I am not saying the Orthogonality Thesis is true and important to ASI, I am instead saying intelligence/values are Oblique and probably nearly Diagonal (though it’s unclear what I mean by “nearly”). I am not saying a project of aligning superintelligence with human values is a priority. I am not taking research approaches that assume a Diagonal/Orthogonal factorization. I left MIRI partially because I didn’t like their security policies (and because I had longer AI timelines), I thought discussion of abstract research ideas was more important. I am not calling for a global AI shutdown so this project (which is in my view confused) can be completed. I am actually against AI regulation on the margin (I don’t have a full argument for this, it’s a political matter at this point).
I think practicality looks more like having near-term preferences related to modest intelligence increases (as with current humans vs humans with neural nets; how do neural nets benefit or harm you, practically? how can you use them to think better and improve your life?), and not expecting your preferences to extend into the distant future with many ontology changes, so don’t worry about grabbing hold of the whole future etc, think about how to reduce value drift while accepting intelligence increases on the margin. This is a bit like CEV except CEV is in a thought experiment instead of reality.
The “Models of ASI should start with realism” bit IS about practicalities, namely, I think focusing on first forecasting absent a strategy of what to do about the future is practical with respect to any possible influence on the far future; practically, I think your attempted jump to practicality (which might be related to philosophical pragmatism) is impractical in this context.
Close. Alignment of already-existing human values with superintelligence is impossible (I think) because of the arguments given. That doesn’t mean humans have no preferences indirectly relating to superintelligence (especially, we have preferences about modest intelligence increases, and there’s some iterative process).
What do you think about my positions on these topics as laid out in and Six Plausible Meta-Ethical Alternatives and Ontological Crisis in Humans?
My overall position can be summarized as being uncertain about a lot of things, and wanting (some legitimate/trustworthy group, i.e., not myself as I don’t trust myself with that much power) to “grab hold of the whole future” in order to preserve option value, in case grabbing hold of the whole future turns out to be important. (Or some other way of preserving option value, such as preserving the status quo / doing AI pause.) I have trouble seeing how anyone can justifiably conclude “so don’t worry about grabbing hold of the whole future” as that requires confidently ruling out various philosophical positions as false, which I don’t know how to do. Have you reflected a bunch and really think you’re justified in concluding this?
E.g. in Ontological Crisis in Humans I wrote “Maybe we can solve many ethical problems simultaneously by discovering some generic algorithm that can be used by an agent to transition from any ontology to another?” which would contradict your “not expecting your preferences to extend into the distant future with many ontology changes” and I don’t know how to rule this out. You wrote in the OP “Current solutions, such as those discussed in MIRI’s Ontological Crises paper, are unsatisfying. Having looked at this problem for a while, I’m not convinced there is a satisfactory solution within the constraints presented.” but to me this seems like very weak evidence for the problem being actually unsolvable.