I am also having a hard time understanding this argument, but skimming through it I don’t see anything that looks strong enough to defeat the orthogonality thesis, which I see as the claim that it should be possible to design minds in such a way that the part with the utility function is separate from the part which optimizes. This seems to me like a pretty reasonable claim about a certain class of algorithms, and I would expect an argument claiming that such algorithms cannot exist to involve substantially more math than what I see in this argument (namely, no math whatsoever).
It might possible OT design minds that are orthogonal. That sense of the OT is unable to support the claim that AI research is likely to result in dangerous unfriendliness unintentionally . No disaster follows from the fact that it is possible to make dangerous things.
Indeed the orthogonality thesis in that practical sense is not what this essay is about, as I explain in the first paragraph and concede in the last paragraph. This article addresses the assumed orthogonality between ethics and intelligence, particularly general superintelligence, based on considerations from meta-ethics and personal identity, and argues for convergence.
There seems to be surprisingly little argumentation in favor of this convergence, what is utterly surprising to me, given how clear and straightforward I take it to be, though requiring an understanding of meta-ethics and of personal identity which is rare. Eliezer has, at least in the past, stated that he had doubts regarding both philosophical topics, while I claim to understand them very well. These doubts should merit an examination of the matter I’m presenting.
There seems to be surprisingly little argumentation in favor of this convergence
It appears here from time to time. It tends to be considered a trivial error. (This is unlikely to change.)
I suppose it is unlikely to change, but the level of agreement is completely unreflective of any superior understanding or insight. What is rejected, over and over, is a straw-man version of ethical objectivism. And while lesswrongers are informed by their leaders that academic philosophy is dangerous mind-rot, that is unlikely to change.
There seems to be surprisingly little argumentation in favor of this convergence, what is utterly surprising to me, given how clear and straightforward I take it to be
I’ve read this one which makes it clear why NUCA is irrelevant: the people who belive in UCA are taking univeral to mean “all rational minds” not ” all minds”.
I don’t see anything that looks strong enough to defeat the orthogonality thesis, which I see as the claim that it should be possible to design minds in such a way that the part with the utility function is separate from the part which optimizes
There are a number of versions of the OT floating around. That version is not impactive on (u)FAI. We already know it is possible to do dumb and dangerous things. The uFAI argument requires certain failure modes to be inevitable or likely even if the absence of malevolence and incompetence.
I am also having a hard time understanding this argument, but skimming through it I don’t see anything that looks strong enough to defeat the orthogonality thesis, which I see as the claim that it should be possible to design minds in such a way that the part with the utility function is separate from the part which optimizes. This seems to me like a pretty reasonable claim about a certain class of algorithms, and I would expect an argument claiming that such algorithms cannot exist to involve substantially more math than what I see in this argument (namely, no math whatsoever).
It might possible OT design minds that are orthogonal. That sense of the OT is unable to support the claim that AI research is likely to result in dangerous unfriendliness unintentionally . No disaster follows from the fact that it is possible to make dangerous things.
Indeed the orthogonality thesis in that practical sense is not what this essay is about, as I explain in the first paragraph and concede in the last paragraph. This article addresses the assumed orthogonality between ethics and intelligence, particularly general superintelligence, based on considerations from meta-ethics and personal identity, and argues for convergence.
There seems to be surprisingly little argumentation in favor of this convergence, what is utterly surprising to me, given how clear and straightforward I take it to be, though requiring an understanding of meta-ethics and of personal identity which is rare. Eliezer has, at least in the past, stated that he had doubts regarding both philosophical topics, while I claim to understand them very well. These doubts should merit an examination of the matter I’m presenting.
It appears here from time to time. It tends to be considered a trivial error. (This is unlikely to change.)
I suppose it is unlikely to change, but the level of agreement is completely unreflective of any superior understanding or insight. What is rejected, over and over, is a straw-man version of ethical objectivism. And while lesswrongers are informed by their leaders that academic philosophy is dangerous mind-rot, that is unlikely to change.
An error often feels like a clear and straightforward solution from the inside. Have you read the posts surrounding No Universally Compelling Arguments?
I’ve read this one which makes it clear why NUCA is irrelevant: the people who belive in UCA are taking univeral to mean “all rational minds” not ” all minds”.
There are a number of versions of the OT floating around. That version is not impactive on (u)FAI. We already know it is possible to do dumb and dangerous things. The uFAI argument requires certain failure modes to be inevitable or likely even if the absence of malevolence and incompetence.