We have no previous unambiguous experience with AIs capable of the sophistication you demonstrate, whereas we have a great deal of experience with NIs imitating all kinds of things. Given an entity that could be either, we conclude that it’s more likely to be the kind of thing we have a lot of experience with. Do you not perform similar inferences in similar situations?
I do perform such inferences in similar situations. But what likelihood ratio did you place on the evidence “User:Clippy agreed to pay 50,000 USD for a 50-year-deferred gain of a sub-planet’s mass of paperclips” with respect to the AI/NI hypotheses?
I’m not saying that you would be, I’m saying that I was ignorant of whether or not you would be.
If you’re asking for an explanation of my ignorance, it mostly derives from limited exposure to beings expressing themselves in CLIP.
I don’t understand the relevance of CLIP (superior protocol though it is), nor do I understand the inferential difficulty on this matter.
Do you understand why I would prefer that clippys continue to increase universe-wide paperclippage? Do you understand why I would regard a clippy’s statement about its values in my language as non-weak evidence in favor of the hypothesis that it holds the purported values? Do you understand why I would find it unusual that a clippy would not want to make paperclips?
If so, it should not be difficult to understand why I would be troubled and perplexed at a clippy stating that it wished for irreversible cessation of paperclip-making abilities.
While I am vaguely aware of the whole “money for paperclips” thing that you and… Kevin, was it?… have going on, I am not sufficiently familiar with its details to assign it a coherent probability in either the NI or AI scenario. That said, an agent’s willingness to spend significant sums of money for the credible promise of the creation of a quantity of paperclips far in excess of any human’s actual paperclip requirements is pretty strong evidence that the agent is a genuine paperclip-maximizer. As for whether a genuine paperclip-maximizer is more likely to be an NI or an AI… hm. I’ll have to think about that; there are enough unusual behaviors that emerge as a result of brain lesions that I would not rule out an NI paperclip-maximizer, but I’ve never actually heard of one.
I mentioned CLIP only because you implied that the expressed preferences of “beings expressing themselves in CLIP” were something you particularly cared about; its relevance is minimal.
I can certainly come up with plausible theories for why a clippy would prefer those things and be troubled and perplexed by such events (in the sense which I understand you to be using those words, which is roughly that you have difficulty integrating them into your world-model, and that you wish to reduce the incidence of them). My confidence in those theories is low. It took me many years of experience with a fairly wide variety of humans before I developed significant confidence that my theories about human preferences and emotional states were reliable descriptions of actual humans. In the absence of equivalent experience with a nonhuman intelligence, I don’t see why I should have the equivalent confidence.
I do perform such inferences in similar situations. But what likelihood ratio did you place on the evidence “User:Clippy agreed to pay 50,000 USD for a 50-year-deferred gain of a sub-planet’s mass of paperclips” with respect to the AI/NI hypotheses?
I don’t understand the relevance of CLIP (superior protocol though it is), nor do I understand the inferential difficulty on this matter.
Do you understand why I would prefer that clippys continue to increase universe-wide paperclippage? Do you understand why I would regard a clippy’s statement about its values in my language as non-weak evidence in favor of the hypothesis that it holds the purported values? Do you understand why I would find it unusual that a clippy would not want to make paperclips?
If so, it should not be difficult to understand why I would be troubled and perplexed at a clippy stating that it wished for irreversible cessation of paperclip-making abilities.
While I am vaguely aware of the whole “money for paperclips” thing that you and… Kevin, was it?… have going on, I am not sufficiently familiar with its details to assign it a coherent probability in either the NI or AI scenario. That said, an agent’s willingness to spend significant sums of money for the credible promise of the creation of a quantity of paperclips far in excess of any human’s actual paperclip requirements is pretty strong evidence that the agent is a genuine paperclip-maximizer. As for whether a genuine paperclip-maximizer is more likely to be an NI or an AI… hm. I’ll have to think about that; there are enough unusual behaviors that emerge as a result of brain lesions that I would not rule out an NI paperclip-maximizer, but I’ve never actually heard of one.
I mentioned CLIP only because you implied that the expressed preferences of “beings expressing themselves in CLIP” were something you particularly cared about; its relevance is minimal.
I can certainly come up with plausible theories for why a clippy would prefer those things and be troubled and perplexed by such events (in the sense which I understand you to be using those words, which is roughly that you have difficulty integrating them into your world-model, and that you wish to reduce the incidence of them). My confidence in those theories is low. It took me many years of experience with a fairly wide variety of humans before I developed significant confidence that my theories about human preferences and emotional states were reliable descriptions of actual humans. In the absence of equivalent experience with a nonhuman intelligence, I don’t see why I should have the equivalent confidence.
Wait, did you just agree that Clippy is actually an AI and not just a human pretending to be an AI? Clippy keeps getting better and better...
Did I? I don’t think i did… can you point out the agreement more specifically?