I like this—I feel it does a decent job of showing how your neuroscience posts fit into the FAI/intelligence explosion narrative. A few minor comments:
Using this term, it should be clear that a machine superoptimizer will not necessarily be modest or honest
I like the “superoptimizer” terminology, but this sentence makes it sound like we can expect superintelligence to behave differently merely by calling it something different. I realise this isn’t what you mean—I just feel it would be better rephrased in terms of “this avoids bias-inducing loaded terminology”.
Thus, though some utilitarians have proposed that all we value is pleasure, our intuitive
negative reaction to hypothetical worlds in which pleasure is (more or less) maximized suggests that pleasure is not the only thing we value.
Very minor point: it would be nice to add a citation here: someone who says that orgasmium is suboptimal or that most people think orgasmium is suboptimal.
Consider the “crying baby” scenario (Greene et al. 2004):
What is it about this particular example that casts doubt on the homuncular “self”? I can believe that we have many cognitive modules that give competing answers to the crying baby dilemma, but how can I tell that just by reading it? (And doesn’t the same thing happen for every moral dilemma we read?)
I like this—I feel it does a decent job of showing how your neuroscience posts fit into the FAI/intelligence explosion narrative. A few minor comments:
I like the “superoptimizer” terminology, but this sentence makes it sound like we can expect superintelligence to behave differently merely by calling it something different. I realise this isn’t what you mean—I just feel it would be better rephrased in terms of “this avoids bias-inducing loaded terminology”.
Very minor point: it would be nice to add a citation here: someone who says that orgasmium is suboptimal or that most people think orgasmium is suboptimal.
What is it about this particular example that casts doubt on the homuncular “self”? I can believe that we have many cognitive modules that give competing answers to the crying baby dilemma, but how can I tell that just by reading it? (And doesn’t the same thing happen for every moral dilemma we read?)