ou can’t turn the galaxy into paper-clips (or whatever) without extensively mastering science, technology, intergalactic flight, nanotechnology—and so on. So, you need scientists and engineers—and other complicated and interesting things. This conclusion seems so obvious as to hardly be worth discussing to me.
Yes, but there would be no persons. There would be no scientists, no joy of discovery, no feeling of curiosity. There would just be a “process” that, from the outside, would look like an avalanche of expanding machinery, and on the inside would have no subjective experience. It would contain a complex intelligence, but there would be no-one to marvel at the complex intelligence, not even itself, because there would be “no-one home” in all likelihood.
For me, what proved decisive in coming to a low estimate of the value of such a system was the realization that the reason that I liked science, technology, etc, was because of my subjective experiences of finding out the answer.
They are unlikely to lack science, technology, or other interesting stuff.
Interestingness is in the eye of the beholder, but this piece argues that the beholder would have no eye; that there would be an optimizing process that lacked the ability to experience joy over any of its discoveries.
While I think you may very plausibly be correct, there is (I think) some reasonable grounds for uncertainty. I can imagine that an advanced algorithm that performs the role of making scientific discoveries to aid in the development of technologies for the great paperclip fleet might indeed have “some one home”. It maybe that this is beneficial to its effectiveness, or might be close to essential.
I can’t make any strong claims about why this would be needed, only that human beings (at least me) do have “some one home”, but if we didn’t know about human beings are we were speculating on what organisms evolution might produce we might find ourselves postulating complex, social creatures who solve complicated tasks, but have “no one home”, and we would obviously be wrong.
Yes, but there would be no persons. There would be no scientists, no joy of discovery, no feeling of curiosity. There would just be a “process” that, from the outside, would look like an avalanche of expanding machinery, and on the inside would have no subjective experience. It would contain a complex intelligence, but there would be no-one to marvel at the complex intelligence, not even itself, because there would be “no-one home” in all likelihood.
For me, what proved decisive in coming to a low estimate of the value of such a system was the realization that the reason that I liked science, technology, etc, was because of my subjective experiences of finding out the answer.
Interestingness is in the eye of the beholder, but this piece argues that the beholder would have no eye; that there would be an optimizing process that lacked the ability to experience joy over any of its discoveries.
While I think you may very plausibly be correct, there is (I think) some reasonable grounds for uncertainty. I can imagine that an advanced algorithm that performs the role of making scientific discoveries to aid in the development of technologies for the great paperclip fleet might indeed have “some one home”. It maybe that this is beneficial to its effectiveness, or might be close to essential.
I can’t make any strong claims about why this would be needed, only that human beings (at least me) do have “some one home”, but if we didn’t know about human beings are we were speculating on what organisms evolution might produce we might find ourselves postulating complex, social creatures who solve complicated tasks, but have “no one home”, and we would obviously be wrong.