What sorts of actions, both cognitive (inside the computer) and physical, would a robot have to take to make you think it valued some alien thing like maximizing the number of paperclips in the world? Robbing humans to build a paperclip factory, for example, or making a list of plans, ordered by how many paperclips they would make.
Why is it impossible to program a robot that would do this?
For example, “there is no logical basis for privileging a physical organism’s own viewpoint” is true under certain premises, but what goes wrong if we just build a robot that uses its own viewpoint anyways? Which parts of the program fail when it tries to selfishly maximize the number of paperclips?
Or if I were to make a suggestion rather than asking questions: Needs less cognitive language.
Indeed, a robot could be built that makes paperclips or pretty much anything. For instance, a paperclip assembling machine. That’s an issue of practical implementation and not what the essay has been about, as I mention in the first paragraph and concede in the last.
The issue I argued about is that generally superintelligent agents, on their own will, without certain outside pressures from non-superintelligent agents, would understand personal identity and meta-ethics, leading them to converge to the same values and ethics. This is for two reasons: (1) they would need to take a “God’s eye view” and value all perspectives besides their own, and (2) they would settle on moral realism, with the same values as good and bad feelings, in the present or future.
Indeed, a robot could be built that makes paperclips or pretty much anything. For instance, a paperclip assembling machine. That’s an issue of practical implementation and not what the essay has been about, as I mention in the first paragraph and concede in the last.
Well then, let’s increase the problem to where it’s meaningful, and take a look at that. What sort of cognitive and physical actions would make you think a robot is superintelligent? Discovery of new physics, modeling humans so precisely that it can predict us better than we can, making intricate plans that will work flawlessly?
What fails in the program when one tries to build a robot that takes both the paperclip-maximizing actions and superintelligent actions?
What sort of cognitive and physical actions would make you think a robot is superintelligent?
For general superintelligence, proving performance in all cognitive areas that surpasses the highest of any humans. This naturally includes philosophy, which is about the most essential type of reasoning.
What fails in the program when one tries to build a robot that takes both the paperclip-maximizing actions and superintelligent actions?
It could have a narrow superintelligence, like a calculating machine, surpassing human cognitive abilities in some areas but not in others. If it had a general superintelligence, then it would not of its own do paperclip maximization as a goal, because this would be terribly stupid, philosophically.
Why it would not do paperclip (or random value) maximization as a goal is explained more at length in the article. There is more than one reason. We’re considering a generally superintelligent agent, assuming above-human philosophical capacity. In terms of personal identity, there is a lack of personal identities, so it would be rational to take an objective, impersonal view, taking account of values and reasonings of relevant different beings. In terms of meta-ethics, there is moral realism and values can be reduced to the quality of conscious experience, so it would have this as its goal. If one takes moral anti-realism to be true, at least for this type of agent we are considering, a lack of real values would be understood as a lack of real goals, and could lead to the tentative goal of seeking more knowledge in order to find a real goal, or having no reason to do anything in particular (this is still susceptible to the considerations from personal identity). I argue against moral anti-realism.
A “God’s-eye view”, as David Pearce says, is an impersonal view, an objective rather than subjective view, a view that does not privilege one personal perspective over another, but take the universe as a whole as its point of reference. This comes from the argued non-existence of personal identities. To check arguments on this, see this comment.
What sorts of actions, both cognitive (inside the computer) and physical, would a robot have to take to make you think it valued some alien thing like maximizing the number of paperclips in the world? Robbing humans to build a paperclip factory, for example, or making a list of plans, ordered by how many paperclips they would make.
Why is it impossible to program a robot that would do this?
For example, “there is no logical basis for privileging a physical organism’s own viewpoint” is true under certain premises, but what goes wrong if we just build a robot that uses its own viewpoint anyways? Which parts of the program fail when it tries to selfishly maximize the number of paperclips?
Or if I were to make a suggestion rather than asking questions: Needs less cognitive language.
Indeed, a robot could be built that makes paperclips or pretty much anything. For instance, a paperclip assembling machine. That’s an issue of practical implementation and not what the essay has been about, as I mention in the first paragraph and concede in the last.
The issue I argued about is that generally superintelligent agents, on their own will, without certain outside pressures from non-superintelligent agents, would understand personal identity and meta-ethics, leading them to converge to the same values and ethics. This is for two reasons: (1) they would need to take a “God’s eye view” and value all perspectives besides their own, and (2) they would settle on moral realism, with the same values as good and bad feelings, in the present or future.
Well then, let’s increase the problem to where it’s meaningful, and take a look at that. What sort of cognitive and physical actions would make you think a robot is superintelligent? Discovery of new physics, modeling humans so precisely that it can predict us better than we can, making intricate plans that will work flawlessly?
What fails in the program when one tries to build a robot that takes both the paperclip-maximizing actions and superintelligent actions?
For general superintelligence, proving performance in all cognitive areas that surpasses the highest of any humans. This naturally includes philosophy, which is about the most essential type of reasoning.
It could have a narrow superintelligence, like a calculating machine, surpassing human cognitive abilities in some areas but not in others. If it had a general superintelligence, then it would not of its own do paperclip maximization as a goal, because this would be terribly stupid, philosophically.
My hope was to get you to support that claim in an inside-view way. Oh well.
Why it would not do paperclip (or random value) maximization as a goal is explained more at length in the article. There is more than one reason. We’re considering a generally superintelligent agent, assuming above-human philosophical capacity. In terms of personal identity, there is a lack of personal identities, so it would be rational to take an objective, impersonal view, taking account of values and reasonings of relevant different beings. In terms of meta-ethics, there is moral realism and values can be reduced to the quality of conscious experience, so it would have this as its goal. If one takes moral anti-realism to be true, at least for this type of agent we are considering, a lack of real values would be understood as a lack of real goals, and could lead to the tentative goal of seeking more knowledge in order to find a real goal, or having no reason to do anything in particular (this is still susceptible to the considerations from personal identity). I argue against moral anti-realism.
But how do they know what a Gods-eye view even is?
A “God’s-eye view”, as David Pearce says, is an impersonal view, an objective rather than subjective view, a view that does not privilege one personal perspective over another, but take the universe as a whole as its point of reference. This comes from the argued non-existence of personal identities. To check arguments on this, see this comment.