Unfortunately it also implies that fully defining pain as a moral disvalue is at least as hard as the problem of consciousness, so this line of investigation seems to be at an immediate impasse, at least for the moment.
This is entirely intuitive for me. I strongly expect moral value to have a lot to do with conscious minds. I see no reason to expect full definitions before we have a much more complete understanding of the mind.
To the people that say that some of these systems are conscious:
1 . An AIXI-like (e.g., AIXI-tl, Monte Carlo AIXI, or some such) agent, programmed with the objective of maximizing the number of button presses.
This one is probably conscious, because it seems like the simplest way to simulate conscious-like behaviour is through consciousness. On the one hand, a specific mind contains significantly more information than a description of the behaviour, but on the other hand much of this information may not matter so it could be generated by a simple pseudorandom number generator.
2 . A brute force optimizer, which was programmed with a model of your mind, that iterated through all possible audio/video bit streams to find the one that maximizing the number of button presses. (As far as philosophical implications are concerned, this seems essentially identical to 1, so the reader doesn’t necessarily have to go learn about AIXI.)
These run through all possible outputs. It’s like passing the Turing test by outputting entirely random text to an astronomical number of testers. I might be concerned about the many simulations of myself, but the optimizer itself is not thinking.
4 . A Giant Lookup Table (GLUT) of all possible sense inputs and motor outputs of a person, connected to a virtual body and room.
The GLUT has been precomputed, so the conscious experience has already taken place (not that conscious experiences necessarily have to occur at specific times; answering that question seems to require a more advanced understanding of consciousness). Rereading outputs that have already been written down does not cause more conscious experience any more than rewatching a video of someone that has already been taken.
Practically, there seems to be a very harsh penalty if I am wrong about any of this, so I might refuse to participate anyway, depending on the cost to me.
I also expect consciousness and moral relevance to be related.
If I understand it correctly, AIXI is similar to brute forcing through possible universes using the Solomonoff prior and searching through actions in particular laws of physics.
So AIXI doesn’t have to be conscious, but it probably winds up torturing soooo much stuff when it’s computed.
The only thing I disagree with is that this might not be a bad thing. While AIXI may torture some conscious beings, it may also create some beings in a state of unimaginable bliss. Since AIXI doesn’t care about our values, there is no reason to expect the computation it carries out to have a near maximal positive utility, but, for the same reason, it will not have a near maximal negative utility (as in very negative, not as in nearly zero). Since it is indifferent to us, it will not approach either of these extremes but, since it is a very powerful entity it will create much positive and negative utility without intending to. I don’t see any reason to think that the negative will outweigh the positive.
I don’t expect that the majority of universes would fulfill anything near my values, but I’m not so sure about if they would fulfill their inhabitants’ values.
I think I’m going to lean towards no though, just because value is fragile. Agents have values, and the vast majority of possible universes wouldn’t be optimized for or fulfill them. The blissful conjunction of values with a structurally sympathetic universe seems unlikely compared to values existing in something which is antithetical to them. Agents would prefer to optimize for something, and there are much more ways to not optimize than there are to optimize.
On the other hand, I guess you could call in the anthropic principle and say that, at the very least, agents won’t exist in universes that don’t allow them to exist. Most of the laws of physics seem to be orthogonal to things that I want to do, and I would rather them continue existing rather than not. And agents that evolve via some natural selection probably won’t desire that many things which are physically impossible to instantiate.
Though, I don’t see that as strongly in favor of bliss.
I think the problem is in the definition of `optimum’. In order to be able to call a state optimum, you must presuppose the laws of physics in order to rule out any better states that are physically impossible. Once we recognize this, it seems that any society must either achieve an optimum or suffer an existential disaster (not necessarily extinction). Value is fragile, but minds are powerful and if they ever get on the right track they will never get off, baring problems that are impossible to foresee.
The only cases that remain to be considered are extinction and non-extinction existential risk. I’m pretty sure that my value system in indifferent between the existence and nonexistence of a region with no conscious life, but there is no reason for other value systems to share that property. I am unsure how the average value system would judge its surroundings, partially because I am unsure what to average over. Even a group that manages to optimize its surroundings may describe its universe’s existence as bad due to the existence of variables that it cannot optimize or other factors, such as a general dislike of anything existing.
If an existential risk does not fully wipe out its species, there is a chance that an optimization process will survive, but with different values from its parent species. On average, the parent species would probably regard this as better than extinction, because the optimization process would share some of its values, while being indifferent to the rest. As weak evidence that this applies to our species, there are many fictional distopias that, while much worse than our current world, seem preferable to extinction.
This is entirely intuitive for me. I strongly expect moral value to have a lot to do with conscious minds. I see no reason to expect full definitions before we have a much more complete understanding of the mind.
To the people that say that some of these systems are conscious:
This one is probably conscious, because it seems like the simplest way to simulate conscious-like behaviour is through consciousness. On the one hand, a specific mind contains significantly more information than a description of the behaviour, but on the other hand much of this information may not matter so it could be generated by a simple pseudorandom number generator.
These run through all possible outputs. It’s like passing the Turing test by outputting entirely random text to an astronomical number of testers. I might be concerned about the many simulations of myself, but the optimizer itself is not thinking.
The GLUT has been precomputed, so the conscious experience has already taken place (not that conscious experiences necessarily have to occur at specific times; answering that question seems to require a more advanced understanding of consciousness). Rereading outputs that have already been written down does not cause more conscious experience any more than rewatching a video of someone that has already been taken.
Practically, there seems to be a very harsh penalty if I am wrong about any of this, so I might refuse to participate anyway, depending on the cost to me.
I also expect consciousness and moral relevance to be related.
If I understand it correctly, AIXI is similar to brute forcing through possible universes using the Solomonoff prior and searching through actions in particular laws of physics.
So AIXI doesn’t have to be conscious, but it probably winds up torturing soooo much stuff when it’s computed.
If anyone knows otherwise, please correct me.
Good point; you are correct.
The only thing I disagree with is that this might not be a bad thing. While AIXI may torture some conscious beings, it may also create some beings in a state of unimaginable bliss. Since AIXI doesn’t care about our values, there is no reason to expect the computation it carries out to have a near maximal positive utility, but, for the same reason, it will not have a near maximal negative utility (as in very negative, not as in nearly zero). Since it is indifferent to us, it will not approach either of these extremes but, since it is a very powerful entity it will create much positive and negative utility without intending to. I don’t see any reason to think that the negative will outweigh the positive.
Huh. Interesting point.
I don’t expect that the majority of universes would fulfill anything near my values, but I’m not so sure about if they would fulfill their inhabitants’ values.
I think I’m going to lean towards no though, just because value is fragile. Agents have values, and the vast majority of possible universes wouldn’t be optimized for or fulfill them. The blissful conjunction of values with a structurally sympathetic universe seems unlikely compared to values existing in something which is antithetical to them. Agents would prefer to optimize for something, and there are much more ways to not optimize than there are to optimize.
On the other hand, I guess you could call in the anthropic principle and say that, at the very least, agents won’t exist in universes that don’t allow them to exist. Most of the laws of physics seem to be orthogonal to things that I want to do, and I would rather them continue existing rather than not. And agents that evolve via some natural selection probably won’t desire that many things which are physically impossible to instantiate.
Though, I don’t see that as strongly in favor of bliss.
I think the problem is in the definition of `optimum’. In order to be able to call a state optimum, you must presuppose the laws of physics in order to rule out any better states that are physically impossible. Once we recognize this, it seems that any society must either achieve an optimum or suffer an existential disaster (not necessarily extinction). Value is fragile, but minds are powerful and if they ever get on the right track they will never get off, baring problems that are impossible to foresee.
The only cases that remain to be considered are extinction and non-extinction existential risk. I’m pretty sure that my value system in indifferent between the existence and nonexistence of a region with no conscious life, but there is no reason for other value systems to share that property. I am unsure how the average value system would judge its surroundings, partially because I am unsure what to average over. Even a group that manages to optimize its surroundings may describe its universe’s existence as bad due to the existence of variables that it cannot optimize or other factors, such as a general dislike of anything existing.
If an existential risk does not fully wipe out its species, there is a chance that an optimization process will survive, but with different values from its parent species. On average, the parent species would probably regard this as better than extinction, because the optimization process would share some of its values, while being indifferent to the rest. As weak evidence that this applies to our species, there are many fictional distopias that, while much worse than our current world, seem preferable to extinction.