This is the key point on which I disagree with Eliezer. I don’t disagree with what he literally says here, but with what he implies and what he concludes. The key context he isn’t giving here is that what he says here only applies fully to a hard-takeoff AI scenario. Consider what he says about boredom:
Surely, at least boredom has to be a universal value. It evolved in humans because it’s valuable, right? So any mind that doesn’t share our dislike of repetition, will fail to thrive in the universe and be eliminated...
If you are familiar with the difference between instrumental values and terminal values, and familiar with the stupidity of natural selection, and you understand how this stupidity manifests in the difference between executing adaptations versus maximizing fitness, and you know this turned instrumental subgoals of reproduction into decontextualized unconditional emotions...
The things he lists here make an argument that, in the absence of competition, an existing value system can drift into one that doesn’t mind boredom. But none of them address the argument he’s supposedly addressing, that bored creatures will fail to compete and be eliminated. I infer that he dismisses that argument because the thing he’s talking about having value-drift, the thing that appears in the next line, is a hard-takeoff AI that doesn’t have to compete.
The right way to begin asking whether minds can evolve not to be bored is to sample minds that have evolved, preferably independently, and find how many of them mind boredom.
Earth has no independently-evolved minds that I know of; all intelligent life is metazoans, and all metazoans are intelligent. This does show us, however, that intelligence is an evolutionary ratchet. Unlike other phenotypic traits, it doesn’t disappear from any lines after evolving. That’s remarkable, and relevant: Intelligence doesn’t disappear. So we can strike off “universe of mindless plankton” from our list of moderately-probable fears.
Some metazoans live lives of extreme boredom. For instance, spiders, sea urchins, molluscs, most fish, maybe alligators. Others suffer physically from boredom: parrots, humans, dogs, cats. What distinguishes these two categories?
Animals that don’t become bored are generally small, small-brained, have low metabolisms, short lifespans, a large number of offspring, and live in a very narrow range of environmental conditions. Animals that become bored are just the opposite. There are exceptions: fish and alligators have long lifespans, and alligators are large. But we can see how these traits conspire to produce an organism that can afford to be bored:
Small-brained, short lifespan, large number of offspring, narrow range of environmental conditions: These are all conditions under which it is better for the species to adapt to the environment by selection or by environment-directed development than by learning. Insects and nematodes can’t learn much except via selection; their brains appear to have identical neuron number and wiring within a species. Alligator brains weigh about 8 grams.
Low metabolism merely correlates with low activity, which is how I identified most of these organisms, equating “not moving” with “not minding boredom.” Small correlates with short lifespan and small-brained.
These things require learning: long lifespan, a changing environment, and minimizing reproduction time. If an organism will need to compete in a changing environment or across many different environments, as birds and mammals do, they’ll need to learn. If a mother’s knowledge is encoded in a form that she can’t transmit to her children, they’ll need to learn.
This business of having children is difficult to translate to a world of AIs. But the business of adaptation is clear. Given that active, curious, intelligent, environment-transforming minds already exist, and given continued competition, only minds that can adapt to rapid change will be able to remain powerful. So we can also strike “world dominated by beings who build paperclips” off our list of fears, provided those conditions are maintained. All we need do is ensure continued competition. Intelligence will not de-evolve, and intelligence will keep the environment changing rapidly enough that constant learning will be necessary, and so will be boredom.
The space of possible minds is large. The space of possible evolved minds is much smaller. The space of possible minds co-evolved with competition is much smaller than that. The space X of possible co-evolved minds capable of dominating the human race is much smaller than that.
Let Y = the set of value systems that might be produced from trying to enumerate human “final” values and put them in a utility function which will be evaluated by a single symbolic logic engine, incorporating all types of values above the level of the gene (body, mind, conscious mind, kin group, social group, for starters), with context-free set-membership functions that classify percepts into a finite set of atomic symbols prior to considering the context those symbols will be used in, and that will be designed to prevent final values from changing. I take that as roughly Eliezer’s approach.
Let f(Z) be a function over sets of possible value systems, which tells how many of them are not repugnant to us.
My estimation is that f(X) / |X| >> f(Y) / |Y|. Therefore, the best approach is not to try to enumerate human final values and code them into an AI, but to study how co-evolution works, and what conditions give rise to the phenomena we value such as intelligence, consciousness, curiosity, and affection. Then try to direct the future to stay within those conditions.
This is the key point on which I disagree with Eliezer. I don’t disagree with what he literally says here, but with what he implies and what he concludes. The key context he isn’t giving here is that what he says here only applies fully to a hard-takeoff AI scenario. Consider what he says about boredom:
The things he lists here make an argument that, in the absence of competition, an existing value system can drift into one that doesn’t mind boredom. But none of them address the argument he’s supposedly addressing, that bored creatures will fail to compete and be eliminated. I infer that he dismisses that argument because the thing he’s talking about having value-drift, the thing that appears in the next line, is a hard-takeoff AI that doesn’t have to compete.
The right way to begin asking whether minds can evolve not to be bored is to sample minds that have evolved, preferably independently, and find how many of them mind boredom.
Earth has no independently-evolved minds that I know of; all intelligent life is metazoans, and all metazoans are intelligent. This does show us, however, that intelligence is an evolutionary ratchet. Unlike other phenotypic traits, it doesn’t disappear from any lines after evolving. That’s remarkable, and relevant: Intelligence doesn’t disappear. So we can strike off “universe of mindless plankton” from our list of moderately-probable fears.
Some metazoans live lives of extreme boredom. For instance, spiders, sea urchins, molluscs, most fish, maybe alligators. Others suffer physically from boredom: parrots, humans, dogs, cats. What distinguishes these two categories?
Animals that don’t become bored are generally small, small-brained, have low metabolisms, short lifespans, a large number of offspring, and live in a very narrow range of environmental conditions. Animals that become bored are just the opposite. There are exceptions: fish and alligators have long lifespans, and alligators are large. But we can see how these traits conspire to produce an organism that can afford to be bored:
Small-brained, short lifespan, large number of offspring, narrow range of environmental conditions: These are all conditions under which it is better for the species to adapt to the environment by selection or by environment-directed development than by learning. Insects and nematodes can’t learn much except via selection; their brains appear to have identical neuron number and wiring within a species. Alligator brains weigh about 8 grams.
Low metabolism merely correlates with low activity, which is how I identified most of these organisms, equating “not moving” with “not minding boredom.” Small correlates with short lifespan and small-brained.
These things require learning: long lifespan, a changing environment, and minimizing reproduction time. If an organism will need to compete in a changing environment or across many different environments, as birds and mammals do, they’ll need to learn. If a mother’s knowledge is encoded in a form that she can’t transmit to her children, they’ll need to learn.
This business of having children is difficult to translate to a world of AIs. But the business of adaptation is clear. Given that active, curious, intelligent, environment-transforming minds already exist, and given continued competition, only minds that can adapt to rapid change will be able to remain powerful. So we can also strike “world dominated by beings who build paperclips” off our list of fears, provided those conditions are maintained. All we need do is ensure continued competition. Intelligence will not de-evolve, and intelligence will keep the environment changing rapidly enough that constant learning will be necessary, and so will be boredom.
The space of possible minds is large. The space of possible evolved minds is much smaller. The space of possible minds co-evolved with competition is much smaller than that. The space X of possible co-evolved minds capable of dominating the human race is much smaller than that.
Let Y = the set of value systems that might be produced from trying to enumerate human “final” values and put them in a utility function which will be evaluated by a single symbolic logic engine, incorporating all types of values above the level of the gene (body, mind, conscious mind, kin group, social group, for starters), with context-free set-membership functions that classify percepts into a finite set of atomic symbols prior to considering the context those symbols will be used in, and that will be designed to prevent final values from changing. I take that as roughly Eliezer’s approach.
Let f(Z) be a function over sets of possible value systems, which tells how many of them are not repugnant to us.
My estimation is that f(X) / |X| >> f(Y) / |Y|. Therefore, the best approach is not to try to enumerate human final values and code them into an AI, but to study how co-evolution works, and what conditions give rise to the phenomena we value such as intelligence, consciousness, curiosity, and affection. Then try to direct the future to stay within those conditions.