Among many other things, and most relevantly… We don’t know what we want. We have to hack ourselves in order to approximate having a utility function. This is fairly predictable from the operation of evolution. Consistency in complex systems is something evolution is very bad at producing.
An artificial agent would most likely be built to know what it wants, and could easily have a utility function.
The consequences of this one difference are profound.
Evolved agents would be in rough equality to other agents. So, their game-theoretic considerations would be different from an artificial agent. The artificial agent could have a design very different from all other agents and also could far surpass other agents. Neither of these is possible in evolution.
In fact, because of the similarity between evolved agents in any given ecosystem, these game-theoretic considerations include not only the possibility of reciprocity or reciprocal altruism, but also the sort of acausal reciprocal morality explored by Drescher and MIRI—“you are like me, so my niceness is correlated with yours, so I’d better ask nicely.”
How would you expect evolved and artificial agents to differ?
Among many other things, and most relevantly… We don’t know what we want. We have to hack ourselves in order to approximate having a utility function. This is fairly predictable from the operation of evolution. Consistency in complex systems is something evolution is very bad at producing.
An artificial agent would most likely be built to know what it wants, and could easily have a utility function.
The consequences of this one difference are profound.
Evolved agents would be in rough equality to other agents. So, their game-theoretic considerations would be different from an artificial agent. The artificial agent could have a design very different from all other agents and also could far surpass other agents. Neither of these is possible in evolution.
In fact, because of the similarity between evolved agents in any given ecosystem, these game-theoretic considerations include not only the possibility of reciprocity or reciprocal altruism, but also the sort of acausal reciprocal morality explored by Drescher and MIRI—“you are like me, so my niceness is correlated with yours, so I’d better ask nicely.”