Expanding a bit on gallabytes’s comment: The language around Moloch often assumes a Nash equilibrium, i.e. a situation in which a rational agent implementing causal decision theory couldn’t do better. Sometimes the agents aren’t general intelligences, but are simpler evolutionarily fit processes responding to feedback.
I’m perhaps a bit of an outlier in that I see less difference among powerful optimization processes (evolution and humans being the only real examples I know of, and both extend to artificial versions) than most.
Over sufficient time periods, evolution is subject to the same constraints and tradeoffs as intelligent world-modeling agents are. Evolution is slow enough that it can bypass some of the identity problems that agents have, but it’s also only going to find viable equilibria.
Lookahead is unnecessary if you can actually perform the iterations. There is some subtlety around path-dependency and needing to survive the iterations in order to arrive at the equilibrium, but for simple cases like this one, it just doesn’t matter. The strategy will be found whether the intermediate states are imagined hypothetically by an intelligence, or just executed physically by a patient experimenter.
What do you mean by “simple cases like this one?” Empirically, evolution quite often ends up in a Nash equilibrium of conflict where a negotiated solution would have less deadweight loss.
Simple cases like the ants or like toy problems where humans usually get the right answer (and some where we don’t). In cases where iterated reasoning can come up with a solution, evolution will be MUCH slower but will come up with answers as good as any modeled reasoning engine.
(note: I’m overstating this by quite a lot. The effects of path-dependency and search breadth for evolution and of modeling limitations and limited capacity for brains can make orders of magnitude difference in the solutions found. In simple theory, though, they’re roughly equivalent.)
The fact that evolution is adequate to produce ants doesn’t really have much bearing on anything here, unless there’s also reason to believe that lookahead can’t do better than ants, which is clearly absurd. Even if the moon were a rich source of calories (say, by having comparatively unimpeded access to sunlight), evolution just doesn’t know how to get there and can’t figure it out by iteration. Humans clearly can in principle, it’s hard for us but obviously within our reach as a species, and not by natural selection for flight.
Panspermia theories have vulcanic activity and meteor strikes moving bacteria world to world. It’s not clear it’s off limit to evolution (or one needs to do some tricky organic world vs inorganic world boundary drawing to get a motivated cognition result).
Some strctures less complex than brains might be selected for “look ahead” like benefit. For example the evolution of sex. Also having features coded in multiple ways in DNA. Some of the DNA encodings might be selected for “evolvability”. Making things like epigenetic switches and in general control genes can be seen as a modelling layer a bit more abstract than concrete features.
Expanding a bit on gallabytes’s comment: The language around Moloch often assumes a Nash equilibrium, i.e. a situation in which a rational agent implementing causal decision theory couldn’t do better. Sometimes the agents aren’t general intelligences, but are simpler evolutionarily fit processes responding to feedback.
I’m perhaps a bit of an outlier in that I see less difference among powerful optimization processes (evolution and humans being the only real examples I know of, and both extend to artificial versions) than most.
Over sufficient time periods, evolution is subject to the same constraints and tradeoffs as intelligent world-modeling agents are. Evolution is slow enough that it can bypass some of the identity problems that agents have, but it’s also only going to find viable equilibria.
Evolution doesn’t have lookahead—or modeling the problem at all—except via evolving things like brains.
Lookahead is unnecessary if you can actually perform the iterations. There is some subtlety around path-dependency and needing to survive the iterations in order to arrive at the equilibrium, but for simple cases like this one, it just doesn’t matter. The strategy will be found whether the intermediate states are imagined hypothetically by an intelligence, or just executed physically by a patient experimenter.
What do you mean by “simple cases like this one?” Empirically, evolution quite often ends up in a Nash equilibrium of conflict where a negotiated solution would have less deadweight loss.
Simple cases like the ants or like toy problems where humans usually get the right answer (and some where we don’t). In cases where iterated reasoning can come up with a solution, evolution will be MUCH slower but will come up with answers as good as any modeled reasoning engine.
(note: I’m overstating this by quite a lot. The effects of path-dependency and search breadth for evolution and of modeling limitations and limited capacity for brains can make orders of magnitude difference in the solutions found. In simple theory, though, they’re roughly equivalent.)
The fact that evolution is adequate to produce ants doesn’t really have much bearing on anything here, unless there’s also reason to believe that lookahead can’t do better than ants, which is clearly absurd. Even if the moon were a rich source of calories (say, by having comparatively unimpeded access to sunlight), evolution just doesn’t know how to get there and can’t figure it out by iteration. Humans clearly can in principle, it’s hard for us but obviously within our reach as a species, and not by natural selection for flight.
Panspermia theories have vulcanic activity and meteor strikes moving bacteria world to world. It’s not clear it’s off limit to evolution (or one needs to do some tricky organic world vs inorganic world boundary drawing to get a motivated cognition result).
Some strctures less complex than brains might be selected for “look ahead” like benefit. For example the evolution of sex. Also having features coded in multiple ways in DNA. Some of the DNA encodings might be selected for “evolvability”. Making things like epigenetic switches and in general control genes can be seen as a modelling layer a bit more abstract than concrete features.