Intriguing! But even granting that, there’s a big difference between extrapolating the values of a screwed-up offshoot of an entropy-optimizing process and extrapolating the value of “maximize entropy”. Or do you suspect that a FOOMing AI would be much less powerful and more prone to interesting errors than Eliezer believes?
Truly maximizing entropy would involve burning everything you can burn, tearing the matter of solar systems apart, accelerating stars towards nova, trying to accelerate the evaporation of black holes and prevent their formation, and other things of this sort. It’d look like a dark spot in the sky that’d get bigger at approximately the speed of light.
Fires are crude entropy maximisers. Living systems destroy energy dradients at all scales, resulting in more comprehensive devastation than mere flames can muster.
Of course, maximisation is often subject to constraints. Your complaint is rather like saying that water doesn’t “truly minimise” its altitude—since otherwise it would end up at the planet’s core. That usage is simply not what the terms “maximise” and “minimise” normally refer to.
Intriguing! But even granting that, there’s a big difference between extrapolating the values of a screwed-up offshoot of an entropy-optimizing process and extrapolating the value of “maximize entropy”. Or do you suspect that a FOOMing AI would be much less powerful and more prone to interesting errors than Eliezer believes?
Truly maximizing entropy would involve burning everything you can burn, tearing the matter of solar systems apart, accelerating stars towards nova, trying to accelerate the evaporation of black holes and prevent their formation, and other things of this sort. It’d look like a dark spot in the sky that’d get bigger at approximately the speed of light.
Fires are crude entropy maximisers. Living systems destroy energy dradients at all scales, resulting in more comprehensive devastation than mere flames can muster.
Of course, maximisation is often subject to constraints. Your complaint is rather like saying that water doesn’t “truly minimise” its altitude—since otherwise it would end up at the planet’s core. That usage is simply not what the terms “maximise” and “minimise” normally refer to.
Yeah! Compelling, but not “interesting”. Likewise, I expect that actually maximizing the fitness of a species would be similarly “boring”.