Roko, see “Optimization and the Singularity” and “DNA Consequentialism and Protein Reinforcement” on the concept of cross-domain optimization. Yes, this is part of what distinguishes human general intelligence and natural selection, from modern-day narrow-domain Artificial Intelligences and nonprimate organisms. However, this doesn’t present any challenge to my criterion of optimization, except for the possibility of needing to reframe your terminal values for a new ontology (which I’ve also discussed in my metaethics, under “Unnatural Categories”). Instrumentally speaking, and assuming away the problem of unnatural categories, you’re just finding new paths to the same goals, leading through portions of reality that you may not have suspected existed. Sort of like humans digging miles under the ground to obtain geothermal energy in order to light their homes—they’re steering the future into the same region as did the maker of candles, but wending through parts of reality and laws of physics that ancient candle-makers didn’t suspect existed.
Roko, see “Optimization and the Singularity” and “DNA Consequentialism and Protein Reinforcement” on the concept of cross-domain optimization. Yes, this is part of what distinguishes human general intelligence and natural selection, from modern-day narrow-domain Artificial Intelligences and nonprimate organisms. However, this doesn’t present any challenge to my criterion of optimization, except for the possibility of needing to reframe your terminal values for a new ontology (which I’ve also discussed in my metaethics, under “Unnatural Categories”). Instrumentally speaking, and assuming away the problem of unnatural categories, you’re just finding new paths to the same goals, leading through portions of reality that you may not have suspected existed. Sort of like humans digging miles under the ground to obtain geothermal energy in order to light their homes—they’re steering the future into the same region as did the maker of candles, but wending through parts of reality and laws of physics that ancient candle-makers didn’t suspect existed.