And who will choose the choosers? No sentient entity at all—they’ll be chosen they way they are today, by a wide variety of markets, except that there too the variety will be far greater.
Such markets and technologies are already far beyond the ability of any single human to comprehend[. . .]
Can you expand on this? The way you say it suggests that it might be your core objection to the thesis of economically explosive strong AI. -- put into words, the way the emotional charge would hook into the argument here would be: “Such a strong AI would have to be at least as smart as the market, and yet it would have been designed by humans, which would mean there had to be a human at least as smart as the market: and belief in this possibility is always hubris, and is characteristically disastrous for its bearer—something you always want to be on the opposite side of an argument from”? (Where “smart” here is meant to express something metaphorically similar to a proof system’s strength: “the system successfully uses unknowably diverse strategies that a lesser system would either never think to invent or never correctly decide how much to trust”.)
I guess, for this explanation to work, it also has to be your core objection to Friendly AI as a mitigation strategy: “No human-conceived AI architecture can subsume or substitue for all the lines of innovation that the future of the economy should produce, much less control such an economy to preserve any predicate relating to human values. Any preservation we are going to get is going to have to be built incrementally from empirical experience with incremental software economic threats to those values, each of which we will necessarily be able to overcome if there had ever been any hope for humankind to begin with; and it would be hubris, and throwing away any true hope we have, to cling to a chimerical hope of anything less partial, uncertain, or temporary.”
Can you expand on this? The way you say it suggests that it might be your core objection to the thesis of economically explosive strong AI. -- put into words, the way the emotional charge would hook into the argument here would be: “Such a strong AI would have to be at least as smart as the market, and yet it would have been designed by humans, which would mean there had to be a human at least as smart as the market: and belief in this possibility is always hubris, and is characteristically disastrous for its bearer—something you always want to be on the opposite side of an argument from”? (Where “smart” here is meant to express something metaphorically similar to a proof system’s strength: “the system successfully uses unknowably diverse strategies that a lesser system would either never think to invent or never correctly decide how much to trust”.)
I guess, for this explanation to work, it also has to be your core objection to Friendly AI as a mitigation strategy: “No human-conceived AI architecture can subsume or substitue for all the lines of innovation that the future of the economy should produce, much less control such an economy to preserve any predicate relating to human values. Any preservation we are going to get is going to have to be built incrementally from empirical experience with incremental software economic threats to those values, each of which we will necessarily be able to overcome if there had ever been any hope for humankind to begin with; and it would be hubris, and throwing away any true hope we have, to cling to a chimerical hope of anything less partial, uncertain, or temporary.”