And that for every X except x0, it is mysteriously impossible to build any computational system which generates a range of actions, predicts the consequences of those actions relative to some ontology and world-model, and then selects among probable consequences using criterion X.
It sounds implausible when you put it like that, but suppose the only practical way to build a superintelligence is through some method that severely constrains the possible goals it might have (e.g., evolutionary methods, or uploading the smartest humans around and letting them self-modify), and attempts to build general purpose AIs/oracles/planning tools get nowhere (i.e., fail to be competitive against humans) until one is already a superintelligence.
Maybe when Bostrom/Armstrong/Yudkowsky talk about “possibility” in connection with the orthogonality thesis, they’re talking purely about theoretical possibility as opposed to practical feasibility. In fact Bostrom made this disclaimer in a footnote:
The orthogonality thesis implies that most any combination of final goal and intelligence level is logically possible; it does not imply that it would be practically easy to endow a superintelligent agent with some arbitrary or human-respecting final goal—even if we knew how to construct the intelligence part.
But then who are they arguing against? Are there any AI researchers who think that even given unlimited computing power and intelligence on the part of the AI builder, it’s still impossible to create AIs with arbitrary (or diverse) goals? This isn’t Pei Wang’s position, for example.
There are multiple variations on the OT, and the kind that just say it is possible can’t support the UFAI argument. The UFAI argument is conjunctive, and each stage in the conjunction needs to have a non-neglible probability, else it is a Pascal’s Mugging
build any computational system which generates a range of actions, predicts the consequences of those actions relative to some ontology and world-model, and then selects among probable consequences using criterion X.
Nothing mysterious here: this naive approach has incredibly low payoff per computation, and even if you start with such system, and get it to be smart enough to make improvements, the first thing it’ll be improving is changing it’s architecture.
If I gave you 10^40 flops, which probably can support ‘super intelligent’ mind, your naive approach would still be dumber than a housecat on many tasks. For some world evolution & utility, you can do inverse of the ‘simulate and choose’ much better (think towering exponents times better) than brute-force ‘try different actions’. In general you can’t. Some functions are easier to find inverse of, than others. A lot easier.
And that for every X except x0, it is mysteriously impossible to build any computational system which generates a range of actions, predicts the consequences of those actions relative to some ontology and world-model, and then selects among probable consequences using criterion X.
It sounds implausible when you put it like that, but suppose the only practical way to build a superintelligence is through some method that severely constrains the possible goals it might have (e.g., evolutionary methods, or uploading the smartest humans around and letting them self-modify), and attempts to build general purpose AIs/oracles/planning tools get nowhere (i.e., fail to be competitive against humans) until one is already a superintelligence.
Maybe when Bostrom/Armstrong/Yudkowsky talk about “possibility” in connection with the orthogonality thesis, they’re talking purely about theoretical possibility as opposed to practical feasibility. In fact Bostrom made this disclaimer in a footnote:
But then who are they arguing against? Are there any AI researchers who think that even given unlimited computing power and intelligence on the part of the AI builder, it’s still impossible to create AIs with arbitrary (or diverse) goals? This isn’t Pei Wang’s position, for example.
There are multiple variations on the OT, and the kind that just say it is possible can’t support the UFAI argument. The UFAI argument is conjunctive, and each stage in the conjunction needs to have a non-neglible probability, else it is a Pascal’s Mugging
I don’t think I’ve seen that particular reversal of the position before. Neat.
Yep. I’m calling that the “no Oracle, no general planning” position in my paper.
Nothing mysterious here: this naive approach has incredibly low payoff per computation, and even if you start with such system, and get it to be smart enough to make improvements, the first thing it’ll be improving is changing it’s architecture.
If I gave you 10^40 flops, which probably can support ‘super intelligent’ mind, your naive approach would still be dumber than a housecat on many tasks. For some world evolution & utility, you can do inverse of the ‘simulate and choose’ much better (think towering exponents times better) than brute-force ‘try different actions’. In general you can’t. Some functions are easier to find inverse of, than others. A lot easier.