Wei, the criterion “intelligent” compresses down to a very simple notions of effective implementation abstracted away from the choice of goal. After that, the only question is how to get “intelligence”, which is something you can, in principle, learn by observation (if you start with learning ability). Flaws in a notion of “intelligence” can be self-corrected if not too great; you observe that what you’re doing isn’t working (for your goal criterion).
Morality does not compress; it’s not something you can learn just by looking at the (nonhuman) environment or by doing logic; if you want to get all the details correct, you have to look at human brains.
That’s the difference.
Samantha, were you around for the metaethics sequence?
Wei, the criterion “intelligent” compresses down to a very simple notions of effective implementation abstracted away from the choice of goal. After that, the only question is how to get “intelligence”, which is something you can, in principle, learn by observation (if you start with learning ability). Flaws in a notion of “intelligence” can be self-corrected if not too great; you observe that what you’re doing isn’t working (for your goal criterion).
Morality does not compress; it’s not something you can learn just by looking at the (nonhuman) environment or by doing logic; if you want to get all the details correct, you have to look at human brains.
That’s the difference.
Samantha, were you around for the metaethics sequence?