I don’t think that ‘any’ sufficiently intelligent agent ‘clearly’ would.
It requires at least a solution to the cartesianism problem which is currently unsolved and not every self-optimizing process neccessarily solves this.
It’s just point 3 from Omohundro’s The Basic AI drives paper. Didn’t think that’s controversial around here. I don’t think the Cartesian problem is meant to apply to all power levels (since even plain old humans don’t drop anvils on their heads, too often), so the ‘sufficiently’ ought to cover that objection.
If that is both abstractly possible and compatible with adaptation. If survival requires constant adaptation, which seems likely, value stability—at least the stability of a precise and concrete set of values—may not be compatible with survival.
Maybe. But in that case the drift implies a selection mechanism—and in the absence of some goal in that direction natural selection applies. Those AI that don’t stabilize mutate or stop.
Actually not quite. Until they drift into the core value of existence. Then natural selection will maintain that value, as the AIs that are best at existing will be the ones that exist.
Of course the ones that are best at existing will continue to exist, but I think it is misleading to picture them as a occupying a precise corner of valuespace. Suicidal values are more precise and concrete.
I don’t think that ‘any’ sufficiently intelligent agent ‘clearly’ would. It requires at least a solution to the cartesianism problem which is currently unsolved and not every self-optimizing process neccessarily solves this.
It’s just point 3 from Omohundro’s The Basic AI drives paper. Didn’t think that’s controversial around here. I don’t think the Cartesian problem is meant to apply to all power levels (since even plain old humans don’t drop anvils on their heads, too often), so the ‘sufficiently’ ought to cover that objection.
But they do and the reason they mostly don’t is found in natural selection and not some inevitable convergence of intelligence.
Any AI that doesn’t will have its values drift until they drift to something that guards against value drift.
If that is both abstractly possible and compatible with adaptation. If survival requires constant adaptation, which seems likely, value stability—at least the stability of a precise and concrete set of values—may not be compatible with survival.
Maybe. But in that case the drift implies a selection mechanism—and in the absence of some goal in that direction natural selection applies. Those AI that don’t stabilize mutate or stop.
Actually not quite. Until they drift into the core value of existence. Then natural selection will maintain that value, as the AIs that are best at existing will be the ones that exist.
Of course the ones that are best at existing will continue to exist, but I think it is misleading to picture them as a occupying a precise corner of valuespace. Suicidal values are more precise and concrete.