It seems like this would have issues once you want the AI to e.g. influence the world in a way that increases the number of happy humans—then you can’t just say “and also ‘happy humans’ have to have an origin causally separated from AI intervention.”
You wouldn’t say “and also ‘happy humans’ have to have an origin causally separated from AI intervention”, you’d say “and also ‘happy humans’ have to have an origin causally downstream from humans”.
Hm. But that doesn’t seem to stop the AI from creating molecular smiley faces whose precise form is causally dependent on what humans are like. The reason the causal origin argument helps for birds is because we can specify a model of the causal origin, and it’s just evolution with no room for AI intervention.
It seems like this would have issues once you want the AI to e.g. influence the world in a way that increases the number of happy humans—then you can’t just say “and also ‘happy humans’ have to have an origin causally separated from AI intervention.”
You wouldn’t say “and also ‘happy humans’ have to have an origin causally separated from AI intervention”, you’d say “and also ‘happy humans’ have to have an origin causally downstream from humans”.
Hm. But that doesn’t seem to stop the AI from creating molecular smiley faces whose precise form is causally dependent on what humans are like. The reason the causal origin argument helps for birds is because we can specify a model of the causal origin, and it’s just evolution with no room for AI intervention.