Looking at the output of a black box is insufficient. You can only know by putting the black box in power, or by deeply understanding it. Humans are born into a world with others in power, so we know that most humans care about each other without knowing why. AI has no history of demonstrating friendliness in the only circumstances where that can be provably found. We can only know in advance by way of thorough understanding.
A strong theory about AI internals should come first. Refuting Yudkowsky’s theory about how it might go wrong is irrelevant.
Well, if someone originally started worrying based on strident predictions of sophisticated internal reasoning with goals independent of external behavior, then realizing that’s currently unsubstantiated should cause them to down-update on AI risk. That’s why it’s relevant. Although I think we should have good theories of AI internals.
I know I reacted to this comment, but I want to emphasize that this:
Well, if someone originally started worrying based on strident predictions of sophisticated internal reasoning with goals independent of external behavior,
Is to first order arguably the entire AI risk argument, that is if we make the assumption that the external behavior gives strong evidence about it’s internal structure, then there is no reason to elevate the AI risk argument at all, given the probably aligned behavior of GPTs when using RLHF.
More generally, the stronger the connection between external behavior and internal goals, the less worried you should be about AI safety, and this is a partial disagreement with people that are more pessimistic, albeit I have other disagreements there.
Humans are born into a world with others in power, so we know that most humans care about each other without knowing why.
I think the actual reason we believe humans could care about each other is because we’ve evolved the ability to do so, and that most humans share the same brain structure, and therefore the same tendency to care for people they consider their “ingroup”.
We do not know, that is the relevant problem.
Looking at the output of a black box is insufficient. You can only know by putting the black box in power, or by deeply understanding it.
Humans are born into a world with others in power, so we know that most humans care about each other without knowing why.
AI has no history of demonstrating friendliness in the only circumstances where that can be provably found. We can only know in advance by way of thorough understanding.
A strong theory about AI internals should come first. Refuting Yudkowsky’s theory about how it might go wrong is irrelevant.
Well, if someone originally started worrying based on strident predictions of sophisticated internal reasoning with goals independent of external behavior, then realizing that’s currently unsubstantiated should cause them to down-update on AI risk. That’s why it’s relevant. Although I think we should have good theories of AI internals.
I know I reacted to this comment, but I want to emphasize that this:
Is to first order arguably the entire AI risk argument, that is if we make the assumption that the external behavior gives strong evidence about it’s internal structure, then there is no reason to elevate the AI risk argument at all, given the probably aligned behavior of GPTs when using RLHF.
More generally, the stronger the connection between external behavior and internal goals, the less worried you should be about AI safety, and this is a partial disagreement with people that are more pessimistic, albeit I have other disagreements there.
I think the actual reason we believe humans could care about each other is because we’ve evolved the ability to do so, and that most humans share the same brain structure, and therefore the same tendency to care for people they consider their “ingroup”.