If you created a misaligned AI, then it would be “thinking back”, and you’d be in an adversarial position where security mindset is appropriate.
Cool, we agree on this point.
my point in that section is that the fundamental laws governing how AI training processes work are not “thinking back”. They’re not adversaries.
I think we agree here on the local point but disagree on its significance to the broader argument. [I’m not sure how much we agree-I think of training dynamics as ‘neutral’, but also I think of them as searching over program-space in order to find a program that performs well on a (loss function, training set) pair, and so you need to be reasoning about search. But I think we agree the training dynamics are not trying to trick you / be adversarial and instead are straightforwardly ‘trying’ to make Number Go Down.]
In my picture, we have the neutral training dynamics paired with the (loss function, training set) which creates the AI system, and whether the resulting AI system is adversarial or not depends mostly on the choice of (loss function, training set). It seems to me that we probably have a disagreement about how much of the space of (loss function, training set) leads to misaligned vs. aligned AI (if it hits ‘AI’ at all), where I think aligned AI is a narrow target to hit that most loss functions will miss, and hitting that narrow target requires security mindset.
To explain further, it’s not that the (loss function, training set) is thinking back at you on its own; it’s that the AI that’s created by training is thinking back at you. So before you decide to optimize X you need to check whether or not you actually want something that’s optimizing X, or if you need to optimize for Y instead.
So from my perspective it seems like you need security mindset in order to pick the right inputs to ML training to avoid getting misaligned models.
Cool, we agree on this point.
I think we agree here on the local point but disagree on its significance to the broader argument. [I’m not sure how much we agree-I think of training dynamics as ‘neutral’, but also I think of them as searching over program-space in order to find a program that performs well on a (loss function, training set) pair, and so you need to be reasoning about search. But I think we agree the training dynamics are not trying to trick you / be adversarial and instead are straightforwardly ‘trying’ to make Number Go Down.]
In my picture, we have the neutral training dynamics paired with the (loss function, training set) which creates the AI system, and whether the resulting AI system is adversarial or not depends mostly on the choice of (loss function, training set). It seems to me that we probably have a disagreement about how much of the space of (loss function, training set) leads to misaligned vs. aligned AI (if it hits ‘AI’ at all), where I think aligned AI is a narrow target to hit that most loss functions will miss, and hitting that narrow target requires security mindset.
To explain further, it’s not that the (loss function, training set) is thinking back at you on its own; it’s that the AI that’s created by training is thinking back at you. So before you decide to optimize X you need to check whether or not you actually want something that’s optimizing X, or if you need to optimize for Y instead.
So from my perspective it seems like you need security mindset in order to pick the right inputs to ML training to avoid getting misaligned models.