Thanks for the post! I just wanted to clarify what concept you’re pointing to with use of the word “deception”.
From Evan’s definition in RFLO, deception needs to involve some internal modelling of the base objective & training process, and instrumentally optimising for the base objective. He’s clarified in other comments that he sees “deception” as only referring to inner alignment failures, not outer (because deception is defined in terms of the interaction between the model and the training process, without introducing humans into the picture). This doesn’t include situations like the first one, where the reward function is underspecified to produce behaviour we want (although it does produce behaviour that looks like it’s what we want, unless we peer under the hood).
To put it another way, it seems like the way deception is used here refers to the general situation where “AI has learnt to do something that humans will misunderstand / misinterpret, regardless of whether the AI actually has an internal representation of the base objective it’s being trained on and the humans doing the training.”
In this situation, I don’t really know what the benefit is of putting these two scenarios into the same class, because they seem pretty different. My intuitions about this might be wrong though. Also I guess this is getting into the inner/outer alignment distinction which opens up quite a large can of worms!
Yeah, I totally agree. My motivation for writing the first section was that people use the word ‘deception’ to refer to both things, and then make what seem like incorrect inferences. For example, current ML systems do the ‘Goodhart deception’ thing, but then I’ve heard people use this to imply that it might be doing ‘consequentialist deception’.
These two things seem close to unrelated, except for the fact that ‘Goodhart deception’ shows us that AI systems are capable of ‘tricking’ humans.
Thanks for the post! I just wanted to clarify what concept you’re pointing to with use of the word “deception”.
From Evan’s definition in RFLO, deception needs to involve some internal modelling of the base objective & training process, and instrumentally optimising for the base objective. He’s clarified in other comments that he sees “deception” as only referring to inner alignment failures, not outer (because deception is defined in terms of the interaction between the model and the training process, without introducing humans into the picture). This doesn’t include situations like the first one, where the reward function is underspecified to produce behaviour we want (although it does produce behaviour that looks like it’s what we want, unless we peer under the hood).
To put it another way, it seems like the way deception is used here refers to the general situation where “AI has learnt to do something that humans will misunderstand / misinterpret, regardless of whether the AI actually has an internal representation of the base objective it’s being trained on and the humans doing the training.”
In this situation, I don’t really know what the benefit is of putting these two scenarios into the same class, because they seem pretty different. My intuitions about this might be wrong though. Also I guess this is getting into the inner/outer alignment distinction which opens up quite a large can of worms!
Yeah, I totally agree. My motivation for writing the first section was that people use the word ‘deception’ to refer to both things, and then make what seem like incorrect inferences. For example, current ML systems do the ‘Goodhart deception’ thing, but then I’ve heard people use this to imply that it might be doing ‘consequentialist deception’.
These two things seem close to unrelated, except for the fact that ‘Goodhart deception’ shows us that AI systems are capable of ‘tricking’ humans.
Okay I see, yep that makes sense to me (-: