To me it seems a solid attempt at conveying [misalignment is possible, even with a good test], but not necessarily [misalignment is likely, even with a good test]. (not that I have a great alternative suggestion)
Important disanalogies seem: 1) Most humans aren’t good at convincingly faking niceness (I think!). The listener may assume a test good enough to successfully exploit this most of the time. 2) The listener will assume that [score highly on niceness] isn’t the human’s only reward. (both things like [desire to feel honest] and [worry of the consequences of being caught cheating]) 3) A fairly large proportion of humans are nice (I think!).
The second could be addressed somewhat by raising the stakes. The first seems hard to remedy within this analogy. I’d be a little concerned that people initially buy it, then think for themselves and conclude “But if we design a really clever niceness test, then it’d almost always work—all we need is clever people to work for a while on some good tests”. Combined with (3), this might seem like a decent solution.
Overall, I think what’s missing is that we’d expect [our clever test looks to us as if it works] well before [our clever test actually works]. My guess is that the layperson isn’t going to have this intuition in the human-niceness-test case.
To me it seems a solid attempt at conveying [misalignment is possible, even with a good test], but not necessarily [misalignment is likely, even with a good test]. (not that I have a great alternative suggestion)
Important disanalogies seem:
1) Most humans aren’t good at convincingly faking niceness (I think!). The listener may assume a test good enough to successfully exploit this most of the time.
2) The listener will assume that [score highly on niceness] isn’t the human’s only reward. (both things like [desire to feel honest] and [worry of the consequences of being caught cheating])
3) A fairly large proportion of humans are nice (I think!).
The second could be addressed somewhat by raising the stakes.
The first seems hard to remedy within this analogy. I’d be a little concerned that people initially buy it, then think for themselves and conclude “But if we design a really clever niceness test, then it’d almost always work—all we need is clever people to work for a while on some good tests”.
Combined with (3), this might seem like a decent solution.
Overall, I think what’s missing is that we’d expect [our clever test looks to us as if it works] well before [our clever test actually works]. My guess is that the layperson isn’t going to have this intuition in the human-niceness-test case.