The way I’d put it is that there are many personas that a person or LLM can play—many masks they can wear—and what we care about is which one they wear in high-stakes situations where e.g. they have tons of power and autonomy and no one is able to check what they are doing or stop them. (You can perhaps think of this one as the “innermost mask”)
The problem you are pointing to, I think, is that behavioral training is insufficient for assurance-of-alignment, and probably insufficient for alignment, full stop.
This doesn’t mean it’s theoretically impossible to align superhuman AIs. It can be done, but your alignment techniques will need to be more sophisticated than “We trained it to behave in ways that appear nice to us, and so far it seems to be doing so.” For example they may involve mechanistic interpretability.
what we care about is which one they wear in high-stakes situations where e.g. they have tons of power and autonomy and no one is able to check what they are doing or stop them. (You can perhaps think of this one as the “innermost mask”)
I think there are also valuable questions to be asked about attractors in persona space—what personas does an LLM gravitate to across a wide range of scenarios, and what sorts of personas does it always or never adopt? I’m not aware of much existing research in this direction, but it seems valuable. If for example we could demonstrate certain important bounds (‘This LLM will never adopt a mass-murderer persona’) there’s potential alignment value there IMO.
The way I’d put it is that there are many personas that a person or LLM can play—many masks they can wear—and what we care about is which one they wear in high-stakes situations where e.g. they have tons of power and autonomy and no one is able to check what they are doing or stop them. (You can perhaps think of this one as the “innermost mask”)
The problem you are pointing to, I think, is that behavioral training is insufficient for assurance-of-alignment, and probably insufficient for alignment, full stop.
This doesn’t mean it’s theoretically impossible to align superhuman AIs. It can be done, but your alignment techniques will need to be more sophisticated than “We trained it to behave in ways that appear nice to us, and so far it seems to be doing so.” For example they may involve mechanistic interpretability.
I agree with Daniel here but would add one thing:
I think there are also valuable questions to be asked about attractors in persona space—what personas does an LLM gravitate to across a wide range of scenarios, and what sorts of personas does it always or never adopt? I’m not aware of much existing research in this direction, but it seems valuable. If for example we could demonstrate certain important bounds (‘This LLM will never adopt a mass-murderer persona’) there’s potential alignment value there IMO.
This is also a very interesting point, thank you!
Thank you! That helps me understanding the problem better, although I’m quite skeptical about mechanistic interpretability.