Iterated Amplification is a fairly specific proposal for indefinitely scalable oversight, which doesn’t involve any human in the loop (if you start with a weak aligned AI). Recursive Reward Modeling is imagining (as I understand it) a human assisted by AIs to continuously do reward modeling; DeepMind’s original post about it lists “Iterated Amplification” as a separate research direction.
“Scalable Oversight”, as I understand it, refers to the research problem of how to provide a training signal to improve highly capable models. It’s the problem which IDA and RRM are both trying to solve. I think your summary of scalable oversight:
(Figuring out how to ease humans supervising models. Hard to cleanly distinguish from ambitious mechanistic interpretability but here we are.)
is inconsistent with how people in the industry use it. I think it’s generally meant to refer to the outer alignment problem, providing the right training objective. For example, here’s Anthropic’s “Measuring Progress on Scalable Oversight for LLMs” from 2022:
To build and deploy powerful AI responsibly, we will need to develop robust techniques for scalable oversight: the ability to provide reliable supervision—in the form of labels, reward signals, or critiques—to models in a way that will remain effective past the point that models start to achieve broadly human-level performance (Amodei et al., 2016).
It references “Concrete Problems in AI Safety” from 2016, which frames the problem in a closely related way, as a kind of “semi-supervised reinforcement learning”. In either case, it’s clear what we’re talking about is providing a good signal to optimize for, not an AI doing mechanistic interpretability on the internals of another model. I thus think it belongs more under the “Control the thing” header.
I think your characterization of “Prosaic Alignment” suffers from related issues. Paul coined the term to refer to alignment techniques for prosaic AI, not techniques which are themselves prosaic. Since prosaic AI is what we’re presently worried about, any technique to align DNNs is prosaic AI alignment, by Paul’s definition.
My understanding is that AI labs, particularly Anthropic, are interested in moving from human-supervised techniques to AI-supervised techniques, as part of an overall agenda towards indefinitely scalable oversight via AI self-supervision. I don’t think Anthropic considers RLAIF an alignment endpoint itself.
Iterated Amplification is a fairly specific proposal for indefinitely scalable oversight, which doesn’t involve any human in the loop (if you start with a weak aligned AI). Recursive Reward Modeling is imagining (as I understand it) a human assisted by AIs to continuously do reward modeling; DeepMind’s original post about it lists “Iterated Amplification” as a separate research direction.
“Scalable Oversight”, as I understand it, refers to the research problem of how to provide a training signal to improve highly capable models. It’s the problem which IDA and RRM are both trying to solve. I think your summary of scalable oversight:
is inconsistent with how people in the industry use it. I think it’s generally meant to refer to the outer alignment problem, providing the right training objective. For example, here’s Anthropic’s “Measuring Progress on Scalable Oversight for LLMs” from 2022:
It references “Concrete Problems in AI Safety” from 2016, which frames the problem in a closely related way, as a kind of “semi-supervised reinforcement learning”. In either case, it’s clear what we’re talking about is providing a good signal to optimize for, not an AI doing mechanistic interpretability on the internals of another model. I thus think it belongs more under the “Control the thing” header.
I think your characterization of “Prosaic Alignment” suffers from related issues. Paul coined the term to refer to alignment techniques for prosaic AI, not techniques which are themselves prosaic. Since prosaic AI is what we’re presently worried about, any technique to align DNNs is prosaic AI alignment, by Paul’s definition.
My understanding is that AI labs, particularly Anthropic, are interested in moving from human-supervised techniques to AI-supervised techniques, as part of an overall agenda towards indefinitely scalable oversight via AI self-supervision. I don’t think Anthropic considers RLAIF an alignment endpoint itself.