I don’t expect such a huge gap between debaters and judges that the judge simply can’t understand the debaters’ concepts
You don’t? But this is a major problem in arguments between people. The variation within humans is already more than enough for this! There’s a gap like that every 35 IQ points or so. I don’t understand why you’re confident this isn’t an issue.
I guess we’ve found our main disagreement, at least?
So in this particular case I am saying: if you penalize debaters that are inconsistent under cross-examination, you are giving an advantage to any debater that implements an honest strategy, and so you should expect training to incentivize honesty.
Now you’re training for multiple objectives:
You want the debater AI to argue for proposition A or not-A according to its role and convince human judges of that.
You want it to not change its position on sub-arguments.
But (2) is ill-defined. Can sub-arguments be combined for less weighting? Are they all worth the same? What if you have several sub-arguments that all depend on a single sub-2-argument? Good arguments for A or not-A should have lots of disagreements—or do you want to train AI that makes all the same sub-arguments for A or not-A and then says “this implies A / not-A”? I don’t think this works.
In response to the linked “HCH” post:
Yes, an agent past some threshold can theoretically make a more-intelligent agent. But that doesn’t say anything about alignment; the supposed “question-answering machine” would be subject to instrumental convergence and mesaoptimizer issues, and you’d get value drift with each HCH stage, just as you would with RSI schemes.
@bhauth@Rohin Shah I think that bhauth has an important point here about the danger of large gaps between judge and debaters. Similarly, between a trusted overseer and a smarter worker. Keeping the gaps small is really important for a lot of oversight plans to work out well!
Here’s some research I am doing which I think answers this point thoroughly: it is possible to smoothly, continuously, incrementally scale-down the capabilities of a model by injecting carefully controlled amounts of noise into its activations. I’m calling this ‘noise injection impairment’.
This removes the need to have precisely created a whole series of models with precise capability steps between each one. You can instead train a single strong model, and scale it all the way down to be just a tiny step above the next most strong model. Then you create as large a number of intermediate steps of capability as you need by reducing the noise magnitude.
Without this technique, then I believe bhauth’s point would stand, and capability gaps between model versions would lead to dangerous failures of various control and monitoring schemes.
I think the basic idea of using more steps of smaller size is worth considering. Maybe it reduces overall drift, but I suspect it doesn’t, because my view is:
Models have many basins of attraction for sub-elements. As model capability increases continuously, there are nearly-discrete points where aspects of the model jump from 1 basin to another, perhaps with cascading effect. I expect this to produce large gaps from small changes to models.
I’m not going to repeat all of the literature on debate here, but as brief pointers:
Factored cognition discusses intuitively why we can hope to approximate exponentially-sized trees of arguments (which would be tremendously bigger than arguments between people)
AI safety via debate makes the same argument for debate (by showing that a polynomial time judge can supervise PSPACE—PSPACE-complete problems typically involve exponential-sized trees)
This paper discusses the experiments you’d do to figure out what the human judge should be doing to make debate more effective
The comments on this post discuss several reasons not to anchor to human institutions. There are even more reasons not to anchor to disagreements between people, but I didn’t find a place where they’ve been written up with a short search. Most centrally, disagreements between people tend to focus on getting both people to understand their position, but the theoretical story for debate does not require this.
(Also, the “arbitrary amounts of time and arbitrary amounts of explanation” was pretty central to my claim; human disagreements are way more bounded than that.)
The scope of our argument seems to have grown beyond what a single comment thread is suitable for.
AI safety via debate is 2 years before Writeup: Progress on AI Safety via Debate so the latter post should be more up-to-date. I think that post does a good job of considering potential problems; the issue is that I think the noted problems & assumptions can’t be handled well, make that approach very limited in what it can do for alignment, and aren’t really dealt with by “Doubly-efficient debate”. I don’t think such debate protocols are totally useless, but they’re certainly not a “solution to alignment”.
You don’t? But this is a major problem in arguments between people. The variation within humans is already more than enough for this! There’s a gap like that every 35 IQ points or so. I don’t understand why you’re confident this isn’t an issue.
I guess we’ve found our main disagreement, at least?
Now you’re training for multiple objectives:
You want the debater AI to argue for proposition A or not-A according to its role and convince human judges of that.
You want it to not change its position on sub-arguments.
But (2) is ill-defined. Can sub-arguments be combined for less weighting? Are they all worth the same? What if you have several sub-arguments that all depend on a single sub-2-argument? Good arguments for A or not-A should have lots of disagreements—or do you want to train AI that makes all the same sub-arguments for A or not-A and then says “this implies A / not-A”? I don’t think this works.
In response to the linked “HCH” post:
Yes, an agent past some threshold can theoretically make a more-intelligent agent. But that doesn’t say anything about alignment; the supposed “question-answering machine” would be subject to instrumental convergence and mesaoptimizer issues, and you’d get value drift with each HCH stage, just as you would with RSI schemes.
@bhauth @Rohin Shah I think that bhauth has an important point here about the danger of large gaps between judge and debaters. Similarly, between a trusted overseer and a smarter worker. Keeping the gaps small is really important for a lot of oversight plans to work out well!
Here’s some research I am doing which I think answers this point thoroughly: it is possible to smoothly, continuously, incrementally scale-down the capabilities of a model by injecting carefully controlled amounts of noise into its activations. I’m calling this ‘noise injection impairment’.
This removes the need to have precisely created a whole series of models with precise capability steps between each one. You can instead train a single strong model, and scale it all the way down to be just a tiny step above the next most strong model. Then you create as large a number of intermediate steps of capability as you need by reducing the noise magnitude.
Without this technique, then I believe bhauth’s point would stand, and capability gaps between model versions would lead to dangerous failures of various control and monitoring schemes.
Link to details of ongoing research: https://www.apartresearch.com/project/sandbag-detection-through-model-degradation
I think the basic idea of using more steps of smaller size is worth considering. Maybe it reduces overall drift, but I suspect it doesn’t, because my view is:
I’m not going to repeat all of the literature on debate here, but as brief pointers:
Factored cognition discusses intuitively why we can hope to approximate exponentially-sized trees of arguments (which would be tremendously bigger than arguments between people)
AI safety via debate makes the same argument for debate (by showing that a polynomial time judge can supervise PSPACE—PSPACE-complete problems typically involve exponential-sized trees)
Cross-examination is discussed here
This paper discusses the experiments you’d do to figure out what the human judge should be doing to make debate more effective
The comments on this post discuss several reasons not to anchor to human institutions. There are even more reasons not to anchor to disagreements between people, but I didn’t find a place where they’ve been written up with a short search. Most centrally, disagreements between people tend to focus on getting both people to understand their position, but the theoretical story for debate does not require this.
(Also, the “arbitrary amounts of time and arbitrary amounts of explanation” was pretty central to my claim; human disagreements are way more bounded than that.)
The scope of our argument seems to have grown beyond what a single comment thread is suitable for.
AI safety via debate is 2 years before Writeup: Progress on AI Safety via Debate so the latter post should be more up-to-date. I think that post does a good job of considering potential problems; the issue is that I think the noted problems & assumptions can’t be handled well, make that approach very limited in what it can do for alignment, and aren’t really dealt with by “Doubly-efficient debate”. I don’t think such debate protocols are totally useless, but they’re certainly not a “solution to alignment”.