Thanks a lot for this comment. These are extremely valid concerns that we’ve been thinking about a lot.
I’d just like the designers of alignment-research boosting tools to have clear arguments that nothing of this sort is likely.
I don’t think this is feasible given our current understanding of epistemology in general and epistemology of alignment research in particular. The problems you listed are potential problems with any methodology, not just AI assisted research. Being able to look at a proposed method and make clear arguments that it’s unlikely to have any undesirable incentives or negative second order effects, etc, is the holy grail of applied epistemology and one of the cores of the alignment problem.
For now, the best we can do is be aware of these concerns, work to improve our understanding of the underlying epistemological problem, design the tools and methods in a way that avoids problems (or at least make them likely to be noticed) according to our current best understanding, and actively address them in the process.
On a high level, it seems wise to me to follow these principles:
Approach this as an epistemology problem
Optimize for augmenting human cognition rather than outsourcing cognitive labor or producing good-looking outputs
Short feedback loops and high bandwidth (both between human<>AI and tool users<>tool designers)
Avoid incentivizing the AI components to goodhart against human evaluation
Avoid producing/releasing infohazards
All of these are hard problems. I could write many pages about each of them, and hopefully will at some point, but for now I’ll only address them briefly in relation to your comment.
1. Approach this as an epistemology problem
We don’t know how to evaluate whether a process is going to be robustly truth-seeking (or {whatever you really want}-seeking). Any measure will be a proxy susceptible to goodhart.
one of the first that occurs to me is that such cyborgism is unlikely to amplify production of useful-looking alignment ideas uniformly in all directions.
Suppose that it makes things 10x faster in various directions that look promising, but don’t lead to solutions, but only 2x faster in directions that do lead to solutions.
This is a concern for any method, including things like “post your work frequently and get a lot of feedback” or “try to formalize stuff”
Introducing AI into it just makes the problem much more explicit and pressing (because of the removal of the “protected meta level”).
I intend to work closely with the Conjecture epistemology/methodologies team in this project. After all, this is kinda the ultimate challenge for epistemology: as the saying goes, you don’t understand something until you can build it.
We need to better understand things like:
What are the current bottlenecks on human cognition and more specifically alignment research, and can/do these tools actually help remove them?
is thinking about “bottlenecks” the right abstraction? especially if there’s a potential to completely transform the workflow, instead of just unblocking what we currently recognize as bottlenecks
What do processes that generate good ideas/solutions look like in practice?
What do the examples we have access to tell us about the underlying mechanisms of effective processes?
To what extent are productive processes legible? Can we make them more legible, and what are the costs/benefits of doing so? How do we avoid goodharting against legibility when it’s incentivized (AI assisted research is one such situation)?
How can you evaluate if an idea is actually good, and doesn’t just “look” good?
What are the different ways an idea can “look” good and how can each of these be operationalized or fail? (e.g. “feels” meaningful/makes you feel less confused, experts in the field think it’s good, LW karma, can be formalized/mathematically verified, can be/is experimentally verified, other processes independently arrive at same idea, inspires more ideas, leads to useful applications, “big if true”, etc)
How can we avoid applying too much optimization pressure to things that “look” good considering that we ultimately only have access to how things “look” to us (or some externalized measure)?
How do asymmetric capabilities affect all this? As you said, AI will amplify cognition more effectively in some ways than others.
Humans already have asymmetric capabilities as well (though it’s unclear clear what “symmetry” would mean...). How does this affect how currently we do research?
How do we leverage asymmetric capabilities without over-relying on them?
How can we tell whether capabilities are intrinsically asymmetric or are just asymmetrically bottlenecked by how we’re trying to use them?
Dual to the concerns re asymmetrical capabilities: What kind of truth-seeking processes can AI enable which are outside the scope of how humans currently do research due to cognitive limitations?
Being explicitly aware of these considerations is the first step. For instance, with regards to the concern about perception of progress due to “speed”:
Warp our perception of promising directions: once the 10x directions seem to be producing progress much faster, it’ll be difficult not to interpret this as evidence they’re more promising.
Obviously you can write much faster and with superficial fluency with an AI assistant, so we need to adjust our evaluation of output in light of that fact.
2. Optimize for augmenting human cognition rather than outsourcing cognitive labor or producing good-looking outputs
One common conception of computers is that they’re problem-solving machines: “computer, what is the result of firing this artillery shell in such-and-such a wind [and so on]?”; “computer, what will the maximum temperature in Tokyo be in 5 days?”; “computer, what is the best move to take when the Go board is in this position?”; “computer, how should this image be classified?”; and so on.
This is a conception common to both the early view of computers as number-crunchers, and also in much work on AI, both historically and today. It’s a model of a computer as a way of outsourcing cognition. In speculative depictions of possible future AI, this cognitive outsourcing model often shows up in the view of an AI as an oracle, able to solve some large class of problems with better-than-human performance.
But a very different conception of what computers are for is possible, a conception much more congruent with work on intelligence augmentation.
...
It’s this kind of cognitive transformation model which underlies much of the deepest work on intelligence augmentation. Rather than outsourcing cognition, it’s about changing the operations and representations we use to think; it’s about changing the substrate of thought itself. And so while cognitive outsourcing is important, this cognitive transformation view offers a much more profound model of intelligence augmentation. It’s a view in which computers are a means to change and expand human thought itself.
I think the cognitive transformation approach is more promising from an epistemological standpoint because the point is to give the humans an inside view of the process by weaving the cognitive operations enabled by the AI into the user’s thinking, rather than just producing good-seeming artifacts. In other words, we want to amplify the human’s generator, not just rely on human evaluation of an external generation process.
This does not solve the goodhart problem (you might feel like the AI is improving your cognition without actually being productive), but it enables a form of “supervision” that is closer to the substrate of cognition and thus gives the human more intimate insight into whether and why things are working or not.
I also expect the cognitive transformation model to be significantly more effective in the near future. But as AIs become more capable it will be more tempting to increase the length of feedback loops & supervise outcomes instead of process. Hopefully building tools and gaining hands-on experience now will give us more leverage to continue using AI as cognitive augmentation rather than just outsourcing cognition once the latter becomes “easier”.
In the short term, process-based ML systems have better differential capabilities: They help us apply ML to tasks where we don’t have access to outcomes. These tasks include long-range forecasting, policy decisions, and theoretical research.
In the long term, process-based ML systems help avoid catastrophic outcomes from systems gaming outcome measures and are thus more aligned.
Both process- and outcome-based evaluation are attractors to varying degrees: Once an architecture is entrenched, it’s hard to move away from it. This lock-in applies much more to outcome-based systems.
Whether the most powerful ML systems will primarily be process-based or outcome-based is up in the air.
So it’s crucial to push toward process-based training now.
A major part of the work here will be designing interfaces which surface the “cognitive primitives” as control levers and make high bandwidth interaction & feedback possible.
Slightly more concretely, GPTs are conditional probability distributions one can control by programming boundary conditions (“prompting”), searching through stochastic ramifications (“curation”), and perhaps also manipulating latents (see this awesome blog post Imagining better interfaces to language models). The probabilistic simulator (or speculator) itself and each of these control methods, I think, have close analogues to how we operate our own minds, and thus I think it’s possible with the right interface to “stitch” the model to our minds in a way that acts as a controllable extension of thought. This is a very different approach to “making GPT useful” than, say, InstructGPT, and it’s why I call it cyborgism.
3. Short feedback loops and high bandwidth (both between human<>AI and tool users<>tool designers)
Short feedback loops and high bandwidth between the human and AI is integral the cognitive augmentation perspective: you want as much of the mission-relevant information to be passing through (and understood by) the human user as possible. Not only is this more helpful to the human, it gives them opportunities to notice problems and course-correct at the process level which may not be transparent at all in more oracle or genie-like approaches.
For similar reasons, we want short feedback loops between the users and designers/engineers of the tools (ideally the user designs the tool—needless to say, I will be among the first of the cyborgs I make). We want to be able to inspect the process on a meta level and notice and address problems like goodhart or mode collapse as soon as possible.
4. Avoid incentivizing the AI components to goodhart against human evaluation
This is obvious but hard to avoid, because we do want to improve the system and human evaluation is the main source of feedback we have. But I think there are concrete ways to avoid the worst here, like being very explicit about where and how much optimization pressure is being applied and avoiding methods which extrapolate proxies of human evaluation with unbounded atomic optimization.
There are various reasons I plan to avoid RLHF (except for purposes of comparison); this is one of them. This is not to say other methods that leverage human feedback are immune to goodhart, but RLHF is particularly risky because you’re creating a model(proxy) of human evaluation of outcomes and optimizing against it (the ability to apply unbounded optimization against the reward model is the reason to make one in the first place rather than training against human judgements directly).
I’m more interested in approaches that interactively prototype effective processes & use them as supervised examples to augment the model’s prior: scouting the space of processes rather than optimizing a fixed measure of what a good outcome looks like. Of course, we must still rely on human judgment to say what a good process is (at various levels of granularity, e.g. curation of AI responses and meta-selection of approaches based on perceived effectiveness), so we still need be wary of goodhart. But I think avoiding direct optimization pressure toward outcome evaluations can go a long way. Supervise Process, not Outcomes contains more in depth reasoning on this point.
That said, it’s important to emphasize that this is not a proposal to solve alignment, but the much easier (though still hard) problem of shaping an AI system to augment alignment research before foom. I don’t expect these methods to scale to aligning a superintelligent AI; I expect conceptual breakthroughs will be necessary for that and iterative approaches alone will fail. The motivation for this project is my belief that AI augmentation can put us in a better position to make those conceptual breakthroughs.
5. Avoid producing/releasing infohazards
I won’t say too much about this now, but anything that we identify to present a risk of accelerating capabilities will be covered under Conjecture’s infohazard policy.
I want to talk about why automation is likely more dangerous and more useful than cyborgization, and the reason is Amdahl’s law.
In other words, the slowest process controls the outcome, and at very high levels, the human is likely to be the biggest bottleneck, since we aren’t special here.
Furthermore, I think that most interesting problems are in the NP complexity class assuming no deceptive alignment has happened. If that’s true, then goodhart that is non-adversarial is not a severe problem even with extreme capabilities, because while getting a solution might be super hard, it’s likely but not proven that p doesn’t equal np, and if that’s true than you can verify whether the solution actually works once you have it easily, even if coming up with solutions are harder.
Thanks a lot for this comment. These are extremely valid concerns that we’ve been thinking about a lot.
I don’t think this is feasible given our current understanding of epistemology in general and epistemology of alignment research in particular. The problems you listed are potential problems with any methodology, not just AI assisted research. Being able to look at a proposed method and make clear arguments that it’s unlikely to have any undesirable incentives or negative second order effects, etc, is the holy grail of applied epistemology and one of the cores of the alignment problem.
For now, the best we can do is be aware of these concerns, work to improve our understanding of the underlying epistemological problem, design the tools and methods in a way that avoids problems (or at least make them likely to be noticed) according to our current best understanding, and actively address them in the process.
On a high level, it seems wise to me to follow these principles:
Approach this as an epistemology problem
Optimize for augmenting human cognition rather than outsourcing cognitive labor or producing good-looking outputs
Short feedback loops and high bandwidth (both between human<>AI and tool users<>tool designers)
Avoid incentivizing the AI components to goodhart against human evaluation
Avoid producing/releasing infohazards
All of these are hard problems. I could write many pages about each of them, and hopefully will at some point, but for now I’ll only address them briefly in relation to your comment.
1. Approach this as an epistemology problem
We don’t know how to evaluate whether a process is going to be robustly truth-seeking (or {whatever you really want}-seeking). Any measure will be a proxy susceptible to goodhart.
This is a concern for any method, including things like “post your work frequently and get a lot of feedback” or “try to formalize stuff”
Introducing AI into it just makes the problem much more explicit and pressing (because of the removal of the “protected meta level”).
I intend to work closely with the Conjecture epistemology/methodologies team in this project. After all, this is kinda the ultimate challenge for epistemology: as the saying goes, you don’t understand something until you can build it.
We need to better understand things like:
What are the current bottlenecks on human cognition and more specifically alignment research, and can/do these tools actually help remove them?
is thinking about “bottlenecks” the right abstraction? especially if there’s a potential to completely transform the workflow, instead of just unblocking what we currently recognize as bottlenecks
What do processes that generate good ideas/solutions look like in practice?
What do the examples we have access to tell us about the underlying mechanisms of effective processes?
To what extent are productive processes legible? Can we make them more legible, and what are the costs/benefits of doing so? How do we avoid goodharting against legibility when it’s incentivized (AI assisted research is one such situation)?
How can you evaluate if an idea is actually good, and doesn’t just “look” good?
What are the different ways an idea can “look” good and how can each of these be operationalized or fail? (e.g. “feels” meaningful/makes you feel less confused, experts in the field think it’s good, LW karma, can be formalized/mathematically verified, can be/is experimentally verified, other processes independently arrive at same idea, inspires more ideas, leads to useful applications, “big if true”, etc)
How can we avoid applying too much optimization pressure to things that “look” good considering that we ultimately only have access to how things “look” to us (or some externalized measure)?
How do asymmetric capabilities affect all this? As you said, AI will amplify cognition more effectively in some ways than others.
Humans already have asymmetric capabilities as well (though it’s unclear clear what “symmetry” would mean...). How does this affect how currently we do research?
How do we leverage asymmetric capabilities without over-relying on them?
How can we tell whether capabilities are intrinsically asymmetric or are just asymmetrically bottlenecked by how we’re trying to use them?
Dual to the concerns re asymmetrical capabilities: What kind of truth-seeking processes can AI enable which are outside the scope of how humans currently do research due to cognitive limitations?
Being explicitly aware of these considerations is the first step. For instance, with regards to the concern about perception of progress due to “speed”:
Obviously you can write much faster and with superficial fluency with an AI assistant, so we need to adjust our evaluation of output in light of that fact.
2. Optimize for augmenting human cognition rather than outsourcing cognitive labor or producing good-looking outputs
This 2017 article Using Artificial Intelligence to Augment Human Intelligence describes a perspective that I share:
I think the cognitive transformation approach is more promising from an epistemological standpoint because the point is to give the humans an inside view of the process by weaving the cognitive operations enabled by the AI into the user’s thinking, rather than just producing good-seeming artifacts. In other words, we want to amplify the human’s generator, not just rely on human evaluation of an external generation process.
This does not solve the goodhart problem (you might feel like the AI is improving your cognition without actually being productive), but it enables a form of “supervision” that is closer to the substrate of cognition and thus gives the human more intimate insight into whether and why things are working or not.
I also expect the cognitive transformation model to be significantly more effective in the near future. But as AIs become more capable it will be more tempting to increase the length of feedback loops & supervise outcomes instead of process. Hopefully building tools and gaining hands-on experience now will give us more leverage to continue using AI as cognitive augmentation rather than just outsourcing cognition once the latter becomes “easier”.
It occurs to me that I’ve just reiterated the argument for process supervision over outcome supervision:
A major part of the work here will be designing interfaces which surface the “cognitive primitives” as control levers and make high bandwidth interaction & feedback possible.
Slightly more concretely, GPTs are conditional probability distributions one can control by programming boundary conditions (“prompting”), searching through stochastic ramifications (“curation”), and perhaps also manipulating latents (see this awesome blog post Imagining better interfaces to language models). The probabilistic simulator (or speculator) itself and each of these control methods, I think, have close analogues to how we operate our own minds, and thus I think it’s possible with the right interface to “stitch” the model to our minds in a way that acts as a controllable extension of thought. This is a very different approach to “making GPT useful” than, say, InstructGPT, and it’s why I call it cyborgism.
3. Short feedback loops and high bandwidth (both between human<>AI and tool users<>tool designers)
Short feedback loops and high bandwidth between the human and AI is integral the cognitive augmentation perspective: you want as much of the mission-relevant information to be passing through (and understood by) the human user as possible. Not only is this more helpful to the human, it gives them opportunities to notice problems and course-correct at the process level which may not be transparent at all in more oracle or genie-like approaches.
For similar reasons, we want short feedback loops between the users and designers/engineers of the tools (ideally the user designs the tool—needless to say, I will be among the first of the cyborgs I make). We want to be able to inspect the process on a meta level and notice and address problems like goodhart or mode collapse as soon as possible.
4. Avoid incentivizing the AI components to goodhart against human evaluation
This is obvious but hard to avoid, because we do want to improve the system and human evaluation is the main source of feedback we have. But I think there are concrete ways to avoid the worst here, like being very explicit about where and how much optimization pressure is being applied and avoiding methods which extrapolate proxies of human evaluation with unbounded atomic optimization.
There are various reasons I plan to avoid RLHF (except for purposes of comparison); this is one of them. This is not to say other methods that leverage human feedback are immune to goodhart, but RLHF is particularly risky because you’re creating a model(proxy) of human evaluation of outcomes and optimizing against it (the ability to apply unbounded optimization against the reward model is the reason to make one in the first place rather than training against human judgements directly).
I’m more interested in approaches that interactively prototype effective processes & use them as supervised examples to augment the model’s prior: scouting the space of processes rather than optimizing a fixed measure of what a good outcome looks like. Of course, we must still rely on human judgment to say what a good process is (at various levels of granularity, e.g. curation of AI responses and meta-selection of approaches based on perceived effectiveness), so we still need be wary of goodhart. But I think avoiding direct optimization pressure toward outcome evaluations can go a long way. Supervise Process, not Outcomes contains more in depth reasoning on this point.
That said, it’s important to emphasize that this is not a proposal to solve alignment, but the much easier (though still hard) problem of shaping an AI system to augment alignment research before foom. I don’t expect these methods to scale to aligning a superintelligent AI; I expect conceptual breakthroughs will be necessary for that and iterative approaches alone will fail. The motivation for this project is my belief that AI augmentation can put us in a better position to make those conceptual breakthroughs.
5. Avoid producing/releasing infohazards
I won’t say too much about this now, but anything that we identify to present a risk of accelerating capabilities will be covered under Conjecture’s infohazard policy.
I want to talk about why automation is likely more dangerous and more useful than cyborgization, and the reason is Amdahl’s law.
In other words, the slowest process controls the outcome, and at very high levels, the human is likely to be the biggest bottleneck, since we aren’t special here.
Furthermore, I think that most interesting problems are in the NP complexity class assuming no deceptive alignment has happened. If that’s true, then goodhart that is non-adversarial is not a severe problem even with extreme capabilities, because while getting a solution might be super hard, it’s likely but not proven that p doesn’t equal np, and if that’s true than you can verify whether the solution actually works once you have it easily, even if coming up with solutions are harder.