Yes it’s true that much of MIRI’s research is about finding a solution to the design problem for intelligent systems that does not rest on a blind search for policies that satisfy some evaluation procedure. But it seems strange to describe this approach as “hope you can find some other way to produce powerful AI”, as though we know of no other approach to engineering sophisticated systems other than search.
I agree that the success of design in other domains is a great sign and reason for hope. But for now such approaches are being badly outperformed by search (in AI).
Maybe it’s unfair to say “find some other way to produce powerful AI” because we already know the way: just design it yourself. But I think “design” is basically just another word for “find some way to do it,” and we don’t yet have any history of competitive designs to imitate or extrapolate from.
Personally, the main reason I’m optimistic about design in the future is that the designers may themselves be AI systems. That may help close the current gap between design and search, since both could then benefit from large amounts of computing power. (And it’s plausible that we are currently bottlenecked on a meta-design problem of figuring out how to build automated designers.) That said, it’s completely unclear whether that will actually beat search.
I consider my job as preparing for the worst w.r.t. search, since that currently seems like a better place to invest resources (and I think it’s reasonably likely that dangerous search will be involved even if our AI ecosystem mostly revolves around design). I do think that I’d fall back to pushing on design if this ended up looking hopeless enough. If that happens, I’m hoping that by that time we’ll have some much harder evidence that search is a lost cause, so that we can get other people to also jump ship from search to design.
I agree re finding hard evidence that search is a lost cause, and I see how your overall work in the field has the property of (hopefully) either finding a safe way to use search, or producing evidence (perhaps weak or perhaps strong) that search is a lost cause.
As I speak to young (and senior!) ML folk, I notice they often struggle to conceive of what a non-search approach to AI really means. I’m excited about elucidating what search and design really are, and getting more people to consider using aspects of design alongside search.
But for now such approaches are being badly outperformed by search (in AI).
I suspect the edge here depends on the level of abstraction. That is, Go bots that use search can badly outperform Go bots that don’t use any search, but using search at the ‘high level’ (like in MuZero) only somewhat outperforms using design at that level (like in AlphaZero).
It wouldn’t surprise me if search always has an edge (at basically any level, exposing things to adjustment by gradient descent makes performance on key metrics better), but if the edge is small it seems plausible to focus on design.
I can see why you’d say AlphaZero has more of a “design” element than MuZero, because of the MCTS. But if on some absolute scale you say that AlphaZero is a design / search hybrid, then presumably you should also say the OpenAI Five is a design / search hybrid, since it uses PPO at the outer layer, which is a designed algorithm. This seems wrong. (Also, it seems like many current proposals for building AGI out of ML would be classified as design / search hybrids.)
Maybe the distinction is that AlphaZero uses MCTS at test time? Would AlphaZero without MCTS at test time be only search? (Aside: it’s not great that when we remove Monte Carlo Tree Search we’re now saying that the design is gone and only search remains)
More generally, I don’t see how AlphaZero is making any headway on this problem:
It seems to me that Christiano’s write-up is a fairly general and compelling knock-down of the black-box approach to design in which we build an evaluation procedure and then rely on search to find a policy that our evaluation procedure ranks highly.
But if on some absolute scale you say that AlphaZero is a design / search hybrid, then presumably you should also say the OpenAI Five is a design / search hybrid, since it uses PPO at the outer layer, which is a designed algorithm. This seems wrong.
I think I’m willing to bite that bullet; like, as far as we know the only stuff that’s “search all the way up” is biological evolution.
But ‘hybrid’ seems a little strange; like, I think design normally has search as a subcomponent (in imaginary space, at least, and I think often also search through reality), and so in some sense any design that isn’t a fully formed vision from God is a design/search hybrid. (If my networks use RELU activations ‘by design’, isn’t that really by the search process of the ML community as a whole? And yet it’s still useful to distinguish networks which determine what nonlinearity to use from local data, which which networks have it determined for them by an external process, which potentially has a story for why that’s the right thing to do.)
Total horse takeover seems relevant as another way to think about intervening to ‘control’ things at varying levels of abstraction.
[The core thing about design that seems important and relevant here is that there’s a “story for why the design will work”, whereas search is more of an observational fact of what was out there when you looked. It seems like it might be easier to build a ‘safe design’ out of smaller sub-designs, whereas trying to search for a safe algorithm using search runs into all the anthropic problems of empiricism.]
I thought this was a great summary, thanks!
I agree that the success of design in other domains is a great sign and reason for hope. But for now such approaches are being badly outperformed by search (in AI).
Maybe it’s unfair to say “find some other way to produce powerful AI” because we already know the way: just design it yourself. But I think “design” is basically just another word for “find some way to do it,” and we don’t yet have any history of competitive designs to imitate or extrapolate from.
Personally, the main reason I’m optimistic about design in the future is that the designers may themselves be AI systems. That may help close the current gap between design and search, since both could then benefit from large amounts of computing power. (And it’s plausible that we are currently bottlenecked on a meta-design problem of figuring out how to build automated designers.) That said, it’s completely unclear whether that will actually beat search.
I consider my job as preparing for the worst w.r.t. search, since that currently seems like a better place to invest resources (and I think it’s reasonably likely that dangerous search will be involved even if our AI ecosystem mostly revolves around design). I do think that I’d fall back to pushing on design if this ended up looking hopeless enough. If that happens, I’m hoping that by that time we’ll have some much harder evidence that search is a lost cause, so that we can get other people to also jump ship from search to design.
Thanks for the note Paul.
I agree re finding hard evidence that search is a lost cause, and I see how your overall work in the field has the property of (hopefully) either finding a safe way to use search, or producing evidence (perhaps weak or perhaps strong) that search is a lost cause.
As I speak to young (and senior!) ML folk, I notice they often struggle to conceive of what a non-search approach to AI really means. I’m excited about elucidating what search and design really are, and getting more people to consider using aspects of design alongside search.
I suspect the edge here depends on the level of abstraction. That is, Go bots that use search can badly outperform Go bots that don’t use any search, but using search at the ‘high level’ (like in MuZero) only somewhat outperforms using design at that level (like in AlphaZero).
It wouldn’t surprise me if search always has an edge (at basically any level, exposing things to adjustment by gradient descent makes performance on key metrics better), but if the edge is small it seems plausible to focus on design.
Thanks for this way of thinking about AlphaZero as a hybrid design/search system—I found this helpful.
I’m confused by this distinction.
I can see why you’d say AlphaZero has more of a “design” element than MuZero, because of the MCTS. But if on some absolute scale you say that AlphaZero is a design / search hybrid, then presumably you should also say the OpenAI Five is a design / search hybrid, since it uses PPO at the outer layer, which is a designed algorithm. This seems wrong. (Also, it seems like many current proposals for building AGI out of ML would be classified as design / search hybrids.)
Maybe the distinction is that AlphaZero uses MCTS at test time? Would AlphaZero without MCTS at test time be only search? (Aside: it’s not great that when we remove Monte Carlo Tree Search we’re now saying that the design is gone and only search remains)
More generally, I don’t see how AlphaZero is making any headway on this problem:
I think I’m willing to bite that bullet; like, as far as we know the only stuff that’s “search all the way up” is biological evolution.
But ‘hybrid’ seems a little strange; like, I think design normally has search as a subcomponent (in imaginary space, at least, and I think often also search through reality), and so in some sense any design that isn’t a fully formed vision from God is a design/search hybrid. (If my networks use RELU activations ‘by design’, isn’t that really by the search process of the ML community as a whole? And yet it’s still useful to distinguish networks which determine what nonlinearity to use from local data, which which networks have it determined for them by an external process, which potentially has a story for why that’s the right thing to do.)
Total horse takeover seems relevant as another way to think about intervening to ‘control’ things at varying levels of abstraction.
[The core thing about design that seems important and relevant here is that there’s a “story for why the design will work”, whereas search is more of an observational fact of what was out there when you looked. It seems like it might be easier to build a ‘safe design’ out of smaller sub-designs, whereas trying to search for a safe algorithm using search runs into all the anthropic problems of empiricism.]