“p(DOOM | Action)” seems too coarse a view from which to draw any sensible conclusions. The utility of pessimism (or realism, from some perspectives) is that it informs decisions about which action to take. It’s not ok that we’re simply doing something. Yes, the pessimistic view has a cost, but that needs to be weighed against the (potential) upside of finding more promising approaches—and realising that p(DOOM | we did things that felt helpful) > p(DOOM | we did things that may work).
Similarly, convincing a researcher that a [doomed-to-the-point-of-uselessness] approach is doomed is likely to be net positive, since they’ll devote time to more worthwhile pursuits, likely increasing p(AWESOME...). Dangers being:
Over-confidence that it’s doomed.
Throwing out wrong-but-useful ideas prematurely.
A few researchers may be discouraged to the point of quitting.
In general, the best action to take will depend on p(Doom); if researchers have very different p(Doom) estimates, it should come as no surprise that they’ll disagree on the best course of action at many levels.
Specifically, the upside/downside of ‘pessimism’ will relate to our initial p(Doom). If p(Doom) starts out at >99.9%, then p(Doom | some vaguely plausible kind of action) is likely also >99%. High values of p(Doom) result from the problem’s being fundamentally difficult—with answers that may be hidden in a tiny region of a high-dimensional space. If the problem is five times as hard as we may reasonably hope, then a shotgun may work. If the problem is five orders of magnitude harder, then we want a laser.
Here I emphasize that we’d eventually need the laser. By all means use a shotgun to locate promising approaches, but if the problem is hard we can’t just babble: we need to prune hard.
It occurs to me that it may be useful to have more community-level processes/norms to signal babbling, so that mistargeted pruning doesn’t impede idea generation.
Pivotal act talk does seem to be disastrous PR, but I think that’s the only sense in which it’s negative. By Critch’s standards, the arguments in that post are weak. When thinking about this, it’s important to be clear what we’re saying on the facts, and what adjustments we’re making for PR considerations. React too much to the negative PR issue (which I agree is real), and there’s the danger of engaging in wishful thinking.
Should we try to do tackle the extremely hard coordination problems? Possibly—but we shouldn’t be banking on such approaches working. (counterfactual resource use deserves consideration)
We should be careful of any PR-motivated adaptations to discourse, since they’re likely to shape thinking in unhelpful ways. (even just talking less about X, rather than actively making misleading claims about X may well have a negative impact on epistemics)
I agree with pretty much all of this, thanks for the clarification. I think the problem is that we need both a laser and a shotgun. We need a laser shotgun. It’s analogous to missile defense, where each individual reentry vehicle is very difficult to shoot down, and we have to shoot down every one or else we’ve failed. I appreciate now why it’s such a hard problem.
I definitely think there needs to be a better introductory material for alignment, particularly regarding what approaches have been investigated and what names they go by. I know the community already knows there is a problem with public communication so I’m not saying anything novel here.
I’d like to see a kind of “choose your own adventure” format webpage where lay people can select by multiple choice how they would go about addressing the problem, and the webpage will explain in not-too-technical language how much progress has been made and where to go for more information. I’m thinking about putting something like this together, I’ll use a wiki format since I don’t have all the answers of course.
Agreed on the introductory material (and various other things in that general direction).
I’m not clear on the case for public communication. We don’t want lay people to have strong and wildly inaccurate opinions, but it seems unachievable to broadly remedy the “wildly inaccurate” part. I haven’t seen a case made that broad public engagement can help (but I haven’t looked hard—has such a case been made?).
I don’t see what the lay public is supposed to do with the information—even supposing they had a broadly accurate high-level picture of the situation. My parents fit this description, and thus far it’s not clear to me what I’d want them to do. It’s easy to imagine negative outcomes (e.g. via politicization), and hard to imagine realistic positive ones (e.g. if 80% of people worldwide had an accurate picture, perhaps that would help, but it’s not going to happen).
It does seem important to communicate effectively to a somewhat narrower audience. The smartest 1% of people being well informed seems likely to be positive, but even here there are pitfalls in aiming to achieve this—e.g. if you get many people as far as [AGI will soon be hugely powerful] but they don’t understand the complexities of the [...and hugely dangerous] part, then you can inadvertently channel more resources into a race. (still net positive though, I’d assume EDIT: the being-well-informed part, that is)
I’ll grant that there’s an argument along the lines of: Perhaps we don’t know what we’d want the public to do, but there may come a time when we do know. If time is short it may then be too late to do the public education necessary. Therefore we should start now.
I don’t think I buy this. The potential downsides of politicization seem great, and it’s hard to think of many plausible ”...and then public pressure to do X saved the day!” scenarios.
This is very much not my area, so perhaps there are good arguments I haven’t seen. I’d just warn against reasoning of the form “Obviously we want to get the public engaged and well-informed, so...”. The “engaged” part is far from obvious (in particular we won’t be able to achieve [engaged if and only if well-informed]).
I’ll have to think about this more, but if you compare this to other things like climate change or gain of function research, it’s a very strange movement indeed that doesn’t want to spread awareness. Note that I agree that regulation is probably more likely to be harmful than helpful, and also note that I personally despise political activists, so I’m not someone normally amenable to “consciousness raising.” It’s just unusual to be against it for something you care about.
Absolutely—but it’s a strange situation in many respects.
It may be that spreading awareness is positive, but I don’t think standard arguments translate directly. There’s also irreversibility to consider: err on the side of not spreading info, and you can spread it later (so long as there’s time); you can’t easily unspread it.
More generally, I think for most movements we should ask ourselves, “How much worse than the status-quo can things plausibly get?”.
For gain-of-function research, we’d need to consider outcomes where the debate gets huge focus, but the sensible side loses (e.g. through the public seeing gof as the only way to prevent future pandemics). This seems unlikely, since I think there are good common-sense arguments against gof at most levels of detail.
For climate change, it’s less clear to me: there seem many plausible ways for things to have gotten worse. Essentially because the only clear conclusion is “something must be done”, but there’s quite a bit less clarity about what—or at least there should be less clarity. (e.g. to the extent that direct climate-positive actions have negative economic consequences, to what extent are there downstream negative-climate impacts; I have no idea, but I’m sure it’s a complex situation)
For AGI, I find it easy to imagine making-things-worse and hard to see plausible routes to making-things-better.
Even the expand-the-field upside needs to be approached with caution. This might be better thought of as something like [expand the field while maintaining/improving the average level of understanding]. Currently, most people who bump into AI safety/alignment will quickly find sources discussing the most important problems. If we expanded the field 100x overnight, then it becomes plausible that most new people don’t focus on the real problems. (e.g. it’s easy enough only to notice the outer-alignment side of things)
Unless time is very short, I’d expect doubling the field each year works out better than 5x each year—because all else would not be equal. (I have no good sense what the best expansion rate or mechanism is—just that it’s not [expand as fast as possible])
But perhaps I’m conflating [aware of the problem] with [actively working on the problem] a bit much. Might not be a bad idea to have large amounts of smart people aware of the problem overnight.
A few thoughts:
“p(DOOM | Action)” seems too coarse a view from which to draw any sensible conclusions. The utility of pessimism (or realism, from some perspectives) is that it informs decisions about which action to take. It’s not ok that we’re simply doing something.
Yes, the pessimistic view has a cost, but that needs to be weighed against the (potential) upside of finding more promising approaches—and realising that p(DOOM | we did things that felt helpful) > p(DOOM | we did things that may work).
Similarly, convincing a researcher that a [doomed-to-the-point-of-uselessness] approach is doomed is likely to be net positive, since they’ll devote time to more worthwhile pursuits, likely increasing p(AWESOME...). Dangers being:
Over-confidence that it’s doomed.
Throwing out wrong-but-useful ideas prematurely.
A few researchers may be discouraged to the point of quitting.
In general, the best action to take will depend on p(Doom); if researchers have very different p(Doom) estimates, it should come as no surprise that they’ll disagree on the best course of action at many levels.
Specifically, the upside/downside of ‘pessimism’ will relate to our initial p(Doom). If p(Doom) starts out at >99.9%, then p(Doom | some vaguely plausible kind of action) is likely also >99%. High values of p(Doom) result from the problem’s being fundamentally difficult—with answers that may be hidden in a tiny region of a high-dimensional space.
If the problem is five times as hard as we may reasonably hope, then a shotgun may work.
If the problem is five orders of magnitude harder, then we want a laser.
Here I emphasize that we’d eventually need the laser. By all means use a shotgun to locate promising approaches, but if the problem is hard we can’t just babble: we need to prune hard.
It occurs to me that it may be useful to have more community-level processes/norms to signal babbling, so that mistargeted pruning doesn’t impede idea generation.
Pivotal act talk does seem to be disastrous PR, but I think that’s the only sense in which it’s negative. By Critch’s standards, the arguments in that post are weak. When thinking about this, it’s important to be clear what we’re saying on the facts, and what adjustments we’re making for PR considerations. React too much to the negative PR issue (which I agree is real), and there’s the danger of engaging in wishful thinking.
Should we try to do tackle the extremely hard coordination problems? Possibly—but we shouldn’t be banking on such approaches working. (counterfactual resource use deserves consideration)
We should be careful of any PR-motivated adaptations to discourse, since they’re likely to shape thinking in unhelpful ways. (even just talking less about X, rather than actively making misleading claims about X may well have a negative impact on epistemics)
I agree with pretty much all of this, thanks for the clarification. I think the problem is that we need both a laser and a shotgun. We need a laser shotgun. It’s analogous to missile defense, where each individual reentry vehicle is very difficult to shoot down, and we have to shoot down every one or else we’ve failed. I appreciate now why it’s such a hard problem.
I definitely think there needs to be a better introductory material for alignment, particularly regarding what approaches have been investigated and what names they go by. I know the community already knows there is a problem with public communication so I’m not saying anything novel here.
I’d like to see a kind of “choose your own adventure” format webpage where lay people can select by multiple choice how they would go about addressing the problem, and the webpage will explain in not-too-technical language how much progress has been made and where to go for more information. I’m thinking about putting something like this together, I’ll use a wiki format since I don’t have all the answers of course.
Agreed on the introductory material (and various other things in that general direction).
I’m not clear on the case for public communication. We don’t want lay people to have strong and wildly inaccurate opinions, but it seems unachievable to broadly remedy the “wildly inaccurate” part. I haven’t seen a case made that broad public engagement can help (but I haven’t looked hard—has such a case been made?).
I don’t see what the lay public is supposed to do with the information—even supposing they had a broadly accurate high-level picture of the situation. My parents fit this description, and thus far it’s not clear to me what I’d want them to do. It’s easy to imagine negative outcomes (e.g. via politicization), and hard to imagine realistic positive ones (e.g. if 80% of people worldwide had an accurate picture, perhaps that would help, but it’s not going to happen).
It does seem important to communicate effectively to a somewhat narrower audience. The smartest 1% of people being well informed seems likely to be positive, but even here there are pitfalls in aiming to achieve this—e.g. if you get many people as far as [AGI will soon be hugely powerful] but they don’t understand the complexities of the [...and hugely dangerous] part, then you can inadvertently channel more resources into a race. (still net positive though, I’d assume EDIT: the being-well-informed part, that is)
I’ll grant that there’s an argument along the lines of:
Perhaps we don’t know what we’d want the public to do, but there may come a time when we do know. If time is short it may then be too late to do the public education necessary. Therefore we should start now.
I don’t think I buy this. The potential downsides of politicization seem great, and it’s hard to think of many plausible ”...and then public pressure to do X saved the day!” scenarios.
This is very much not my area, so perhaps there are good arguments I haven’t seen.
I’d just warn against reasoning of the form “Obviously we want to get the public engaged and well-informed, so...”. The “engaged” part is far from obvious (in particular we won’t be able to achieve [engaged if and only if well-informed]).
I’ll have to think about this more, but if you compare this to other things like climate change or gain of function research, it’s a very strange movement indeed that doesn’t want to spread awareness. Note that I agree that regulation is probably more likely to be harmful than helpful, and also note that I personally despise political activists, so I’m not someone normally amenable to “consciousness raising.” It’s just unusual to be against it for something you care about.
Absolutely—but it’s a strange situation in many respects.
It may be that spreading awareness is positive, but I don’t think standard arguments translate directly. There’s also irreversibility to consider: err on the side of not spreading info, and you can spread it later (so long as there’s time); you can’t easily unspread it.
More generally, I think for most movements we should ask ourselves, “How much worse than the status-quo can things plausibly get?”.
For gain-of-function research, we’d need to consider outcomes where the debate gets huge focus, but the sensible side loses (e.g. through the public seeing gof as the only way to prevent future pandemics). This seems unlikely, since I think there are good common-sense arguments against gof at most levels of detail.
For climate change, it’s less clear to me: there seem many plausible ways for things to have gotten worse. Essentially because the only clear conclusion is “something must be done”, but there’s quite a bit less clarity about what—or at least there should be less clarity. (e.g. to the extent that direct climate-positive actions have negative economic consequences, to what extent are there downstream negative-climate impacts; I have no idea, but I’m sure it’s a complex situation)
For AGI, I find it easy to imagine making-things-worse and hard to see plausible routes to making-things-better.
Even the expand-the-field upside needs to be approached with caution. This might be better thought of as something like [expand the field while maintaining/improving the average level of understanding]. Currently, most people who bump into AI safety/alignment will quickly find sources discussing the most important problems. If we expanded the field 100x overnight, then it becomes plausible that most new people don’t focus on the real problems. (e.g. it’s easy enough only to notice the outer-alignment side of things)
Unless time is very short, I’d expect doubling the field each year works out better than 5x each year—because all else would not be equal. (I have no good sense what the best expansion rate or mechanism is—just that it’s not [expand as fast as possible])
But perhaps I’m conflating [aware of the problem] with [actively working on the problem] a bit much. Might not be a bad idea to have large amounts of smart people aware of the problem overnight.