I’ve decided that I want to to make more of a point to write down my macro-strategic thoughts, because writing things down often produces new insights and refinements, and so that other folks can engage with them.
This is one frame or lens that I tend to think with a lot. This might be more of a lens or a model-let than a full break-down.
There are two broad classes of problems that we need to solve: we have some pre-paradigmatic science to figure out, and we have have the problem of civilizational sanity.
Preparadigmatic science
There are a number of hard scientific or scientific-philosophical problems that we’re facing down as a species.
Most notably, the problem of AI alignment, but also finding technical solutions to various risks caused by bio-techinlogy, possibly getting our bearings with regards to what civilization collapse means and how it is likely to come about, possibly getting a handle on the risk of a simulation shut-down, possibly making sense of the large scale cultural, political, cognitive shifts that are likely to follow from new technologies that disrupt existing social systems (like VR?).
Basically, for every x-risk, and every big shift to human civilization, there is work to be done even making sense of the situation, and framing the problem.
As this work progresses it eventually transitions into incremental science / engineering, as the problems are clarified and specified, and the good methodologies for attacking those problems solidify.
(Work on bio-risk, might already be in this phase. And I think that work towards human genetic enhancement is basically incremental science.)
To my rough intuitions, it seems like these problems, in order of pressingness are:
AI alignment
Bio-risk
Human genetic enhancement
Social, political, civilizational collapse
…where that ranking is mostly determined by which one will have a very large impact on the world first.
So there’s the object-level work of just trying to make progress on these puzzles, plus a bunch of support work for doing that object level work.
The support work includes
Operations that makes the research machines run (ex: MIRI ops)
Recruitment (and acclimation) of people who can do this kind of work (ex: CFAR)
Creating and maintaining infrastructure that enables intellectually fruitful conversations (ex: LessWrong)
Developing methodology for making progress on the problems (ex: CFAR, a little, but in practice I think that this basically has to be done by the people trying to do the object level work.)
Other stuff.
So we have a whole ecosystem of folks who are supporting this preparadgimatic development.
Civilizational Sanity
I think that in most worlds, if we completely succeeded at the pre-paradigmatic science, and the incremental science and engineering that follows it, the world still wouldn’t be saved.
Broadly, one way or the other, there are huge technological and social changes heading our way, and human decision makers are going to decide how to respond to those changes, possibly in ways that will have very long term repercussions on the trajectory of earth-originating life.
As a central example, if we more-or-less-completely solved AI alignment, from a full theory of agent-foundations, all the way down to the specific implementation, we would still find ourselves in a world, where humanity has attained god-like power over the universe, which we could very well abuse, and end up with a much much worse future than we might otherwise have had. And by default, I don’t expect humanity to refrain from using new capabilities rashly and unwisely.
Completely solving alignment does give us a big leg up on this problem, because we’ll have the aid of superintelligent assistants in our decision making, or we might just have an AI system implement our CEV in classic fashion.
I would say that “aligned superintelligent assistants” and “AIs implementing CEV”, are civilizational sanity interventions: technologies or institutions that help humanity’s high level decision-makers to make wise decisions in response to huge changes that, by default, they will not comprehend.
I gave some examples of possible Civ Sanity interventions here.
Also, think that some forms of governance / policy work that OpenPhil, OpenAI, and FHI have done, count as part of this category, though I want to cleanly distinguish between pushing for object-level policy proposals that you’ve already figured out, and instantiating systems that make it more likely that good policies will be reached and acted upon in general.
Overall, this class of interventions seems neglected by our community, compared to doing and supporting preparadigmatic research. That might be justified. There’s reason to think that we are well equipped to make progress on hard important research problems, but changing the way the world works, seems like it might be harder on some absolute scale, or less suited to our abilities.
I’ve decided that I want to to make more of a point to write down my macro-strategic thoughts, because writing things down often produces new insights and refinements, and so that other folks can engage with them.
This is one frame or lens that I tend to think with a lot. This might be more of a lens or a model-let than a full break-down.
There are two broad classes of problems that we need to solve: we have some pre-paradigmatic science to figure out, and we have have the problem of civilizational sanity.
Preparadigmatic science
There are a number of hard scientific or scientific-philosophical problems that we’re facing down as a species.
Most notably, the problem of AI alignment, but also finding technical solutions to various risks caused by bio-techinlogy, possibly getting our bearings with regards to what civilization collapse means and how it is likely to come about, possibly getting a handle on the risk of a simulation shut-down, possibly making sense of the large scale cultural, political, cognitive shifts that are likely to follow from new technologies that disrupt existing social systems (like VR?).
Basically, for every x-risk, and every big shift to human civilization, there is work to be done even making sense of the situation, and framing the problem.
As this work progresses it eventually transitions into incremental science / engineering, as the problems are clarified and specified, and the good methodologies for attacking those problems solidify.
(Work on bio-risk, might already be in this phase. And I think that work towards human genetic enhancement is basically incremental science.)
To my rough intuitions, it seems like these problems, in order of pressingness are:
AI alignment
Bio-risk
Human genetic enhancement
Social, political, civilizational collapse
…where that ranking is mostly determined by which one will have a very large impact on the world first.
So there’s the object-level work of just trying to make progress on these puzzles, plus a bunch of support work for doing that object level work.
The support work includes
Operations that makes the research machines run (ex: MIRI ops)
Recruitment (and acclimation) of people who can do this kind of work (ex: CFAR)
Creating and maintaining infrastructure that enables intellectually fruitful conversations (ex: LessWrong)
Developing methodology for making progress on the problems (ex: CFAR, a little, but in practice I think that this basically has to be done by the people trying to do the object level work.)
Other stuff.
So we have a whole ecosystem of folks who are supporting this preparadgimatic development.
Civilizational Sanity
I think that in most worlds, if we completely succeeded at the pre-paradigmatic science, and the incremental science and engineering that follows it, the world still wouldn’t be saved.
Broadly, one way or the other, there are huge technological and social changes heading our way, and human decision makers are going to decide how to respond to those changes, possibly in ways that will have very long term repercussions on the trajectory of earth-originating life.
As a central example, if we more-or-less-completely solved AI alignment, from a full theory of agent-foundations, all the way down to the specific implementation, we would still find ourselves in a world, where humanity has attained god-like power over the universe, which we could very well abuse, and end up with a much much worse future than we might otherwise have had. And by default, I don’t expect humanity to refrain from using new capabilities rashly and unwisely.
Completely solving alignment does give us a big leg up on this problem, because we’ll have the aid of superintelligent assistants in our decision making, or we might just have an AI system implement our CEV in classic fashion.
I would say that “aligned superintelligent assistants” and “AIs implementing CEV”, are civilizational sanity interventions: technologies or institutions that help humanity’s high level decision-makers to make wise decisions in response to huge changes that, by default, they will not comprehend.
I gave some examples of possible Civ Sanity interventions here.
Also, think that some forms of governance / policy work that OpenPhil, OpenAI, and FHI have done, count as part of this category, though I want to cleanly distinguish between pushing for object-level policy proposals that you’ve already figured out, and instantiating systems that make it more likely that good policies will be reached and acted upon in general.
Overall, this class of interventions seems neglected by our community, compared to doing and supporting preparadigmatic research. That might be justified. There’s reason to think that we are well equipped to make progress on hard important research problems, but changing the way the world works, seems like it might be harder on some absolute scale, or less suited to our abilities.