Some “meta-cruxes” for AI x-risk debates
[Epistemic status: As I say below, I’ve been thinking about this topic for several years and I’ve worked on it as part of my PhD research. But none of this is based on any rigorous methodology, just my own impressions from reading the literature.]
I’ve been thinking about possible cruxes in AI x-risk debates for several years now. I was even doing that as part of my PhD research, although my PhD is currently on pause because my grant ran out. In particular, I often wonder about “meta-cruxes”—i.e., cruxes related to debates or uncertainties that are more about different epistemological or decision-making approaches rather than about more object-level arguments.
The following are some of my current top candidates for “meta-cruxes” related to AI x-risk debates. There are some others I might list, but I think these are probably the biggest ones. (Of course, these cruxes influence lots of debates, not just AI x-risk debates. But I’ve mostly focused on AI x-risk debates for my PhD research so I’ll focus on that here as well.)
Hypothetical vs. empirical arguments
In many AI x-risk debates, it often feels like those who are more worried about the risks are essentially saying, “here’s this chain of logic and analysis that leads to an all-things-considered conclusion that AI x-risk is something we should be very concerned about.” And then those who aren’t so worried often respond with something like, “well, if you can give me empirical evidence, or perhaps proven theorems, that clearly demonstrate the problem then I’ll pay attention, but until then it’s pie in the sky philosophizing and speculation.” The former approach seems particularly common among philosophy or rationalist types, while the latter approach seems most common among engineers and practicing scientist types—although there are of course lots of exceptions on both sides.
This also feels closely related to Bayesianism vs. Frequentist or Popperian approaches in philosophy of science.
Object-level arguments vs. reference class forecasting
Even among those who take more abstract, less-completely-demonstrated arguments seriously, many seem to give much more weight to broad economic models, reference class forecasting using broadly-construed reference classes, or the like, over more gears-level arguments, narrow trend extrapolations that conflict with broader trends, or similar.
(Note that I’m calling this object-level vs. reference class forecasting for lack of a better term at the moment. I also don’t know of a good way to more precisely define each group, although I’m fairly confident that the clusters of thought I’m trying to gesture at are real and distinct enough to be significant. See also “reference class tennis” and Taboo Outside View.)
When deciding what to believe and/or what to do, should we defer to our epistemic superiors, and if so how much? Should we take into account the views of our epistemic peers, and if so to what extent? Supporters of the epistemic modesty POV lean much more heavily towards taking into account the views of epistemic superiors and/or peers, while others say we should form our own personal opinions and act on those. There’s a lot of nuance here, and sometimes it turns out that when people seem to be disagreeing about epistemic modesty they’re actually just using different notions of what that’s supposed to mean. I’m not even totally convinced that there is any substantive debate here at all. But I think it’s more likely that there is a substantive debate and that there are at least sizeable clusters of people who lean towards opposite ends of the spectrum on this.
Typically in my experience those who lean more towards epistemic modesty will have much less confident views and will have more of a wide distribution over possible outcomes and forecasts, while those who lean against epistemic modesty seem more likely (though definitely not guaranteed!) to have much more confident opinions. For some people those confident opinions lead them to take AI risks extremely seriously, while for other people their confident opinions lead them to dismiss the risks.
Note that this crux is not as symmetrical as most of the others on this list. On the one hand, if everybody came to agree with epistemic modesty then I would expect them to converge towards much more similar distributions over predicted outcomes. (There would still be differences though based on how to weight different “expert” views against each other, who to consider an expert in the first place, etc.) If everybody came to agree that we should not go with epistemic modesty, however, then I’d expect people’s views to diverge more, on average, as more people moved towards the extremes.
Risk tolerance
I suspect that many (but certainly not all) of the people advocating for taking AI risks more seriously are more towards the risk averse end of the spectrum, and that many (but not all) of the people advocating for full steam ahead, don’t pay attention to the risks, are more on the risk seeking end of the spectrum.
As I said, there are quite a few other “meta-cruxes” similar to these, but I think these are likely the most important.
I would add
Conflict theory vs. comparative advantage
Is it possible for the wrong kind of technological development to make things worse, or does anything that increases aggregate productivity always make everyone better off in the long run?
Cosmopolitanism vs. human protectionism
Is it acceptable, or good, to let humans go extinct if they will be replaced by an entity that’s more sophisticated or advanced in some way, or should humans defend humanity simply because we’re human?
I agree that the first can be framed as a meta-crux, but actually I think the way you framed it is more of an object-level forecasting question, or perhaps a strong prior on the forecasted effects of technological progress. If on the other hand you framed it more as conflict theory vs. mistake theory, then I’d say that’s more on the meta level.
For the second, I agree that’s for some people, but I’m skeptical of how prevalent the cosmopolitan view is, which is why I didn’t include it in the post.