The most useful / potentially-behavior-changing part of the post for me is the section describing how certain groups shouldn’t develop detailed models of AI risk (pasted below). But the arguments are light on details. I’d like to see a second post building a more detailed model of why you think these outcomes are net negative.
The specific outcomes we want to avoid are:
The higher echelons of some government or military develop an accurate model of AI Risk.
They’d want to enforce their government’s superiority, or national superiority, or ideological superiority, and they’d trample over the rest of humanity.
There are no eudaimonia-interested governments on Earth.
The accurate model of AI Risk makes its way into the public consciousness.
The “general public”, as I’ve outlined, is not safe either. And in particular, what we don’t want is some “transparency policy” where the AGI-deploying group is catering to the public’s whims regarding the AGI’s preferences.
Just look at modern laws, and the preferences they imply! Humanity-in-aggregate is not eudaimonia-aligned either.
A large subset of wealthy or influential people not pre-selected by their interest in EA/LW ideas form an accurate model of AI Risk.
We’d either get some revenue-maximizer for a given corporation, or a dystopian dictatorship, or some such outcome.
And even if the particular influential person is conventionally nice, we get all the problems with sampling a random nice individual from the general population (the off-distribution problem).
The most useful / potentially-behavior-changing part of the post for me is the section describing how certain groups shouldn’t develop detailed models of AI risk (pasted below). But the arguments are light on details. I’d like to see a second post building a more detailed model of why you think these outcomes are net negative.