The two parts that stand out most to me are the Causality and Ontology Change sections.
Regarding Causality, I agree that there will be little investment into robotics as a mechanism for intervening on the world and building causal models. However, I don’t see why practicing on videogames doesn’t produce this sort of interventionist data and why AIs wouldn’t learn causal models from that. And it doesn’t seem that expensive to create the data. It’s already happened a bunch with AIs trained on mutltiplayer videogames, and later on it will get cheaper, and overall investment will continue to increase by orders of magnitude. So I expect this will happen.
Regarding Ontology Change, I currently think of this argument as a variant of “The AI needs to understand and model itself well enough to be able understand when its concepts are mistaken. To build this sort of self-aware AGI requires lots of new insights about how agents work that we do not have. Therefore we cannot build one.”
From an x-risk reduction perspective, I try to think of the worst case. Even if I believe that machine learning systems will not find programs that have the sorts of self-reflective properties that an agent needs in order to be able to notice that its concepts are mistaken (which I am honestly uncertain about), I also think that 15 years from now the world would still be massively overrun with Machine Learning, and the smartest minds in the world would be excited about AGI, and the top researchers would work on things related to this question. This is my main counterargument to a lot of arguments that we’re not there yet theoretically — I expect the investment into AI research over the next decade will increase massively with respect to today.
However the last four paragraphs of the post are a direct response to this, and feel notably hopeful to me.
So, the kind of advance we’re worried about must come from the rare maverick dreamer types who have their sights fixed on a distant vision of “true” AGI and are willing to spend years scribbling in the wilderness to get there.
Such an advance is of course not impossible—but it’s a very different threat model from the armies of machine learning researchers and engineers making rapid incremental progress on deep neural nets because they are immediately rewarded with professional success for doing so.
You could probably find all the intellectually generative “AI dreamer” types and talk to them individually—those sorts of people tend to share their ideas in writing.
If the lines of communication remain open—if the current AI debate doesn’t tribalize to the point that “pro-AI” and “anti-AI” factions hate each other and can’t have friendly discussions—then it might be remarkably tractable to just, y’know, persuade a handful of individuals that they should maybe not work too hard to get the world to take notice of their theoretical ideas.
Really great post.
The two parts that stand out most to me are the Causality and Ontology Change sections.
Regarding Causality, I agree that there will be little investment into robotics as a mechanism for intervening on the world and building causal models. However, I don’t see why practicing on videogames doesn’t produce this sort of interventionist data and why AIs wouldn’t learn causal models from that. And it doesn’t seem that expensive to create the data. It’s already happened a bunch with AIs trained on mutltiplayer videogames, and later on it will get cheaper, and overall investment will continue to increase by orders of magnitude. So I expect this will happen.
Regarding Ontology Change, I currently think of this argument as a variant of “The AI needs to understand and model itself well enough to be able understand when its concepts are mistaken. To build this sort of self-aware AGI requires lots of new insights about how agents work that we do not have. Therefore we cannot build one.”
From an x-risk reduction perspective, I try to think of the worst case. Even if I believe that machine learning systems will not find programs that have the sorts of self-reflective properties that an agent needs in order to be able to notice that its concepts are mistaken (which I am honestly uncertain about), I also think that 15 years from now the world would still be massively overrun with Machine Learning, and the smartest minds in the world would be excited about AGI, and the top researchers would work on things related to this question. This is my main counterargument to a lot of arguments that we’re not there yet theoretically — I expect the investment into AI research over the next decade will increase massively with respect to today.
However the last four paragraphs of the post are a direct response to this, and feel notably hopeful to me.