I think there are two really important applications, which have the potential to radically reshape the world:
Research
The ability to develop and test out new ideas, adding to the body of knowledge we have accumulated
Automating this would be a massive deal for the usual reasons about feeding back into growth rates, facilitating something like a singularity
In particular the automation of further AI development is likely to be important
There are many types of possible research, and automation may look quite different for e.g. empirical medical research vs fundamental physics vs political philosophy
The sequence in which we get the ability to automate different types of research could be pretty important for determining what trajectory the world is on
Executive capacity
The ability to look at the world, form views about how it should be different, and form and enact plans to make it different
(People sometimes use “agency” to describe a property in this vicinity)
This is the central thing that leads to new things getting done in the world. If this were fully automated we might have large fully autonomous companies building more and more complex things towards effective purposes.
This is also the thing which, (if/)when automated, creates concerns about AI takeover risk.
I agree. I tentatively think (and have been arguing in private for a while) that these are ‘basically the same thing’. They’re both ultimately about
forming good predictions on the basis of existing models
efficiently choosing ‘experiments’ to navigate around uncertainties
(and thereby improve models!)
using resources (inc. knowledge) to acquire more resources
They differ (just as research disciplines differ from other disciplines, and executing in one domain differs from other domains) in the specifics, especially on what existing models are useful and the ‘research taste’ required to generate experiment ideas and estimate value-of-information. But the high level loop is kinda the same.
Unclear to me what these are bottlenecked by, but I think the latent ‘research taste’ may be basically it (potentially explains why some orgs are far more effective than others, why talented humans take a while to transfer between domains, why mentorship is so valuable, why the scientific revolution took so long to get started...?)
In particular, the ‘big two’ are both characterised by driving beyond the frontier of the well-understood which means by necessity they’re about efficiently deliberately setting up informative/serendipitous scenarios to get novel informative data. When you’re by necessity navigating beyond the well-understood, you have to bottom out your plans with heuristic guesses about VOI, and you have to make plans which (at least sometimes) have good VOI. Those have to ground out somewhere, and that’s the ‘research taste’ at the system-1-ish level.
I think it’s most likely that for a while centaurs will significantly outperform fully automated systems
Agree, and a lot of my justification comes from this feeling that ‘research taste’ is quite latent, somewhat expensive to transfer, and a bottleneck for the big 2.
I agree. I tentatively think (and have been arguing in private for a while) that these are ‘basically the same thing’. They’re both ultimately about
forming good predictions on the basis of existing models
efficiently choosing ‘experiments’ to navigate around uncertainties
(and thereby improve models!)
using resources (inc. knowledge) to acquire more resources
They differ (just as research disciplines differ from other disciplines, and executing in one domain differs from other domains) in the specifics, especially on what existing models are useful and the ‘research taste’ required to generate experiment ideas and estimate value-of-information. But the high level loop is kinda the same.
Unclear to me what these are bottlenecked by, but I think the latent ‘research taste’ may be basically it (potentially explains why some orgs are far more effective than others, why talented humans take a while to transfer between domains, why mentorship is so valuable, why the scientific revolution took so long to get started...?)
In particular, the ‘big two’ are both characterised by driving beyond the frontier of the well-understood which means by necessity they’re about efficiently deliberately setting up informative/serendipitous scenarios to get novel informative data. When you’re by necessity navigating beyond the well-understood, you have to bottom out your plans with heuristic guesses about VOI, and you have to make plans which (at least sometimes) have good VOI. Those have to ground out somewhere, and that’s the ‘research taste’ at the system-1-ish level.
Agree, and a lot of my justification comes from this feeling that ‘research taste’ is quite latent, somewhat expensive to transfer, and a bottleneck for the big 2.