Selection theorems for general intelligence seems like a research agenda that would be useful for developing a theory of robust alignment.
Questions
What kinds of cognitive tasks/problem domains does optimising systems on select for general capabilities?
Which tasks select for superintelligences in the limit of arbitrarily powerful optimisation pressure
Necessary and sufficient conditions for selecting for general intelligence
Taxonomy of generally intelligent systems
Relationship between the optimisation target of the selected for system and the task/problem domain it was optimised for performance on
Motivations
Understanding the type signatures of generally intelligent systems (including those that are vastly exceed humans) could guide the development of robust alignment techniques (i.e. alignment techniques that scale to arbitrarily powerful capabilities.)
Caveats
A major demerit of pursuing this agenda is that concrete results would probably represent significant capabilities insights.
Selection Theorems for General Intelligence
Selection theorems for general intelligence seems like a research agenda that would be useful for developing a theory of robust alignment.
Questions
What kinds of cognitive tasks/problem domains does optimising systems on select for general capabilities?
Which tasks select for superintelligences in the limit of arbitrarily powerful optimisation pressure
Necessary and sufficient conditions for selecting for general intelligence
Taxonomy of generally intelligent systems
Relationship between the optimisation target of the selected for system and the task/problem domain it was optimised for performance on
Motivations
Understanding the type signatures of generally intelligent systems (including those that are vastly exceed humans) could guide the development of robust alignment techniques (i.e. alignment techniques that scale to arbitrarily powerful capabilities.)
Caveats
A major demerit of pursuing this agenda is that concrete results would probably represent significant capabilities insights.