But I will say, when I’ve talked with alignment researchers at MIRI, something they want more than people working on agent foundations, or Paul’s agenda, are people who grok a bunch of the models and still have disagreements, and work on ideas from a new perspective. I hope your strategy helps discover people who deeply understand and have a novel approach to the alignment problem.
In the later stages of AGI design, hoping that these people come a long is kind of like hoping that explosives experts come a long from the outside and critique the atom bomb design done by nuclear physicists.
They won’t have good models of your AGI design because you have adopted the onion strategy of agi discussion and they also have to deal with disinformation about what even AGI is or can do.
To be successful in that kind of scenario you need to be sufficiently self-sceptical to know where your models are weak and pull in expertise you do not have. Nothing can be “and then a miracle occurs”. Everything that is not a gears level understanding of the situation should be a like a niggling uncomfort that you must deal with.
I’ve not seen any significant effort adopt this mind set.
I’ve not seen any significant effort adopt this mind set.
Could I ask where you’ve looked? MIRI seems to be trying pretty explicitly to develop this mind set, while Paul Christiano has had extensive back and forth on assumptions for his general approach on his medium blog.
As far as I can tell MIRI are asking other people to develop it for thinking about how to build AGI. They have started hiring for developers rather than mathematicians, but AFAIK they haven’t been hiring for the solid state physicists you might want to avoid things like row hammer being a problem.
How to use AGI, they don’t seem to have been hiring for psychologists/economists/political scientists/experts on the process of technological development that might help inform/fill in the gaps in their policy/strategy expertise. Unless they have lots of unknown resources.
In the later stages of AGI design, hoping that these people come a long is kind of like hoping that explosives experts come a long from the outside and critique the atom bomb design done by nuclear physicists.
They won’t have good models of your AGI design because you have adopted the onion strategy of agi discussion and they also have to deal with disinformation about what even AGI is or can do.
To be successful in that kind of scenario you need to be sufficiently self-sceptical to know where your models are weak and pull in expertise you do not have. Nothing can be “and then a miracle occurs”. Everything that is not a gears level understanding of the situation should be a like a niggling uncomfort that you must deal with.
I’ve not seen any significant effort adopt this mind set.
Could I ask where you’ve looked? MIRI seems to be trying pretty explicitly to develop this mind set, while Paul Christiano has had extensive back and forth on assumptions for his general approach on his medium blog.
As far as I can tell MIRI are asking other people to develop it for thinking about how to build AGI. They have started hiring for developers rather than mathematicians, but AFAIK they haven’t been hiring for the solid state physicists you might want to avoid things like row hammer being a problem.
How to use AGI, they don’t seem to have been hiring for psychologists/economists/political scientists/experts on the process of technological development that might help inform/fill in the gaps in their policy/strategy expertise. Unless they have lots of unknown resources.