Agreed with Rohin that a key consideration is whether you are trying to form truer beliefs, or to contribute novel ideas, and this in turn depends on what role you are playing in the collective enterprise that is AI safety.
If you’re the person in charge of humanity’s AI safety strategy, or a journalist tasked with informing the public, or a policy person talking to governments, it makes a ton of sense to build a “good gears-level model of what their top 5 alignment researchers believe and why”. If you’re a researcher, tasked with generating novel ideas that the rest of the community will filter and evaluate, this is probably not where you want to start!
In particular I basically buy the “unique combination of facts” model of invention: you generate novel ideas when you have a unique subset of the collective’s knowledge, so the ideas seems obvious to you and weird or wrong to everyone else.
Two examples from leading scientists (admittedly in more paradigmatic fields):
I remember Geoff Hinton saying at his Turing award lecture that he strongly advised new grad students not to read the literature before trying, for months, to solve the problem themselves.
If you read all the time what other people have done you will think the way they thought. If you want to think new thoughts that are different, then do what a lot of creative people do—get the problem reasonably clear and then refuse to look at any answers until you’ve thought the problem through carefullyhow you would do it, how you could slightly change the problem to be the correct one. So yes, you need to keep up. You need to keep up more to find out what the problems are than to read to find the solutions.
Agreed with Rohin that a key consideration is whether you are trying to form truer beliefs, or to contribute novel ideas, and this in turn depends on what role you are playing in the collective enterprise that is AI safety.
If you’re the person in charge of humanity’s AI safety strategy, or a journalist tasked with informing the public, or a policy person talking to governments, it makes a ton of sense to build a “good gears-level model of what their top 5 alignment researchers believe and why”. If you’re a researcher, tasked with generating novel ideas that the rest of the community will filter and evaluate, this is probably not where you want to start!
In particular I basically buy the “unique combination of facts” model of invention: you generate novel ideas when you have a unique subset of the collective’s knowledge, so the ideas seems obvious to you and weird or wrong to everyone else.
Two examples from leading scientists (admittedly in more paradigmatic fields):
I remember Geoff Hinton saying at his Turing award lecture that he strongly advised new grad students not to read the literature before trying, for months, to solve the problem themselves.
Richard Hamming in You and Your Research: