The first four points you raised seem to rely on prestige or social proof.
I’m trying to avoid applying my own potentially biased judgement, and it seems pretty necessary to use either my own judgement or some kind of social proof. I admit this has flaws.
But I also think that the prestige of programs like MATS makes the talent quality extremely high (though I may believe Linda on why this is okay), and that Forrest Landry’s writing is probably crankery and if alignment is impossible it’s likely for a totally different reason.
We also do not focus on getting participants to submit papers to highly selective journals or ML conferences (though not necessarily highly selective for quality of research with regards to preventing AI-induced extinction).
I think we just have different attitudes to this. I will note that ML conferences have other benefits, like networking, talking to experienced researchers, and getting a sense for the field (for me going to ICML and NeurIPS was very helpful), and for domains people already care about, peer review is a basic check that work is “real”—novel, well-communicated, and meeting some minimum quality bar. Interpretability is becoming one of those domains.
It is relevant to consider the quality of research thinking coming out of the camp. If you or someone else had the time to look through some of those posts, I’m curious to get your sense.
I unfortunately don’t have the time or expertise to do this, because these posts are in many different areas. One I do understand is this post because it cites mine and I know Jeremy Gillen. The quality and volume of work seem a bit lower than my post, which took 9 person-weeks and is (according to me) not quite good enough to publish or further pursue, though it may develop into a workshop paper. The soft optimization post took 24 person-weeks (assuming 4 people half-time for 12 weeks) plus some of Jeremy’s time. I had no training in probability theory or statistics, although I was somewhat lucky in finding a topic that did not require it.
If you clicked through Paul’s somewhat hyperbolic comment of “the entire scientific community would probably consider this writing to be crankery” and consider my response, what are your thoughts on whether that response is reasonable or not? Ie. consider whether the response is relevant, soundly premised, and consistently reasoned.
I have no idea because I don’t understand it. It reads vaguely like a summary of crankery. Possibly I would need to read Forrest Landry’s work, but given that it’s also difficult to read and I currently give 90%+ that it’s crankery, you must understand why I don’t.
The soft optimization post took 24 person-weeks (assuming 4 people half-time for 12 weeks) plus some of Jeremy’s time.
Team member here. I think this is a significant overestimate, I’d guess at 12-15 person-weeks. If it’s relevant I can ask all former team members how much time they spent; it was around 10h per week for me. Given that we were beginners and spent a lot of time learning about the topic, I feel we were doing fine and learnt a lot.
Working on this part-time was difficult for me and the fact that people are not working on these things full-time in the camp should be considered when judging research output.
I have no idea because I don’t understand it. It reads vaguely like a summary of crankery. Possibly I would need to read Forrest Landry’s work, but given that it’s also difficult to read...
This is honest.
Maybe it would be good to wait for people who can spend the time to consider the argument to come back on this?
I mentioned that Anders Sandberg has spent 6 hours discussing the argument in-depth. Several others are looking into the argument.
What feels concerning is when people rely on surface-level impressions, such as the ones you cited, to make judgements about an argument where the inferential gap is high.
It’s not good for the epistemic health of our community when insiders spread quick confident judgements about work by outside researchers. It can create an epistemic echo chamber.
...and I currently give 90%+ that it’s crankery, you must understand why I don’t
I do get this, given the sheer number of projects in AI Safety that may seem worth considering.
Even if your quick probability guess is 95% for the reasoning being scientifically unsound, what about the remaining 5%?
What is the value of information given the possibility of discovering that alignment efforts will unfortunately not work out? How much would such a discovery change our actions, and the areas of action we would explore and start to understand better?
Historically, changes in scientific paradigms came from unexpected places. Arguments were often written in ways that felt weird and inscrutable to insiders (take a look at Gödel’s first incompleteness theorem).
How much should a community rely on people’s first intuitions on whether some new supposedly paradigm-shifting argument is crankery or not?
Should the presentation of a formal argument (technical proof) be judged on the basis of social proof?
I’m trying to avoid applying my own potentially biased judgement, and it seems pretty necessary to use either my own judgement or some kind of social proof. I admit this has flaws.
But I also think that the prestige of programs like MATS makes the talent quality extremely high (though I may believe Linda on why this is okay), and that Forrest Landry’s writing is probably crankery and if alignment is impossible it’s likely for a totally different reason.
I think we just have different attitudes to this. I will note that ML conferences have other benefits, like networking, talking to experienced researchers, and getting a sense for the field (for me going to ICML and NeurIPS was very helpful), and for domains people already care about, peer review is a basic check that work is “real”—novel, well-communicated, and meeting some minimum quality bar. Interpretability is becoming one of those domains.
I unfortunately don’t have the time or expertise to do this, because these posts are in many different areas. One I do understand is this post because it cites mine and I know Jeremy Gillen. The quality and volume of work seem a bit lower than my post, which took 9 person-weeks and is (according to me) not quite good enough to publish or further pursue, though it may develop into a workshop paper. The soft optimization post took 24 person-weeks (assuming 4 people half-time for 12 weeks) plus some of Jeremy’s time. I had no training in probability theory or statistics, although I was somewhat lucky in finding a topic that did not require it.
I have no idea because I don’t understand it. It reads vaguely like a summary of crankery. Possibly I would need to read Forrest Landry’s work, but given that it’s also difficult to read and I currently give 90%+ that it’s crankery, you must understand why I don’t.
Team member here. I think this is a significant overestimate, I’d guess at 12-15 person-weeks. If it’s relevant I can ask all former team members how much time they spent; it was around 10h per week for me. Given that we were beginners and spent a lot of time learning about the topic, I feel we were doing fine and learnt a lot.
Working on this part-time was difficult for me and the fact that people are not working on these things full-time in the camp should be considered when judging research output.
This is honest.
Maybe it would be good to wait for people who can spend the time to consider the argument to come back on this?
I mentioned that Anders Sandberg has spent 6 hours discussing the argument in-depth. Several others are looking into the argument.
What feels concerning is when people rely on surface-level impressions, such as the ones you cited, to make judgements about an argument where the inferential gap is high.
It’s not good for the epistemic health of our community when insiders spread quick confident judgements about work by outside researchers. It can create an epistemic echo chamber.
I do get this, given the sheer number of projects in AI Safety that may seem worth considering.
Having said that, the argument is literally about why AGI could not be sufficiently controlled to stay safe.
Even if your quick probability guess is 95% for the reasoning being scientifically unsound, what about the remaining 5%?
What is the value of information given the possibility of discovering that alignment efforts will unfortunately not work out? How much would such a discovery change our actions, and the areas of action we would explore and start to understand better?
Historically, changes in scientific paradigms came from unexpected places. Arguments were often written in ways that felt weird and inscrutable to insiders (take a look at Gödel’s first incompleteness theorem).
How much should a community rely on people’s first intuitions on whether some new supposedly paradigm-shifting argument is crankery or not?
Should the presentation of a formal argument (technical proof) be judged on the basis of social proof?