[T]his error strikes me as … emblematic of a certain common failure mode within the rationalist community (of which I count myself a part). This common failure mode is to over-value our own intelligence and under-value institutional knowledge (whether from the scientific community or the Amazon marketplace), and thus not feel the need to tread carefully when the two come into conflict.
In that comment and the resulting thread, we discuss the implications of that with respect to the rationalist community’s understanding of Alzheimer’s disease, a disease I’ve studied in great depth. I’ve mostly found the community to have very strong opinions on that subject and disdain for the scientific community studying it, but very superficial engagement with the relevant scientific literature. Every single time I’ve debated the matter in detail with someone (maybe 5–10 times total), I’ve persuaded them that 1) the scientific community has a much better understanding of the disease than they realized and 2) that the amyloid hypothesis is compelling as a causal explanation. However, people in the rationalist community often have strongly-held, wrong opinions before (or in lieu of) these debates with me.
Ironically, the same thing happened in that thread: my interlocutor, John Wentworth, appreciated my corrections. However, I ultimately found the discussion a bit unsatisfying, because I don’t know that he made any meta-updates from it concerning the level of confidence that he started with without having seriously engaged with the literature.
Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.
And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.
Though it also calls into question how well John Wentworth’s epistemic processes are working.
I wrote about this here:
In that comment and the resulting thread, we discuss the implications of that with respect to the rationalist community’s understanding of Alzheimer’s disease, a disease I’ve studied in great depth. I’ve mostly found the community to have very strong opinions on that subject and disdain for the scientific community studying it, but very superficial engagement with the relevant scientific literature. Every single time I’ve debated the matter in detail with someone (maybe 5–10 times total), I’ve persuaded them that 1) the scientific community has a much better understanding of the disease than they realized and 2) that the amyloid hypothesis is compelling as a causal explanation. However, people in the rationalist community often have strongly-held, wrong opinions before (or in lieu of) these debates with me.
Ironically, the same thing happened in that thread: my interlocutor, John Wentworth, appreciated my corrections. However, I ultimately found the discussion a bit unsatisfying, because I don’t know that he made any meta-updates from it concerning the level of confidence that he started with without having seriously engaged with the literature.
Potentially relevant: Random facts can come back to bite you.
Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.
And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.
Though it also calls into question how well John Wentworth’s epistemic processes are working.