Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.
And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.
Though it also calls into question how well John Wentworth’s epistemic processes are working.
Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.
And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.
Though it also calls into question how well John Wentworth’s epistemic processes are working.