In CFAR, MIRI have the ultimate hedge. If the whole MIRI mission is misdirected or wrong headed, CFAR is designed to create the people who will notice that and do whatever does most need to be done.
I would phrase this more along the lines of “If nothing MIRI does works, or for that matter if everything works but it’s still not enough, CFAR tries to get a fully generic bonus on paths unseen in advance.”
Switch out ‘harmful’ for ‘aiming at the wrong goals’, since that’s the possibility cipher raised and Eliezer didn’t. (Those goals might make MIRI useless; harmful isn’t the only possibility.)
I’d guess that Eliezer’s rephrasing reflects (1) his vagueness about the means by which CFAR would act as game-changer, and (2) his being much more worried that MIRI lacks the ingenuity and intellectual firepower to achieve its goals than worried that MIRI’s deepest values and concerns are misplaced. CFAR might also help in some low-probability scenarios, but it’s the likelier scenarios that make Eliezer a CFAR supporter.
Only if you have an extremely high opinion of the work CFAR does to the extent that it is sufficient to overcome the extremely strong signalling and group affiliation effects that MIRI is as vulnerable to as anyone else. (anyone who has been reading LW for more than an hour can think of the obvious examples.)
I mean, the main way CFAR might be able to overcome this isn’t by being super extremely unbiased, but by bringing a wide diversity of good thinkers into the network (with diverse starting views, diverse group affiliations, and diverse basic thinking styles). This is totally a priority for us.
In CFAR, MIRI have the ultimate hedge. If the whole MIRI mission is misdirected or wrong headed, CFAR is designed to create the people who will notice that and do whatever does most need to be done.
I would phrase this more along the lines of “If nothing MIRI does works, or for that matter if everything works but it’s still not enough, CFAR tries to get a fully generic bonus on paths unseen in advance.”
Do you choose that rephrasing because you don’t see how MIRI’s work could be harmful or because there is nothing CFAR can do in that case?
Switch out ‘harmful’ for ‘aiming at the wrong goals’, since that’s the possibility cipher raised and Eliezer didn’t. (Those goals might make MIRI useless; harmful isn’t the only possibility.)
I’d guess that Eliezer’s rephrasing reflects (1) his vagueness about the means by which CFAR would act as game-changer, and (2) his being much more worried that MIRI lacks the ingenuity and intellectual firepower to achieve its goals than worried that MIRI’s deepest values and concerns are misplaced. CFAR might also help in some low-probability scenarios, but it’s the likelier scenarios that make Eliezer a CFAR supporter.
Only if you have an extremely high opinion of the work CFAR does to the extent that it is sufficient to overcome the extremely strong signalling and group affiliation effects that MIRI is as vulnerable to as anyone else. (anyone who has been reading LW for more than an hour can think of the obvious examples.)
I mean, the main way CFAR might be able to overcome this isn’t by being super extremely unbiased, but by bringing a wide diversity of good thinkers into the network (with diverse starting views, diverse group affiliations, and diverse basic thinking styles). This is totally a priority for us.