The amount of criticism does not reflect the amount of extraordinary statements being made here.
This is an important point brought up in a comment. Even though XiXiDu isn’t a true outside perspective by any means, that’s how I would imagine many newcomers to react, at first glance.
What we need to keep in mind is that the inferential gap between LW’s claims and the general populace is gargantuan. We’re dealing with people who often still believe in a man in the sky. Even if we restricted the target audience to the “decision makers”, who are generally better at compartmentalizing their unreasonable beliefs away, the inferential gap is still very large. Especially because we’re not talking about an unbiased audience. There are strong economic and reputational incentives to pumping out the most intelligent artificial agent, and very weak incentives (who wants to anonymously potentially save the planet if you can get rich instead?) to curtail one’s efforts due to friendliness considerations. Even in a best case scenario, the world is largely doomed if this goes the same way that Tragedies of the Commons usually go.
Yeah, people at MIRI know about the enormity of the task, and yet we should perpetually remind ourselves of it, because most of the claims are just so self-evidently obvious that it’s easy to forget that they’re like an ugh-field to outsiders who don’t want to slay their holy cows and are looking for some motivated-cognition getaway. It’s easy to forget that what is akin to 2+2=4 (e.g., orthogonality thesis) for us is “what? crackpot!” territory for others (even if it’s getting slightly more mainstream-palatable, see e.g. Elon Musk’s recent comments).
Therefore I think that criticisms—immediately available, immediately answered—of those claims are really important. Any official document should have a highlighted link to, if not an appendix of, the most convincing criticisms of the new claim, together with answers. If we lack the manpower, even just encouraging top-level authors to compile criticisms from the comments, or to include a final section with their own best “devil’s advocate” arguments and their responses.
The people who need convincing aren’t the ones who are nodding along anyways. It’s those who suspiciously narrow their eyes while reading, going “surely they are crackpots”, then come up with some convenient “unaddressed/devastating criticism” which serves as a pretense to shake their heads and close the site. But—if there are immediate criticisms, steelmanned, and addressed, a significant fraction might come around. At least their go-to excuse would be invalidated.
A convincing narrative is even more so if it has convincing critics, who are convincingly addressed. As much as XiXiDu may have been considered a thorn in MIRI’s side, the value of publicly addressing criticisms as strengthening MIRI’s arguments may have been underestimated. Also, reporters who during their “research” come across XiXiDu’s blog and see his criticisms addressed, even if only by linking to LW articles, will have a harder job at their usual “lol look at the nerd rapture”-hackjobs.
ETA: Sorry for the subpar phrasing, my writing environment is … not … ideal.
I agree with much of this, but it seems like Eliezer (and MIRI?) mostly want to reach people with strong mathematical talent. We observe much more interest among such people since Eliezer started; in particular he trolled Wei Dai into creating UDT. I haven’t looked, but I would guess XiXiDu has criticized him for some part of the way he proposed TDT with the associated trolling of mathematicians.
This is an important point brought up in a comment. Even though XiXiDu isn’t a true outside perspective by any means, that’s how I would imagine many newcomers to react, at first glance.
What we need to keep in mind is that the inferential gap between LW’s claims and the general populace is gargantuan. We’re dealing with people who often still believe in a man in the sky. Even if we restricted the target audience to the “decision makers”, who are generally better at compartmentalizing their unreasonable beliefs away, the inferential gap is still very large. Especially because we’re not talking about an unbiased audience. There are strong economic and reputational incentives to pumping out the most intelligent artificial agent, and very weak incentives (who wants to anonymously potentially save the planet if you can get rich instead?) to curtail one’s efforts due to friendliness considerations. Even in a best case scenario, the world is largely doomed if this goes the same way that Tragedies of the Commons usually go.
Yeah, people at MIRI know about the enormity of the task, and yet we should perpetually remind ourselves of it, because most of the claims are just so self-evidently obvious that it’s easy to forget that they’re like an ugh-field to outsiders who don’t want to slay their holy cows and are looking for some motivated-cognition getaway. It’s easy to forget that what is akin to 2+2=4 (e.g., orthogonality thesis) for us is “what? crackpot!” territory for others (even if it’s getting slightly more mainstream-palatable, see e.g. Elon Musk’s recent comments).
Therefore I think that criticisms—immediately available, immediately answered—of those claims are really important. Any official document should have a highlighted link to, if not an appendix of, the most convincing criticisms of the new claim, together with answers. If we lack the manpower, even just encouraging top-level authors to compile criticisms from the comments, or to include a final section with their own best “devil’s advocate” arguments and their responses.
The people who need convincing aren’t the ones who are nodding along anyways. It’s those who suspiciously narrow their eyes while reading, going “surely they are crackpots”, then come up with some convenient “unaddressed/devastating criticism” which serves as a pretense to shake their heads and close the site. But—if there are immediate criticisms, steelmanned, and addressed, a significant fraction might come around. At least their go-to excuse would be invalidated.
A convincing narrative is even more so if it has convincing critics, who are convincingly addressed. As much as XiXiDu may have been considered a thorn in MIRI’s side, the value of publicly addressing criticisms as strengthening MIRI’s arguments may have been underestimated. Also, reporters who during their “research” come across XiXiDu’s blog and see his criticisms addressed, even if only by linking to LW articles, will have a harder job at their usual “lol look at the nerd rapture”-hackjobs.
ETA: Sorry for the subpar phrasing, my writing environment is … not … ideal.
I agree with much of this, but it seems like Eliezer (and MIRI?) mostly want to reach people with strong mathematical talent. We observe much more interest among such people since Eliezer started; in particular he trolled Wei Dai into creating UDT. I haven’t looked, but I would guess XiXiDu has criticized him for some part of the way he proposed TDT with the associated trolling of mathematicians.
Just out of curiosity, what do you mean when you say that Eliezer “trolled Wei Dai” (and other mathematicians)?
“I do have a proposed alternative ritual of cognition which computes this decision, which this margin is too small to contain”
Why was this trolling? This was in fact true, although Wei Dai’s UDT ended up giving rise to a better framework for future and more general DT work.