As I understand it, the scenario is that you’re hearing a complicated argument, and you don’t fully grok or internalize it. As advised by “Making Your Explicit Reasoning Trustworthy”, you have decided not to believe it fully.
The problem comes in the second argument—should you take the advice of the person (or meme) that you at least somewhat mistrust in “correcting” for your mistrust? As you point out, if the person (or the meme) is self-serving, then the original proposal and the correction procedure will fit together neatly to cause a successful mugging.
I think concerns like this suggest that we ought to be using some sort of robust statistics—that is, the direction of the argument and the fact that the conclusion is extreme should influence our conclusions, but the magnitude cannot be allowed to influence our conclusions.
The person need not even be self-serving. All people respond to incentives, and since publishing popular results is rewarding (in fame; often financially as well) the creators of novel arguments will become more likely to believe those arguments.
As I understand it, the scenario is that you’re hearing a complicated argument, and you don’t fully grok or internalize it. As advised by “Making Your Explicit Reasoning Trustworthy”, you have decided not to believe it fully.
The problem comes in the second argument—should you take the advice of the person (or meme) that you at least somewhat mistrust in “correcting” for your mistrust? As you point out, if the person (or the meme) is self-serving, then the original proposal and the correction procedure will fit together neatly to cause a successful mugging.
I think concerns like this suggest that we ought to be using some sort of robust statistics—that is, the direction of the argument and the fact that the conclusion is extreme should influence our conclusions, but the magnitude cannot be allowed to influence our conclusions.
The person need not even be self-serving. All people respond to incentives, and since publishing popular results is rewarding (in fame; often financially as well) the creators of novel arguments will become more likely to believe those arguments.