Write several pieces of analysis code, ideally in different languages, and check that the results are the same? Even better, have someone else replicate your analysis code. That way you have a somewhat independent source of confirmation.
Also, use practices like tons of unit testing which minimize the chance for bugs in your code. All this must be done before you see the results, of course.
Is this confirmation bias really that bad in practice? Scientists get credit for upsetting previous consensus. So this may lead potentially disruptive research to happen slightly less often. But it remains the case that an attempted change to the consensus—a “surprising” result will still get changed eventually, by someone who doesn’t question the surprising result, or questions it but thoroughly reviews their code and stands by it. So evidence for change will come slightly less often than it could, but changes will still be correct. Doesn’t seem like a big deal.
Science got the charge on an electron right, even after Milliken’s mistake.
Write several pieces of analysis code, ideally in different languages, and check that the results are the same? Even better, have someone else replicate your analysis code. That way you have a somewhat independent source of confirmation.
Also, use practices like tons of unit testing which minimize the chance for bugs in your code. All this must be done before you see the results, of course.
Is this confirmation bias really that bad in practice? Scientists get credit for upsetting previous consensus. So this may lead potentially disruptive research to happen slightly less often. But it remains the case that an attempted change to the consensus—a “surprising” result will still get changed eventually, by someone who doesn’t question the surprising result, or questions it but thoroughly reviews their code and stands by it. So evidence for change will come slightly less often than it could, but changes will still be correct. Doesn’t seem like a big deal.
Science got the charge on an electron right, even after Milliken’s mistake.