Different minds use different criteria to evaluate an argument. Suppose that half the population were perfect rationalists, whose criteria for judging an argument depended only on Occam’s razor and Bayesian updates. The other half are hard-coded biblical literalists, who only believe statements based on religious authority. So half the population will consider “Here are the short equations, showing that this concept has low Komelgorov complexity” to be a valid argument, the other half consider, “Pope Clement said …” to be a strong argument.
Suppose that any position that has strong religious and strong rationalist arguments for it is so obvious that no one is doubting or discussing it. Then most propositions believed by half the population have strong rationalist support, or strong religious support, but not both. If you are a rationalist and see one fairly good rationalist argument for X, you search for more info about X. Any religious arguments get dismissed as nonsense.
The end result is that the rationalists are having a serious discussion about AI risk among themselves. The religous dismiss AI as ludicrous based on some bible verse.
The religious people are having a serious discussion about the second coming of Christ and judgement day, which the rationalists dismiss as ludicrous.
The end result is a society where most of the people who have read much about AI risk think its a thing, and most of the people who have read much about judgement day think its a thing.
If you took some person from one side and forced them to read all the arguments on the other, they still wouldn’t believe. Each side has the good arguments under their criteria of what a good argument is.
The rationalists say that the religious have poor epistemic luck, there is nothing we can do to help them now, when super-intelligence comes it can rewire their brains. The religious say that the rationalists are cursed by the devil, when judgement day comes, they will be converted by the glory of god.
The rationalists are designing a super-intelligence, the religious are praying for judgement day.
Bad ideas and good ones can have similar social dynamics because most of the social dynamics around an idea depends on human nature.
Different minds use different criteria to evaluate an argument. Suppose that half the population were perfect rationalists, whose criteria for judging an argument depended only on Occam’s razor and Bayesian updates. The other half are hard-coded biblical literalists, who only believe statements based on religious authority. So half the population will consider “Here are the short equations, showing that this concept has low Komelgorov complexity” to be a valid argument, the other half consider, “Pope Clement said …” to be a strong argument.
Suppose that any position that has strong religious and strong rationalist arguments for it is so obvious that no one is doubting or discussing it. Then most propositions believed by half the population have strong rationalist support, or strong religious support, but not both. If you are a rationalist and see one fairly good rationalist argument for X, you search for more info about X. Any religious arguments get dismissed as nonsense.
The end result is that the rationalists are having a serious discussion about AI risk among themselves. The religous dismiss AI as ludicrous based on some bible verse.
The religious people are having a serious discussion about the second coming of Christ and judgement day, which the rationalists dismiss as ludicrous.
The end result is a society where most of the people who have read much about AI risk think its a thing, and most of the people who have read much about judgement day think its a thing.
If you took some person from one side and forced them to read all the arguments on the other, they still wouldn’t believe. Each side has the good arguments under their criteria of what a good argument is.
The rationalists say that the religious have poor epistemic luck, there is nothing we can do to help them now, when super-intelligence comes it can rewire their brains. The religious say that the rationalists are cursed by the devil, when judgement day comes, they will be converted by the glory of god.
The rationalists are designing a super-intelligence, the religious are praying for judgement day.
Bad ideas and good ones can have similar social dynamics because most of the social dynamics around an idea depends on human nature.