Good question. The sequences focus on thinking correctly more than arguing successfully, and I think most people who stick around here develop these intuitions through a process of learning to think more like Eliezer does.
The first possible cause I see for why strong intuitions are not convertible into convincing arguments is long inferential distances—the volume of words is simply too great to fit into a reasonably-sized exchange. But the Hanson-Yudkowsky Foom Debate was unreasonably long, and as I understand it, both parties left with their strong intuitions fairly intact.
The post-mortem from the Foom debate seemed to center around emotional attachment to ideas, and their intertwining with identity. This looks like the most useful level for bystander-based examination. I’d be interested to know how well, say, priming yourself for disinterested detachment and re-examining both arguments works for a Foom-sized debate as opposed to one of ordinary length.
evaluate how reliable each party’s intuitions are likely to be, and then use that to form a probabilistic mixture of the two sides’ positions.
I’d break this down into
Outside view of each party, if other intuitions are available for evaluation.
Outside view of each intuition, although the debaters probably already did this for each other.
A probabilistic graph for each party, involving the intuitions and the arguments. Through what paths did the intuitions generate the arguments? If there was any causality going the other direction, how did that work?
What other methods for evaluating inexpressibly strong intuitions are there?
Good question. The sequences focus on thinking correctly more than arguing successfully, and I think most people who stick around here develop these intuitions through a process of learning to think more like Eliezer does.
The first possible cause I see for why strong intuitions are not convertible into convincing arguments is long inferential distances—the volume of words is simply too great to fit into a reasonably-sized exchange. But the Hanson-Yudkowsky Foom Debate was unreasonably long, and as I understand it, both parties left with their strong intuitions fairly intact.
The post-mortem from the Foom debate seemed to center around emotional attachment to ideas, and their intertwining with identity. This looks like the most useful level for bystander-based examination. I’d be interested to know how well, say, priming yourself for disinterested detachment and re-examining both arguments works for a Foom-sized debate as opposed to one of ordinary length.
I’d break this down into
Outside view of each party, if other intuitions are available for evaluation.
Outside view of each intuition, although the debaters probably already did this for each other.
A probabilistic graph for each party, involving the intuitions and the arguments. Through what paths did the intuitions generate the arguments? If there was any causality going the other direction, how did that work?
What other methods for evaluating inexpressibly strong intuitions are there?