Strong +1s to many of the points here. Some things I’d highlight:
Eliezer is not doing the type of reasoning that can justifiably defend the level of confidence he claims to have. If he were, he’d have much more to say about the specific details of consequentialism, human evolution, and the other key intuitions shaping his thinking. In my debate with him he mentioned many times how difficult he’s found it to explain these ideas to people. I think if he understood these ideas well enough to justify the confidence of his claims, then he wouldn’t have found that as difficult. (I’m sympathetic about Eliezer having in the past engaged with many interlocutors who were genuinely very bad at understanding his arguments. However, it does seem like the lack of detail in those arguments is now a bigger bottleneck.)
I think that the intuitions driving Eliezer’s disagreements with many other alignment researchers are interesting and valuable, and would love to have better-fleshed-out explanations of them publicly available. Eliezer would probably have an easier time focusing on developing his own ideas if other people in the alignment community who were pessimistic about various research directions, and understood the broad shape of his intuitions, were more open and direct about that pessimism. This is something I’ve partly done in this post; and I’m glad that Paul’s partly done it here.
I like the analogy of a mathematician having intuitions about the truth of a theorem. I currently think of Eliezer as someone who has excellent intuitions about the broad direction of progress at a very high level of abstraction—but where the very fact that these intuitions are so abstract rules out the types of path-dependencies that I expect solutions to alignment will actually rely on. At this point, people who find Eliezer’s intuitions compelling should probably focus on fleshing them out in detail—e.g. using toy models, or trying to decompose the concept of consequentialism—rather than defending them at a high level.
I think if he understood these ideas well enough to justify the confidence of his claims, then he wouldn’t have found that as difficult.
But what makes you so confident that it’s not possible for subject-matter experts to have correct intuitions that outpace their ability to articulate legible explanations to others?
Of course, it makes sense for other people who don’t trust the (purported) expert to require an explanation, and not just take the (purported) expert’s word for it. (So, I agree that fleshing out detailed examples is important for advancing our collective state of knowledge.) But the (purported) expert’s own confidence should track correctness, not how easy it is to convince people using words.
But what makes you so confident that it’s not possible for subject-matter experts to have correct intuitions that outpace their ability to articulate legible explanations to others?
Yepp, this is a judgement call. I don’t have any hard and fast rules for how much you should expect experts’ intuitions to plausibly outpace their ability to explain things. A few things which inform my opinion here:
Explaining things to other experts should be much easier than explaining them to the public.
Explaining things to other experts should be much easier than actually persuading those experts.
It’s much more likely that someone has correct intuitions if they have a clear sense of what evidence would make their intuitions stronger.
I don’t think Eliezer is doing particularly well on any of these criteria. In particular, the last one was why I pressed Eliezer to make predictions rather than postdictions in my debate with him. The extent to which Eliezer seemed confused that I cared about this was a noticeable update for me in the direction of believing that Eliezer’s intuitions are less solid than he thinks.
It may be the case that Eliezer has strong object-level intuitions about the details of how intelligence works which he’s not willing to share publicly, but which significantly increase his confidence in his public claims. If so, I think the onus is on him to highlight that so people can make a meta-level update on it.
I agree that intuitions might get you to high confidence without the ability to explain ideas legibly.
That said, I think expert intuitions still need to usually (always?) be grounded out in predictions about something (potentially including the many implicit predictions that are often required to do stuff). It seems to me like Eliezer is probably relying on a combination of:
Predicting stuff from afar. I think that can usually be made legible with a few years’ lead time. I’m sympathetic to the difficulty of doing this (despite my frequent snarky tone), though without doing it I think Eliezer himself should have more doubts about the possibility of hindsight bias if this is really his main source of evidence. In theory this could also be retrodictions about history which would make things more complicated in some ways but faster in others.
Testing intuitions against other already-trusted forms of reasoning, and particularly concrete arguments. In this regime, I don’t think it’s necessarily the case that Eliezer ought to be able to easily write down a convincing version of the arguments, but I do think we should expect him to systematically be right more often when we dig into arguments evaluated using those same already-trusted forms of reasoning. And so on this perspective it’s also quite useful to get into the argument.
I’m also sympathetic to Richard’s point as a more contingent claim about the nature of expert intuition on technical topics—there just aren’t many domains where you can get this much confidence out of intuitions you can’t explain more crisply than this. Though I wouldn’t be super confident about that, and (like you) I would mostly lean on the claim about collective epistemology if trying to push Eliezer to do something differently.
But what makes you so confident that it’s not possible for subject-matter experts to have correct intuitions that outpace their ability to articulate legible explanations to others?
That’s irrelevant, because what Richard wrote was a truism. An Eliezer who understands his own confidence in his ideas will “always” be better at inspiring confidence in those ideas in others. Richard’s statement leads to a conclusion of import (Eliezer should develop arguments to defend his intuitions) precisely because it’s correct whether Eliezer’s intuitions are correct or incorrect.
Strong +1s to many of the points here. Some things I’d highlight:
Eliezer is not doing the type of reasoning that can justifiably defend the level of confidence he claims to have. If he were, he’d have much more to say about the specific details of consequentialism, human evolution, and the other key intuitions shaping his thinking. In my debate with him he mentioned many times how difficult he’s found it to explain these ideas to people. I think if he understood these ideas well enough to justify the confidence of his claims, then he wouldn’t have found that as difficult. (I’m sympathetic about Eliezer having in the past engaged with many interlocutors who were genuinely very bad at understanding his arguments. However, it does seem like the lack of detail in those arguments is now a bigger bottleneck.)
I think that the intuitions driving Eliezer’s disagreements with many other alignment researchers are interesting and valuable, and would love to have better-fleshed-out explanations of them publicly available. Eliezer would probably have an easier time focusing on developing his own ideas if other people in the alignment community who were pessimistic about various research directions, and understood the broad shape of his intuitions, were more open and direct about that pessimism. This is something I’ve partly done in this post; and I’m glad that Paul’s partly done it here.
I like the analogy of a mathematician having intuitions about the truth of a theorem. I currently think of Eliezer as someone who has excellent intuitions about the broad direction of progress at a very high level of abstraction—but where the very fact that these intuitions are so abstract rules out the types of path-dependencies that I expect solutions to alignment will actually rely on. At this point, people who find Eliezer’s intuitions compelling should probably focus on fleshing them out in detail—e.g. using toy models, or trying to decompose the concept of consequentialism—rather than defending them at a high level.
But what makes you so confident that it’s not possible for subject-matter experts to have correct intuitions that outpace their ability to articulate legible explanations to others?
Of course, it makes sense for other people who don’t trust the (purported) expert to require an explanation, and not just take the (purported) expert’s word for it. (So, I agree that fleshing out detailed examples is important for advancing our collective state of knowledge.) But the (purported) expert’s own confidence should track correctness, not how easy it is to convince people using words.
Yepp, this is a judgement call. I don’t have any hard and fast rules for how much you should expect experts’ intuitions to plausibly outpace their ability to explain things. A few things which inform my opinion here:
Explaining things to other experts should be much easier than explaining them to the public.
Explaining things to other experts should be much easier than actually persuading those experts.
It’s much more likely that someone has correct intuitions if they have a clear sense of what evidence would make their intuitions stronger.
I don’t think Eliezer is doing particularly well on any of these criteria. In particular, the last one was why I pressed Eliezer to make predictions rather than postdictions in my debate with him. The extent to which Eliezer seemed confused that I cared about this was a noticeable update for me in the direction of believing that Eliezer’s intuitions are less solid than he thinks.
It may be the case that Eliezer has strong object-level intuitions about the details of how intelligence works which he’s not willing to share publicly, but which significantly increase his confidence in his public claims. If so, I think the onus is on him to highlight that so people can make a meta-level update on it.
I agree that intuitions might get you to high confidence without the ability to explain ideas legibly.
That said, I think expert intuitions still need to usually (always?) be grounded out in predictions about something (potentially including the many implicit predictions that are often required to do stuff). It seems to me like Eliezer is probably relying on a combination of:
Predicting stuff from afar. I think that can usually be made legible with a few years’ lead time. I’m sympathetic to the difficulty of doing this (despite my frequent snarky tone), though without doing it I think Eliezer himself should have more doubts about the possibility of hindsight bias if this is really his main source of evidence. In theory this could also be retrodictions about history which would make things more complicated in some ways but faster in others.
Testing intuitions against other already-trusted forms of reasoning, and particularly concrete arguments. In this regime, I don’t think it’s necessarily the case that Eliezer ought to be able to easily write down a convincing version of the arguments, but I do think we should expect him to systematically be right more often when we dig into arguments evaluated using those same already-trusted forms of reasoning. And so on this perspective it’s also quite useful to get into the argument.
I’m also sympathetic to Richard’s point as a more contingent claim about the nature of expert intuition on technical topics—there just aren’t many domains where you can get this much confidence out of intuitions you can’t explain more crisply than this. Though I wouldn’t be super confident about that, and (like you) I would mostly lean on the claim about collective epistemology if trying to push Eliezer to do something differently.
That’s irrelevant, because what Richard wrote was a truism. An Eliezer who understands his own confidence in his ideas will “always” be better at inspiring confidence in those ideas in others. Richard’s statement leads to a conclusion of import (Eliezer should develop arguments to defend his intuitions) precisely because it’s correct whether Eliezer’s intuitions are correct or incorrect.