Geoff describes being harmed by some sort of initial rejection by the rationality/EA community (around 2011? 2010?).
One of the interesting things about that timeframe is that a lot of the stuff is online; here’s the 2012 discussion (Jan 9th, Jan 10th, Sep 19th), for example. (I tried to find his earliest comment that I remembered, but I don’t think it was with the Geoff_Anders account or it wasn’t on LessWrong; I think it was before Leverage got started, and people responded pretty skeptically then also?)
One takeaway: Eliezer’s interaction with Geoff does seem like Eliezer was making some sort of mistake. Not sure what the core is, but, one part is like conflating [evidence, the kind that can be interpersonally verified] with [evidence, the kind that accumulates subconsciously as many abstract percepts and heuristics, which can be observably useful while still pre-theoretic, pre-legible]. Like, maybe Eliezer wants to only talk with people where either (1) they already have enough conceptual overlap that abstract cutting-edge theories also always cash out as perceivable predictions, or (2) aren’t trying to share pre-legible theories. But that’s different from Geoff making some terrible incurable mistake of reasoning. (Terrible incurable mistakes are certainly correlated with illegibility, but that’s not something to Goodhart.)
I’m sort of surprised that you’d interpret that as a mistake. It seems to me like Eliezer is running a probabilistic strategy, which has both type I and type II errors, and so a ‘mistake’ is something like “setting the level wrong to get a bad balance of errors” instead of “the strategy encountered an error in this instance.” But also I don’t have the sense that Eliezer was making an error.
. It seems to me like Eliezer is running a probabilistic strategy
It sounds like this describes every strategy? I guess you mean, he’s explicitly taking into account that he’ll make errors, and playing the probabilities to get good expected value. So this makes sense, like I’m not saying he was making a strategic mistake by not, say, working with Geoff. I’m saying:
(internally) Well this is obviously wrong. Minds just don’t work by those sorts of bright-line psychoanalytic rules written out in English, and proposing them doesn’t get you anywhere near the level of an interesting cognitive algorithm.[...]
(out loud) What does CT say I should experience seeing, that existing cognitive science wouldn’t tell me to expect?
Geoff: (Something along the lines of “CT isn’t there yet”[...])
(out loud) Okay, then I don’t believe in CT because without evidence there’s no way you could know it even if it was true.
sounds like he’s conflating shareable and non-shareable evidence. Geoff could have seen a bunch of stuff and learned heuristics that he couldn’t articulately express other than with silly-seeming “bright-line psychoanalytic rules written out in English”. Again, it can make sense to treat this as “for my purposes, equivalent to being obviously wrong”. But like, it’s not really equivalent, you just *don’t know* whether the person has hidden evidence.
Even if all you have is a bunch of stuff and learned heuristics, you should be able to make testable predictions with them. Otherwise, how can you tell whether they’re any good or not?
Whether the evidence that persuaded you is sharable or not doesn’t affect this. For example, you might have a prior that a new psychotherapy technique won’t outperform a control because you’ve read like 30 different cases where a leading psychiatrist invented a new therapy technique, reported great results, and then couldn’t train anyone else to get the same results he did. That’s my prior, and I suspect it’s Eliezer’s, but if I wanted to convince you of it I’d have a tough time because there’s not really a single crux, just those 30 different cases that slowly accumulated. And yet, even though I can’t share the source of my belief, I can use it to make concrete testable predictions: when they do an RCT for the 31st therapy technique, it won’t outperform the control.
Geoff-in-Eliezer’s-ancedote has not reached this point. This is especially bad for a developing theory: if Geoff makes a change to CT, how will he tell if the new CT is better or worse than the old one? Geoff-replying-to-Eliezer takes this criticism seriously, and says he can make concrete, if narrow, predictions about specific people he’s charted.
you should be able to make testable predictions with them
Certainly. But you might not be able to make testable predictions for which others will readily agree with your criteria for judgement. In the exchange, Geoff gives some “evidence”, and in other places he gives additional “evidence”. It’s not really convincing to me, but it at least has the type signature of evidence. Eliezer responds:
Which sounds a lot like standard cognitive dissonance theory
This is eliding that Geoff probably has significant skill in identifying more detail of how beliefs and goals interact, beyond just what someone would know if they heard about cognitive dissonance theory. Like basically I’m saying that if Eliezer sat with Geoff for a few hours through a few sessions of Geoff doing his thing with some third person, Eliezer would see Geoff behave in a way that suggests falsifiable understanding that Eliezer doesn’t have. (Again, not saying he should have done that or anything.)
One of the interesting things about that timeframe is that a lot of the stuff is online; here’s the 2012 discussion (Jan 9th, Jan 10th, Sep 19th), for example. (I tried to find his earliest comment that I remembered, but I don’t think it was with the Geoff_Anders account or it wasn’t on LessWrong; I think it was before Leverage got started, and people responded pretty skeptically then also?)
Thanks!
One takeaway: Eliezer’s interaction with Geoff does seem like Eliezer was making some sort of mistake. Not sure what the core is, but, one part is like conflating [evidence, the kind that can be interpersonally verified] with [evidence, the kind that accumulates subconsciously as many abstract percepts and heuristics, which can be observably useful while still pre-theoretic, pre-legible]. Like, maybe Eliezer wants to only talk with people where either (1) they already have enough conceptual overlap that abstract cutting-edge theories also always cash out as perceivable predictions, or (2) aren’t trying to share pre-legible theories. But that’s different from Geoff making some terrible incurable mistake of reasoning. (Terrible incurable mistakes are certainly correlated with illegibility, but that’s not something to Goodhart.)
I’m sort of surprised that you’d interpret that as a mistake. It seems to me like Eliezer is running a probabilistic strategy, which has both type I and type II errors, and so a ‘mistake’ is something like “setting the level wrong to get a bad balance of errors” instead of “the strategy encountered an error in this instance.” But also I don’t have the sense that Eliezer was making an error.
It sounds like this describes every strategy? I guess you mean, he’s explicitly taking into account that he’ll make errors, and playing the probabilities to get good expected value. So this makes sense, like I’m not saying he was making a strategic mistake by not, say, working with Geoff. I’m saying:
sounds like he’s conflating shareable and non-shareable evidence. Geoff could have seen a bunch of stuff and learned heuristics that he couldn’t articulately express other than with silly-seeming “bright-line psychoanalytic rules written out in English”. Again, it can make sense to treat this as “for my purposes, equivalent to being obviously wrong”. But like, it’s not really equivalent, you just *don’t know* whether the person has hidden evidence.
Even if all you have is a bunch of stuff and learned heuristics, you should be able to make testable predictions with them. Otherwise, how can you tell whether they’re any good or not?
Whether the evidence that persuaded you is sharable or not doesn’t affect this. For example, you might have a prior that a new psychotherapy technique won’t outperform a control because you’ve read like 30 different cases where a leading psychiatrist invented a new therapy technique, reported great results, and then couldn’t train anyone else to get the same results he did. That’s my prior, and I suspect it’s Eliezer’s, but if I wanted to convince you of it I’d have a tough time because there’s not really a single crux, just those 30 different cases that slowly accumulated. And yet, even though I can’t share the source of my belief, I can use it to make concrete testable predictions: when they do an RCT for the 31st therapy technique, it won’t outperform the control.
Geoff-in-Eliezer’s-ancedote has not reached this point. This is especially bad for a developing theory: if Geoff makes a change to CT, how will he tell if the new CT is better or worse than the old one? Geoff-replying-to-Eliezer takes this criticism seriously, and says he can make concrete, if narrow, predictions about specific people he’s charted.
Certainly. But you might not be able to make testable predictions for which others will readily agree with your criteria for judgement. In the exchange, Geoff gives some “evidence”, and in other places he gives additional “evidence”. It’s not really convincing to me, but it at least has the type signature of evidence. Eliezer responds:
This is eliding that Geoff probably has significant skill in identifying more detail of how beliefs and goals interact, beyond just what someone would know if they heard about cognitive dissonance theory. Like basically I’m saying that if Eliezer sat with Geoff for a few hours through a few sessions of Geoff doing his thing with some third person, Eliezer would see Geoff behave in a way that suggests falsifiable understanding that Eliezer doesn’t have. (Again, not saying he should have done that or anything.)