I want to add the gear of “even if it actually turns out that OpenPhil was making the right judgment calls the whole time in hindsight, the fact that it’s hard from the outside to know that has some kind of weird Epistemic Murkiness effects that are confusing to navigate, at the very least kinda suck, and maybe are Quite Bad.”
I’ve been trying to articulate the costs of this sort of thing lately and having trouble putting it into words, and maybe it’ll turn out this problem was less of a big deal than it currently feels like to me. But, something like the combo of
a) the default being for many people to trust OpenPhil
b) many people who are paying attention think that they should at least be uncertain about it, and somewhere on a “slightly wary” to “paranoid” scale. and...
c) this at least causes a lot of wasted cognitive cycles
d) it’s… hard to figure out how big a deal to make of it. A few people (i.e. habryka or previously Benquo or Jessicata) make it their thing to bring up concerns frequently. Some of those concerns are, indeed, overly paranoid, but, like, it wasn’t actually reasonable to calibrate the wariness/conflict-theory-detector to zero, you have to make guesses. This is often exhausting and demoralizing for the people doing it. People typically only select into this sort of role if they’re a bit more prone to conflict about it, which means a lot of the work is kinda thankless because people are pushing back on you for being too conflicty. Something about this compounds over time.
e) the part that feels hardest to articulate and maybe is fake is that, there’s something of a “group epistemic process” going on in the surrounding community, and everyone either not tracking this sort of thing, or tracking it but not sure how to take it or what to do about it… I’m not sure how to describe it better than “I dunno something about the group orienting process subtly epistemically fucked” and/or “people just actually take sanity-damage from it.”
(“subtly epistemically fucked” might operationalize as “it takes an extra 1-3 years for things to become consensus knowledge/beliefs than it’d otherwise take”)
Some of those concerns are, indeed, overly paranoid
I am actually curious if you have any overly paranoid predictions from me. I was today lamenting that despite feeling paranoid on this stuff all the time, I de-facto have still been quite overly optimistic in almost all of my predictions on this topic (like, I only gave SPARC a 50% chance of being defunded a few months ago, which I think was dumb, and I was not pessimistic enough to predict the banning of all right-associated project, and not pessimistic enough to predict a bunch of other grant decisions that I feel weird talking publicly about).
The predictions that seemed (somewhat) overly paranoid of yours were more about Anthropic than OpenPhil, and the dynamic seemed similar and I didn’t check that hard while writing the comment. (maybe some predictions about how/why the OpenAI board drama went down, which was at the intersection of all three orgs, which I don’t think have been explicitly revealed to have been “too paranoid” but I’d still probably take bets against)
(I think I agree that overall you were more like “not paranoid enough” than “too paranoid”, although I’m not very confident)
My sense is my predictions about Anthropic have also not been pessimistic enough, though we have not yet seen most of the evidence. Maybe a good time to make bets.
I kinda don’t want to litigate it right now, but, I was thinking “I can think of one particular Anthropic prediction Habryka made that seemed false and overly pessimistic to me”, which doesn’t mean I think you’re overall uncalibrated about Anthropic, and/or not pessimistic enough.
And (I think Habryka got this but for benefit of others), a major point of my original comment was not just “you might be overly paranoid/pessimistic in some cases”, but, ambiguity about how paranoid/pessimistic is appropriate to be results in some kind of confusing, miasmic social-epistemic process (where like maybe you are exactly calibrated on how pessimistic to be, but it comes across as too aggro to other people, who pushback). This can be bad whether you’re somewhat-too-pessimistic, somewhat-too-optimistic, or exactly calibrated.
My recollection is that Habryka seriously considered hypotheses that involved worse and more coordinated behavior than reality, but that this is different from “this was his primary hypothesis that he gave the most probability mass to”. And then he did some empiricism and falsified the hypotheses and I’m glad those hypotheses were considered and investigated.
Here’s an example of him giving 20-25% to a hypothesis about conspiratorial behavior that I believe has turned out to be false.
Yep, that hypothesis seems mostly wrong, though I more feel like I received 1-2 bits of evidence against it. If the board had stabilized with Sam being fired, even given all I know, I would have still thought a merger with Anthropic to be like ~5%-10% likely.
I want to add the gear of “even if it actually turns out that OpenPhil was making the right judgment calls the whole time in hindsight, the fact that it’s hard from the outside to know that has some kind of weird Epistemic Murkiness effects that are confusing to navigate, at the very least kinda suck, and maybe are Quite Bad.”
I’ve been trying to articulate the costs of this sort of thing lately and having trouble putting it into words, and maybe it’ll turn out this problem was less of a big deal than it currently feels like to me. But, something like the combo of
a) the default being for many people to trust OpenPhil
b) many people who are paying attention think that they should at least be uncertain about it, and somewhere on a “slightly wary” to “paranoid” scale. and...
c) this at least causes a lot of wasted cognitive cycles
d) it’s… hard to figure out how big a deal to make of it. A few people (i.e. habryka or previously Benquo or Jessicata) make it their thing to bring up concerns frequently. Some of those concerns are, indeed, overly paranoid, but, like, it wasn’t actually reasonable to calibrate the wariness/conflict-theory-detector to zero, you have to make guesses. This is often exhausting and demoralizing for the people doing it. People typically only select into this sort of role if they’re a bit more prone to conflict about it, which means a lot of the work is kinda thankless because people are pushing back on you for being too conflicty. Something about this compounds over time.
e) the part that feels hardest to articulate and maybe is fake is that, there’s something of a “group epistemic process” going on in the surrounding community, and everyone either not tracking this sort of thing, or tracking it but not sure how to take it or what to do about it… I’m not sure how to describe it better than “I dunno something about the group orienting process subtly epistemically fucked” and/or “people just actually take sanity-damage from it.”
(“subtly epistemically fucked” might operationalize as “it takes an extra 1-3 years for things to become consensus knowledge/beliefs than it’d otherwise take”)
Anyway, thanks for bringing it up.
I am actually curious if you have any overly paranoid predictions from me. I was today lamenting that despite feeling paranoid on this stuff all the time, I de-facto have still been quite overly optimistic in almost all of my predictions on this topic (like, I only gave SPARC a 50% chance of being defunded a few months ago, which I think was dumb, and I was not pessimistic enough to predict the banning of all right-associated project, and not pessimistic enough to predict a bunch of other grant decisions that I feel weird talking publicly about).
The predictions that seemed (somewhat) overly paranoid of yours were more about Anthropic than OpenPhil, and the dynamic seemed similar and I didn’t check that hard while writing the comment. (maybe some predictions about how/why the OpenAI board drama went down, which was at the intersection of all three orgs, which I don’t think have been explicitly revealed to have been “too paranoid” but I’d still probably take bets against)
(I think I agree that overall you were more like “not paranoid enough” than “too paranoid”, although I’m not very confident)
My sense is my predictions about Anthropic have also not been pessimistic enough, though we have not yet seen most of the evidence. Maybe a good time to make bets.
I kinda don’t want to litigate it right now, but, I was thinking “I can think of one particular Anthropic prediction Habryka made that seemed false and overly pessimistic to me”, which doesn’t mean I think you’re overall uncalibrated about Anthropic, and/or not pessimistic enough.
And (I think Habryka got this but for benefit of others), a major point of my original comment was not just “you might be overly paranoid/pessimistic in some cases”, but, ambiguity about how paranoid/pessimistic is appropriate to be results in some kind of confusing, miasmic social-epistemic process (where like maybe you are exactly calibrated on how pessimistic to be, but it comes across as too aggro to other people, who pushback). This can be bad whether you’re somewhat-too-pessimistic, somewhat-too-optimistic, or exactly calibrated.
My recollection is that Habryka seriously considered hypotheses that involved worse and more coordinated behavior than reality, but that this is different from “this was his primary hypothesis that he gave the most probability mass to”. And then he did some empiricism and falsified the hypotheses and I’m glad those hypotheses were considered and investigated.
Here’s an example of him giving 20-25% to a hypothesis about conspiratorial behavior that I believe has turned out to be false.
Yep, that hypothesis seems mostly wrong, though I more feel like I received 1-2 bits of evidence against it. If the board had stabilized with Sam being fired, even given all I know, I would have still thought a merger with Anthropic to be like ~5%-10% likely.