Researcher at the Center on Long-Term Risk. All opinions my own.
Anthony DiGiovanni
Karma: 1,048
Resource guide: Unawareness, indeterminacy, and cluelessness
Sorry to be clear, I don’t claim LW has overlooked these topics (except unawareness and alternatives to classical Bayesian epistemology, which I do think have been quite severely neglected). The reason I wrote this post was that the following claims seem non-obvious:
Thinking further about wisdom concepts these days is not just a distraction from “notkilleveryoneism”.
The concepts in the checklist do in fact seem to satisfy conditions (1)+(2) (the definition of “wisdom concepts”). (My impression is that it’s somewhat common for people to think many of the concepts I list admit “objective” answers (i.e. just believe and do what “works” / has the best empirical track record), which all sufficiently intelligent agents will converge to. ETA: Relatedly, it might not be salient to some readers that the answer to “is this decision a catastrophic mistake?” could be sensitive to all these topics.)
The sub-questions I list are open questions. (E.g., I expect it to be controversial that agents aren’t necessarily rationally required to avoid diachronic sure losses.)
Clarifying “wisdom”: Foundational topics for aligned AIs to prioritize before irreversible decisions
There are indeed connections between these ideas, but I think it’s very important not to round unawareness off to either of those two. Unawareness is its own epistemic problem with its own implications. (E.g., it’s not the same as non-realizability because there are many hypotheses that are not self-referential of which we’re unaware/coarsely aware.)
4. Why existing approaches to cause prioritization are not robust to unawareness
3. Why impartial altruists should suspend judgment under unawareness
As a followup: Hopefully this post of mine further clarifies my position, specifically the “Unawareness and superforecasting” section.
2. Why intuitive comparisons of large-scale impact are unjustified
Thanks!
People use “cluelessness” to mean various importantly different things, which is why I de-emphasized that term in this sequence. I think unawareness is a (major) source of what Greaves called complex cluelessness, which is a situation where:(CC1) We have some reasons to think that the unforeseeable consequences of A1 would
systematically tend to be substantially better than those of A2;
(CC2) We have some reasons to think that the unforeseeable consequences of A2 would
systematically tend to be substantially better than those of A1;
(CC3) It is unclear how to weigh up these reasons against one another.(It’s a bit unclear how “unforeseeable” is defined. In context / in the usual ways people tend to talk about complex cluelessness, I think it’s meant to encompass cases where the problem isn’t unawareness but rather other obstacles to setting precise credences.)
But unawareness itself means “many possible consequences of our actions haven’t even occurred to us in much detail, if at all” (as unpacked in the introduction section). ETA: I think it’s important to conceptually separate this from complex cluelessness, because you might think unawareness is a challenge that demands a response beyond straightforward Bayesianism, even if you disagree that it implies complex cluelessness.
(Note: A mod moved the subsequent posts to drafts for this reason. I’ll repost them spaced out.)
1. The challenge of unawareness for impartial altruist action guidance: Introduction
“Messy” tasks vs. hard-to-verify tasks
(Followup to here.)
I’ve found LLMs pretty useful for giving feedback on writing, including writing a fairly complex philosophical piece. Recently I wondered, “Hm, is this evidence that LLMs’ capabilities can generalize well to hard-to-verify tasks? That would be an update for me (toward super-short timelines, for one).”
I haven’t thought deeply about this yet, but I think the answer is: no. We should disentangle the intuitive “messiness” of the task of giving writing advice, from how difficult success is to verify:
Yes, the function in my brain that maps “writing feedback I’m given” to “how impressed I am with the feedback” seems pretty messy. My evaluation function for writing feedback is more fuzzy than, say, a function that checks if some piece of code works. It’s genuinely cool that LLMs can learn the former, especially without fine-tuning on me in particular.
But it’s still easy to verify whether someone has a positive vs. negative reaction[1] to some writing feedback. The model doesn’t need to actually be good at (1) making the thing I’m writing more effective at its intended purposes — i.e., helping readers have more informed and sensible views on the topic — in order to be good at (2) making me think “yeah that suggestion seems reasonable.”
Presumably there’s some correlation between the two. Humans rely on this correlation when giving writing advice all the time. But the correlation could still be rather weak (and its sign rather context-dependent), and LLMs’ advice would look just as impressive to me.
Maybe this is obvious? My sense, though, is that these two things could easily get conflated.
- ^
“Reaction” is meant to capture all the stuff that makes someone consider some feedback “useful” immediately when (or not too long after) they read it. I’m not saying LLMs are trying to make me feel good about my writing in this context, though that’s probably true to some extent.
what would your preferred state of the timelines discourse be?
My main recommendation would be, “Don’t pin down probability distributions that are (significantly) more precise than seems justified.” I can’t give an exact set of guidelines for what constitutes “more precise than seems justified” (such is life as a bounded agent!). But to a first approximation:
Suppose I’m doing some modeling, and I find myself thinking, “Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…”
And suppose I can’t point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition[1] that AI progress is going fast enough that 2060 is unreasonable.
Then: I wouldn’t pick just one of 40 or 50 for the median, or just one number in between. I’d include them all.
I totally agree that we can’t pin down the parameters to high precision
I’m not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I’d worry that’s misleading.) My guess is that we’re not yet on the same page about what “pin down the parameters to high precision” means.
I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff
Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.
communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would
If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I’d think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.
- ^
I don’t want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it’s a compression of (i) lots of cached understanding I’ve gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of “2060 is way too far off” wouldn’t be a sufficient justification, I think.
I think once an AI is extremely good at AI R&D, lots of these skills will transfer to other domains, so it won’t have to be that much more capable to generalize to all domains, especially if trained in environments designed for teaching general skills.
This step, especially, really struck me as under-argued relative to how important it seems to be for the conclusion. This isn’t to pick on the authors of AI 2027 in particular. I’m generally confused as to why arguments for an (imminent) intelligence explosion don’t say more on this point, as far as I’ve read. (I’m reminded of this comic.) But I might well have missed something!
arguing against having probabilistic beliefs about events which are unprecedented
Sorry, I’m definitely not saying this. First, in the linked post (see here), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I’m not drawing a sharp line between “precedented” and “unprecedented.” My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops[1] and unusual complexity here, I don’t see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour’s discussion in Sec. 3 here.)
the level of precedentedness is continous
Right, and that’s consistent with my view. I’m saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.
forecasters have successfully done OK at predicting increasingly unprecedented events
As I note here, our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.
- ^
On the actual information of interest (i.e. information about AGI/ASI), that is, not just proxies like forecasting progress in weaker or narrower AI.
- ^
Despite (2), it’s important to try anyways
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
- ^
As the linked post explains, “high precision” here does not mean “the credible interval for the parameter is narrow”. It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.
- ^
Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices.
Strongly agree — cf. nostalgebraist’s posts making this point on the bio anchors and AI 2027 models. I have the sense this is a pretty fundamental epistemic crux between camps of people making predictions (or suspending judgment!) about AI takeoff.
It sounds like you’re viewing the goal of thinking about DT as: “Figure out your object-level intuitions about what to do in specific abstract problem structures. Then, when you encounter concrete problems, you can ask which abstract problem structure the concrete problems correspond to and then act accordingly.”
I think that approach has its place. But there’s at least another very important (IMO more important) goal of DT: “Figure out your meta-level intuitions about why you should do one thing vs. another, across different abstract problem structures.” (Basically figuring out our “non-pragmatic principles” as discussed here.) I don’t see how just asking Claude helps with that, if we don’t have evidence that Claude’s meta-level intuitions match ours. Our object-level verdicts would just get reinforced without probing their justification. Garbage in, garbage out.
often, we don’t think of commitment as literally closing off choice—e.g., it’s still a “choice” to keep a promise
For what it’s worth, if a “commitment” is of this form, I struggle to see what the motivation for paying in Parfit’s hitchhiker would be. The “you want to be the sort of person who pays” argument doesn’t do anything for me, because that’s answering a different question than “should you choose to pay [insofar as you have a ‘choice’]?” I worry there’s a motte-and-bailey between different notions of “commitment” going on. I’d be curious for your reactions to my thoughts on this here.
(Replying now bc of the “missed the point” reaction:) To be clear, my concern is that someone without more context might pattern-match the claim “Anthony thinks we shouldn’t have probabilistic beliefs” to “Anthony thinks we have full Knightian uncertainty about everything / doesn’t think we can say any A is more or less likely than any B”. From my experience having discussions about imprecision, conceptual rounding errors are super common, so I think this is a reasonable concern even if you personally find it obvious that “probabilistic” should be read as “using a precise probability distribution”.