ETA: The original version of this comment conflated “evolution” and “reproductive fitness”, I’ve updated it now (see also my reply to Ben Pace’s comment).
Realism about rationality is important to the theory of rationality (we should know what kind of theoretical object rationality is), but not so important for the question of whether we need to know about rationality.
MIRI in general and you in particular seem unusually (to me) confident that:
1. We can learn more than we already know about rationality of “ideal” agents (or perhaps arbitrary agents?).
2. This understanding will allow us to build AI systems that we understand better than the ones we build today.
3. We will be able to do this in time for it to affect real AI systems. (This could be either because it is unusually tractable and can be solved very quickly, or because timelines are very long.)
This is primarily based on what research you and MIRI do, some of MIRI’s strategy writing, writing like the Rocket Alignment problem and law thinking, and an assumption that you are choosing to do this research because you think it is an effective way to reduce AI risk (given your skills).
(Another possibility is that you think that building AI the way we do now is so incredibly doomed that even though the story outlined above is unlikely, you see no other path by which to reduce x-risk, which I suppose might be implied by your other comment here.)
My current best argument for this position is realism about rationality; in this world, it seems like truly understanding rationality would enable a whole host of both capability and safety improvements in AI systems, potentially directly leading to a design for AGI (which would also explain the info hazards policy). I’d be interested in an argument for the three points listed above without realism about rationality (I agree with 1, somewhat agree with 2, and don’t agree with 3).
If you don’t have realism about rationality, then I basically agree with this sentence, though I’d rephrase it:
MIRI-cluster is essentially saying “biologists should want to invent evolution. Look at all the similarities across different animals. Don’t you want to explain that?” Whereas the non-MIRI cluster is saying “biologists don’t need to know about evolution.”
(ETA: In my head I was replacing “evolution” with “reproductive fitness”; I don’t agree with the sentence as phrased, I would agree with it if you talked only about understanding reproductive fitness, rather than also including e.g. the theory of natural selection, genetics, etc. In the rest of your comment you were talking about reproductive fitness, I don’t know why you suddenly switched to evolution; it seems completely different from everything you were talking about before.)
To my knowledge, the theory of evolution (ETA: mathematical understanding of reproductive fitness) has not had nearly the same impact on our ability to make big things as (say) any theory of physics. The Rocket Alignment Problem explicitly makes an analogy to an invention that required a theory of gravitation / momentum etc. Even physics theories that talk about extreme situations can enable applications; e.g. GPS would not work without an understanding of relativity. In contrast, I struggle to name a way that evolution (ETA: insights based on reproductive fitness) affects an everyday person (ignoring irrelevant things like atheism-religion debates). There are lots of applications based on an understanding of DNA, but DNA is a “real” thing. (This would make me sympathetic to a claim that rationality research would give us useful intuitions that lead us to discover “real” things that would then be important, but I don’t think that’s the claim.) My underlying model is that when you talk about something so “real” that you can make extremely precise predictions about it, you can create towers of abstractions upon it, without worrying that they might leak. You can’t do this with “non-real” things.
So I’d rephrase the sentence as: (ETA: changed the sentence a bit to talk about fitness instead of evolution)
MIRI-cluster is essentially saying “biologists should want to understand reproductive fitness. Look at all the similarities across different animals. Don’t you want to explain that?” Whereas the non-MIRI cluster is saying “Yeah, it’s a fascinating question to understand what makes animals fit, but given that we want to understand how antidepressants work, it is a better strategy to directly study what happens when an animal takes an antidepressant.”
Which you could round off to “biologists don’t need to know about reproductive fitness”, in the sense that it is not the best use of their time.
ETA: I also have a model of you being less convinced by realism about rationality than others in the “MIRI crowd”; in particular, selection vs. control seems decidedly less “realist” than mesa-optimizers (which didn’t have to be “realist”, but was quite “realist” the way it was written, especially in its focus on search).
Huh? A lot of these points about evolution register to me as straightforwardly false. Understanding the theory of evolution moved us from “Why are there all these weird living things? Why do they exist? What is going on?” to “Each part of these organisms has been designed by a local hill-climbing process to maximise reproduction.” If I looked into it, I expect I’d find out that early medicine found it very helpful to understand how the system was built. This is like me handing you a massive amount of code that has a bunch of weird outputs and telling you to make it work better and more efficiently, and the same thing but where I tell you what company made the code, why they made it, and how they made it, and loads of examples of other pieces of code they made in this fashion.
If I knew how to operationalise it I would take a pretty strong bet that the theory of natural selection has been revolutionary in the history of medicine.
A lot of these points about evolution register to me as straightforwardly false.
I don’t know which particular points you mean. The only one that it sounds like you’re arguing against is
he theory of evolution has not had nearly the same impact on our ability to make big things [...] I struggle to name a way that evolution affects an everyday person
Were there others?
I would take a pretty strong bet that the theory of natural selection has been revolutionary in the history of medicine.
I think the mathematical theory of natural selection + the theory of DNA / genes were probably very influential in both medicine and biology, because they make very precise predictions and the real world is a very good fit for the models they propose. (That is, they are “real”, in the sense that “real” is meant in the OP.) I don’t think that an improved mathematical understanding of what makes particular animals more fit has had that much of an impact on anything.
Separately, I also think the general insight of “each part of these organisms has been designed by a local hill-climbing process to maximise reproduction” would not have been very influential in either medicine or biology, had it not been accompanied by the math (and assuming no one ever developed the math).
On reflection, my original comment was quite unclear about this, I’ll add a note to it to clarify.
I do still stand by the thing that I meant in my original comment, which is that to the extent that you think rationality is like reproductive fitness (the claim made in the OP that Abram seems to agree with), where it is a very complicated mess of a function that we don’t hope to capture in a simple equation; I don’t think that improved understanding of that sort of thing has made much of an impact on our ability to do “big things” (as a proxy, things that affect normal people).
Within evolution, the claim would be that there has not been much impact from gaining an improved mathematical understanding of the reproductive fitness of some organism, or the “reproductive fitness” of some meme for memetic evolution.
I think the mathematical theory of natural selection + the theory of DNA / genes were probably very influential in both medicine and biology, because they make very precise predictions and the real world is a very good fit for the models they propose. (That is, they are “real”, in the sense that “real” is meant in the OP.)
In contrast, I think the general insight of “each part of these organisms has been designed by a local hill-climbing process to maximise reproduction” would not have been very influential in either medicine or biology, had it not been accompanied by the math.
But surely you wouldn’t get the mathematics of natural selection without the general insight, and so I think the general insight deserves to get a bunch of the credit. And both the mathematics of natural selection and the general insight seem pretty tied up to the notion of ‘reproductive fitness’.
But surely you wouldn’t get the mathematics of natural selection without the general insight, and so I think the general insight deserves to get a bunch of the credit. And both the mathematics of natural selection and the general insight seem pretty tied up to the notion of ‘reproductive fitness’.
Here is my understanding of what Abram thinks:
Rationality is like “reproductive fitness”, in that it is hard to formalize and turn into hard math. Regardless of how much theoretical progress we make on understanding rationality, it is never going to turn into something that can make very precise, accurate predictions about real systems. Nonetheless, qualitative understanding of rationality, of the sort that can make rough predictions about real systems, is useful for AI safety.
Hopefully that makes it clear why I’m trying to imagine a counterfactual where the math was never developed.
It’s possible that I’m misunderstanding Abram and he actually thinks that we will be able to make precise, accurate predictions about real systems; but if that’s the case I think he in fact is “realist about rationality” and this post is in fact pointing at a crux between him and Richard (or him and me), though not as well as he would like.
(Another possibility is that you think that building AI the way we do now is so incredibly doomed that even though the story outlined above is unlikely, you see no other path by which to reduce x-risk, which I suppose might be implied by your other comment here.)
This seems like the closest fit, but my view has some commonalities with points 1-3 nonetheless.
(I agree with 1, somewhat agree with 2, and don’t agree with 3).
It sounds like our potential cruxes are closer to point 3 and to the question of how doomed current approaches are. Given that, do you still think rationality realism seems super relevant (to your attempted steelman of my view)?
My current best argument for this position is realism about rationality; in this world, it seems like truly understanding rationality would enable a whole host of both capability and safety improvements in AI systems, potentially directly leading to a design for AGI (which would also explain the info hazards policy).
I guess my position is something like this. I think it may be quite possible to make capabilities “blindly”—basically the processing-power heavy type of AI progress (applying enough tricks so you’re not literally recapitulating evolution, but you’re sorta in that direction on a spectrum). Or possibly that approach will hit a wall at some point. But in either case, better understanding would be essentially necessary for aligning systems with high confidence. But that same knowledge could potentially accelerate capabilities progress.
So I believe in some kind of knowledge to be had (ie, point #1).
Yeah, so, taking stock of the discussion again, it seems like:
There’s a thing-I-believe-which-is-kind-of-like-rationality-realism.
Points 1 and 2 together seem more in line with that thing than “rationality realism” as I understood it from the OP.
You already believe #1, and somewhat believe #2.
We are both pessimistic about #3, but I’m so pessimistic about doing things without #3 that I work under the assumption anyway (plus I think my comparative advantage is contributing to those worlds).
We probably do have some disagreement about something like “how real is rationality?”—but I continue to strongly suspect it isn’t that cruxy.
(ETA: In my head I was replacing “evolution” with “reproductive fitness”; I don’t agree with the sentence as phrased, I would agree with it if you talked only about understanding reproductive fitness, rather than also including e.g. the theory of natural selection, genetics, etc. In the rest of your comment you were talking about reproductive fitness, I don’t know why you suddenly switched to evolution; it seems completely different from everything you were talking about before.)
I checked whether I thought the analogy was right with “reproductive fitness” and decided that evolution was a better analogy for this specific point. In claiming that rationality is as real as reproductive fitness, I’m claiming that there’s a theory of evolution out there.
Sorry it resulted in a confusing mixed metaphor overall.
But, separately, I don’t get how you’re seeing reproductive fitness and evolution as having radically different realness, such that you wanted to systematically correct. I agree they’re separate questions, but in fact I see the realness of reproductive fitness as largely a matter of the realness of evolution—without the overarching theory, reproductive fitness functions would be a kind of irrelevant abstraction and therefore less real.
To my knowledge, the theory of evolution (ETA: mathematical understanding of reproductive fitness) has not had nearly the same impact on our ability to make big things as (say) any theory of physics. The Rocket Alignment Problem explicitly makes an analogy to an invention that required a theory of gravitation / momentum etc. Even physics theories that talk about extreme situations can enable applications; e.g. GPS would not work without an understanding of relativity. In contrast, I struggle to name a way that evolution(ETA: insights based on reproductive fitness) affects an everyday person (ignoring irrelevant things like atheism-religion debates). There are lots of applications based on an understanding of DNA, but DNA is a “real” thing. (This would make me sympathetic to a claim that rationality research would give us useful intuitions that lead us to discover “real” things that would then be important, but I don’t think that’s the claim.)
I think this is due more to stuff like the relevant timescale than the degree of real-ness. I agree real-ness is relevant, but it seems to me that the rest of biology is roughly as real as reproductive fitness (ie, it’s all very messy compared to physics) but has far more practical consequences (thinking of medicine). On the other side, astronomy is very real but has few industry applications. There are other aspects to point at, but one relevant factor is that evolution and astronomy study things on long timescales.
Reproductive fitness would become very relevant if we were sending out seed ships to terraform nearby planets over geological time periods, in the hope that our descendants might one day benefit. (Because we would be in for some surprises if we didn’t understand how organisms seeded on those planets would likely evolve.)
So—it seems to me—the question should not be whether an abstract theory of rationality is the sort of thing which on-outside-view has few or many economic consequences, but whether it seems like the sort of thing that applies to building intelligent machines in particular!
My underlying model is that when you talk about something so “real” that you can make extremely precise predictions about it, you can create towers of abstractions upon it, without worrying that they might leak. You can’t do this with “non-real” things.
Reproductive fitness does seem to me like the kind of abstraction you can build on, though. For example, the theory of kin selection is a significant theory built on top of it.
As for reaching high confidence, yeah, there needs to be a different model of how you reach high confidence.
The security mindset model of reaching high confidence is not that you have a model whose overall predictive accuracy is high enough, but rather that you have an argument for security which depends on few assumptions, each of which is individually very likely. E.G., in computer security you don’t usually need exact models of attackers, and a system which relies on those is less likely to be secure.
I think we disagree primarily on 2 (and also how doomy the default case is, but let’s set that aside).
In claiming that rationality is as real as reproductive fitness, I’m claiming that there’s a theory of evolution out there.
I think that’s a crux between you and me. I’m no longer sure if it’s a crux between you and Richard. (ETA: I shouldn’t call this a crux, I wouldn’t change my mind on whether MIRI work is on-the-margin more valuable if I changed my mind on this, but it would be a pretty significant update.)
Reproductive fitness does seem to me like the kind of abstraction you can build on, though. For example, the theory of kin selection is a significant theory built on top of it.
Yeah, I was ignoring that sort of stuff. I do think this post would be better without the evolutionary fitness example because of this confusion. I was imagining the “unreal rationality” world to be similar to what Daniel mentions below:
I think I was imagining an alternative world where useful theories of rationality could only be about as precise as theories of liberalism, or current theories about why England had an industrial revolution when it did, and no other country did instead.
But, separately, I don’t get how you’re seeing reproductive fitness and evolution as having radically different realness, such that you wanted to systematically correct. I agree they’re separate questions, but in fact I see the realness of reproductive fitness as largely a matter of the realness of evolution—without the overarching theory, reproductive fitness functions would be a kind of irrelevant abstraction and therefore less real.
Yeah, I’m going to try to give a different explanation that doesn’t involve “realness”.
When groups of humans try to build complicated stuff, they tend to do so using abstraction. The most complicated stuff is built on a tower of many abstractions, each sitting on top of lower-level abstractions. This is most evident (to me) in software development, where the abstraction hierarchy is staggeringly large, but it applies elsewhere, too: the low-level abstractions of mechanical engineering are “levers”, “gears”, “nails”, etc.
A pretty key requirement for abstractions to work is that they need to be as non-leaky as possible, so that you do not have to think about them as much. When I code in Python and I write “x + y”, I can assume that the result will be the sum of the two values, and this is basically always right. Notably, I don’t have to think about the machine code that deals with the fact that overflow might happen. When I write in C, I do have to think about overflow, but I don’t have to think about how to implement addition at the bitwise level. This becomes even more important at the group level, because communication is expensive, slow, and low-bandwidth relative to thought, and so you need non-leaky abstractions so that you don’t need to communicate all the caveats and intuitions that would accompany a leaky abstraction.
One way to operationalize this is that to be built on, an abstraction must give extremely precise (and accurate) predictions.
It’s fine if there’s some complicated input to the abstraction, as long as that input can be estimated well in practice. This is what I imagine is going on with evolution and reproductive fitness—if you can estimate reproductive fitness, then you can get very precise and accurate predictions, as with e.g. the Price equation that Daniel mentioned. (And you can estimate fitness, either by using things like the Price equation + real data, or by controlling the environment where you set up the conditions that make something reproductively fit.)
If a thing cannot provide extremely precise and accurate predictions, then I claim that humans mostly can’t build on top of it. We can use it to make intuitive arguments about things very directly related to it, but can’t generalize it to something more far-off. Some examples from these comment threads of what “inferences about directly related things” looks like:
current theories about why England had an industrial revolution when it did
[biology] has far more practical consequences (thinking of medicine)
understanding why overuse of antibiotics might weaken the effect of antibiotics [based on knowledge of evolution]
Note that in all of these examples, you can more or less explain the conclusion in terms of the thing it depends on. E.g. You can say “overuse of antibiotics might weaken the effect of antibiotics because the bacteria will evolve / be selected to be resistant to the antibiotic”.
In contrast, for abstractions like “logic gates”, “assembly language”, “levers”, etc, we have built things like rockets and search engines that certainly could not have been built without those abstractions, but nonetheless you’d be hard pressed to explain e.g. how a search engine works if you were only allowed to talk with abstractions at the level of logic gates. This is because the precision afforded by those abstractions allows us to build huge hierarchies of better abstractions.
So now I’d go back and state our crux as:
Is there a theory of rationality that is sufficiently precise to build hierarchies of abstraction?
I would guess not. It sounds like you would guess yes.
I think this is upstream of 2. When I say I somewhat agree with 2, I mean that you can probably get a theory of rationality that makes imprecise predictions, which allows you to say things about “directly relevant things”, which will probably let you say some interesting things about AI systems, just not very much. I’d expect that, to really affect ML systems, given how far away from regular ML research MIRI research is, you would need a theory that’s precise enough to build hierarchies with.
(I think I’d also expect that you need to directly use the results of the research to build an AI system, rather than using it to inform existing efforts to build AI.)
(You might wonder why I’m optimistic about conceptual ML safety work, which is also not precise enough to build hierarchies of abstraction. The basic reason is that ML safety is “directly relevant” to existing ML systems, and so you don’t need to build hierarchies of abstraction—just the first imprecise layer is plausibly enough. You can see this in the fact that there are already imprecise concepts that are directly talking about safety.)
The security mindset model of reaching high confidence is not that you have a model whose overall predictive accuracy is high enough, but rather that you have an argument for security which depends on few assumptions, each of which is individually very likely. E.G., in computer security you don’t usually need exact models of attackers, and a system which relies on those is less likely to be secure.
Your few assumptions need to talk about the system you actually build. On the model I’m outlining, it’s hard to state the assumptions for the system you actually build, and near-impossible to be very confident in those assumptions, because they are (at least) one level of hierarchy higher than the (assumed imprecise) theory of rationality.
ETA: I also have a model of you being less convinced by realism about rationality than others in the “MIRI crowd”; in particular, selection vs. control seems decidedly less “realist” than mesa-optimizers (which didn’t have to be “realist”, but was quite “realist” the way it was written, especially in its focus on search).
Just a quick reply to this part for now (but thanks for the extensive comment, I’ll try to get to it at some point).
It makes sense. My recent series on myopia also fits this theme. But I don’t get much* push-back on these things. Some others seem even less realist than I am. I see myself as trying to carefully deconstruct my notions of “agency” into component parts that are less fake. I guess I do feel confused why other people seem less interested in directly deconstructing agency the way I am. I feel somewhat like others kind of nod along to distinctions like selection vs control but then go back to using a unitary notion of “optimization”. (This applies to people at MIRI and also people outside MIRI.)
*The one person who has given me push-back is Scott.
My underlying model is that when you talk about something so “real” that you can make extremely precise predictions about it, you can create towers of abstractions upon it, without worrying that they might leak. You can’t do this with “non-real” things.
For what it’s worth, I think I disagree with this even when “non-real” means “as real as the theory of liberalism”. One example is companies—my understanding is that people have fake theories about how companies should be arranged, that these theories can be better or worse (and evaluated as so without looking at how their implementations turn out), that one can maybe learn these theories in business school, and that implementing them creates more valuable companies (at least in expectation). At the very least, my understanding is that providing management advice to companies in developing countries significantly raises their productivity, and found this study to support this half-baked memory.
(next paragraph is super political, but it’s important to my point)
I live in what I honestly, straightforwardly believe is the greatest country in the world (where greatness doesn’t exactly mean ‘moral goodness’ but does imply the ability to support moral goodness—think some combination of wealth and geo-strategic dominance), whose government was founded after a long series of discussions about how best to use the state to secure individual liberty. If I think about other wealthy countries, it seems to me that ones whose governments built upon this tradition of the interaction between liberty and governance are over-represented (e.g. Switzerland, Singapore, Hong Kong). The theory of liberalism wasn’t complete or real enough to build a perfect government, or even a government reliable enough to keep to its founding principles (see complaints American constitutionalists have about how things are done today), but it was something that can be built upon.
At any rate, I think it’s the case that the things that can be built off of these fake theories aren’t reliable enough to satisfy a strict Yudkowsky-style security mindset. But I do think it’s possible to productively build off of them.
On the model proposed in this comment, I think of these as examples of using things / abstractions / theories with imprecise predictions to reason about things that are “directly relevant”.
If I agreed with the political example (and while I wouldn’t say that myself, it’s within the realm of plausibility), I’d consider that a particularly impressive version of this.
I’m confused how my examples don’t count as ‘building on’ the relevant theories—it sure seems like people reasoned in the relevant theories and then built things in the real world based on the results of that reasoning, and if that’s true (and if the things in the real world actually successfully fulfilled their purpose), then I’d think that spending time and effort developing the relevant theories was worth it. This argument has some weak points (the US government is not highly reliable at preserving liberty, very few individual businesses are highly reliable at delivering their products, the theories of management and liberalism were informed by a lot of experimentation), but you seem to be pointing at something else.
people reasoned in the relevant theories and then built things in the real world based on the results of that reasoning
Agreed. I’d say they built things in the real world that were “one level above” their theories.
if that’s true, [...] then I’d think that spending time and effort developing the relevant theories was worth it
Agreed.
you seem to be pointing at something else
Agreed.
Overall I think these relatively-imprecise theories let you build things “one level above”, which I think your examples fit into. My claim is that it’s very hard to use them to build things “2+ levels above”.
Separately, I claim that:
“real AGI systems” are “2+ levels above” the sorts of theories that MIRI works on.
MIRI’s theories will always be the relatively-imprecise theories that can’t scale to “2+ levels above”.
(All of this with weak confidence.)
I think you disagree with the underlying model, but assuming you granted that, you would disagree with the second claim; I don’t know what you’d think of the first.
Overall I think these relatively-imprecise theories let you build things “one level above”, which I think your examples fit into. My claim is that it’s very hard to use them to build things “2+ levels above”.
I think that I sort of agree if ‘levels above’ means levels of abstraction, where one system uses an abstraction of another and requires the mesa-system to satisfy some properties. In this case, the more layers of abstraction you have, the more requirements you’re demanding which can independently break, which exponentially reduces the chance that you’ll have no failure.
But also, to the extent that your theory is mathematisable and comes with ‘error bars’, you have a shot at coming up with a theory of abstractions that is robust to failure of your base-level theory. So some transistors on my computer can fail, evidencing the imprecision of the simple theory of logic gates, but my computer can still work fine because the abstractions on top of logic gates accounted for some amount of failure of logic gates. Similarly, even if you have some uncorrelated failures of individual economic rationality, you can still potentially have a pretty good model of a market. I’d say that the lesson is that the more levels of abstraction you have to go up, the more difficult it is to make each level robust to failures of the previous level, and as such the more you’d prefer the initial levels be ‘exact’.
“real AGI systems” are “2+ levels above” the sorts of theories that MIRI works on.
I’d say that they’re some number of levels above (of abstraction) and also levels below (of implementation). So for an unrealistic example, if you develop logical induction decision theory, you have your theory of logical induction, then you depend on that theory to have your decision theory (first level of abstraction), and then you depend on your decision theory to have multiple LIDT agents behave well together (second level of abstraction). Separately, you need to actually implement your logical inductor by some machine learning algorithm (first level of implementation), which is going to depend on numpy and floating point arithmetic and such (second and third (?) levels of implementation), which depends on computing hardware and firmware (I don’t know how many levels of implementation that is).
When I read a MIRI paper, it typically seems to me that the theories discussed are pretty abstract, and as such there are more levels below than above. The levels below seem mostly unproblematic (except for machine learning, which in the form of deep learning is often under-theorised). They are also mathematised enough that I’m optimistic about upwards abstraction having the possibility of robustness. There are some exceptions (e.g. the mesa-optimisers paper), but they seem like they’re on the path to greater mathematisability.
MIRI’s theories will always be the relatively-imprecise theories that can’t scale to “2+ levels above”
I’m not sure about this, but I disagree with the version that replaces ‘MIRI’s theories’ with ‘mathematical theories of embedded rationality’, basically for the reasons that Vanessa discusses.
I disagree with the version that replaces ‘MIRI’s theories’ with ‘mathematical theories of embedded rationality’
Yeah, I think this is the sense in which realism about rationality is an important disagreement.
But also, to the extent that your theory is mathematisable and comes with ‘error bars’
Yeah, I agree that this would make it easier to build multiple levels of abstractions “on top”. I also would be surprised if mathematical theories of embedded rationality came with tight error bounds (where “tight” means “not so wide as to be useless”). For example, current theories of generalization in deep learning do not provide tight error bounds to my knowledge, except in special cases that don’t apply to the main successes of deep learning.
When I read a MIRI paper, it typically seems to me that the theories discussed are pretty abstract, and as such there are more levels below than above. [...] They are also mathematised enough that I’m optimistic about upwards abstraction having the possibility of robustness.
Agreed.
The levels below seem mostly unproblematic (except for machine learning, which in the form of deep learning is often under-theorised).
I am basically only concerned about machine learning, when I say that you can’t build on the theories. My understanding of MIRI’s mainline story of impact is that they develop some theory that AI researchers use to change the way they do machine learning that leads to safe AI. This sounds to me like there are multiple levels of inference: “MIRI’s theory” → “machine learning” → “AGI”. This isn’t exactly layers of abstraction, but I think the same principle applies, and this seems like too many layers.
You could imagine other stories of impact, and I’d have other questions about those, e.g. if the story was “MIRI’s theory will tell us how to build aligned AGI without machine learning”, I’d be asking when the theory was going to include computational complexity.
In contrast, I struggle to name a way that evolution affects an everyday person
I’m not sure what exactly you mean, but examples that come to mind:
Crops and domestic animals that have been artificially selected for various qualities.
The medical community encouraging people to not use antibiotics unnecessarily.
[Inheritance but not selection] The fact that your kids will probably turn out like you without specific intervention on your part to make that happen.
Crops and domestic animals that have been artificially selected for various qualities.
I feel fairly confident this was done before we understood evolution.
The fact that your kids will probably turn out like you without specific intervention on your part to make that happen.
Also seems like a thing we knew before we understood evolution.
The medical community encouraging people to not use antibiotics unnecessarily.
That one seems plausible; though I’d want to know more about the history of how this came up. It also seems like the sort of thing that we’d have figured out even if we didn’t understand evolution, though it would have taken longer, and would have involved more deaths.
Going back to the AI case, my takeaway from this example is that understanding non-real things can still help if you need to get everything right the first time. And in fact, I do think that if you posit a discontinuity, such that we have to get everything right before that discontinuity, then the non-MIRI strategy looks worse because you can’t gather as much empirical evidence (though I still wouldn’t be convinced that the MIRI strategy is the right one).
Ah, I didn’t quite realise you meant to talk about “human understanding of the theory of evolution” rather than evolution itself. I still suspect that the theory of evolution is so fundamental to our understanding of biology, and our understanding of biology so useful to humanity, that if human understanding of evolution doesn’t contribute much to human welfare it’s just because most applications deal with pretty long time-scales.
(Also I don’t get why this discussion is treating evolution as ‘non-real’: stuff like the Price equation seems pretty formal to me. To me it seems like a pretty mathematisable theory with some hard-to-specify inputs like fitness.)
(Also I don’t get why this discussion is treating evolution as ‘non-real’: stuff like the Price equation seems pretty formal to me. To me it seems like a pretty mathematisable theory with some hard-to-specify inputs like fitness.)
Yeah, I agree, see my edits to the original comment and also my reply to Ben. Abram’s comment was talking about reproductive fitness the entire time and then suddenly switched to evolution at the end; I didn’t notice this and kept thinking of evolution as reproductive fitness in my head, and then wrote a comment based on that where I used the word evolution despite thinking about reproductive fitness and the general idea of “there is a local hill-climbing search on reproductive fitness” while ignoring the hard math.
How does evolutionary psychology help us during our everyday life? We already know that people like having sex and that they execute all these sorts of weird social behaviors. Why does providing the ultimate explanation for our behavior provide more than a satisfaction of our curiosity?
ETA: The original version of this comment conflated “evolution” and “reproductive fitness”, I’ve updated it now (see also my reply to Ben Pace’s comment).
MIRI in general and you in particular seem unusually (to me) confident that:
1. We can learn more than we already know about rationality of “ideal” agents (or perhaps arbitrary agents?).
2. This understanding will allow us to build AI systems that we understand better than the ones we build today.
3. We will be able to do this in time for it to affect real AI systems. (This could be either because it is unusually tractable and can be solved very quickly, or because timelines are very long.)
This is primarily based on what research you and MIRI do, some of MIRI’s strategy writing, writing like the Rocket Alignment problem and law thinking, and an assumption that you are choosing to do this research because you think it is an effective way to reduce AI risk (given your skills).
(Another possibility is that you think that building AI the way we do now is so incredibly doomed that even though the story outlined above is unlikely, you see no other path by which to reduce x-risk, which I suppose might be implied by your other comment here.)
My current best argument for this position is realism about rationality; in this world, it seems like truly understanding rationality would enable a whole host of both capability and safety improvements in AI systems, potentially directly leading to a design for AGI (which would also explain the info hazards policy). I’d be interested in an argument for the three points listed above without realism about rationality (I agree with 1, somewhat agree with 2, and don’t agree with 3).
If you don’t have realism about rationality, then I basically agree with this sentence, though I’d rephrase it:
(ETA: In my head I was replacing “evolution” with “reproductive fitness”; I don’t agree with the sentence as phrased, I would agree with it if you talked only about understanding reproductive fitness, rather than also including e.g. the theory of natural selection, genetics, etc. In the rest of your comment you were talking about reproductive fitness, I don’t know why you suddenly switched to evolution; it seems completely different from everything you were talking about before.)
To my knowledge,
the theory of evolution(ETA: mathematical understanding of reproductive fitness) has not had nearly the same impact on our ability to make big things as (say) any theory of physics. The Rocket Alignment Problem explicitly makes an analogy to an invention that required a theory of gravitation / momentum etc. Even physics theories that talk about extreme situations can enable applications; e.g. GPS would not work without an understanding of relativity. In contrast, I struggle to name a way thatevolution(ETA: insights based on reproductive fitness) affects an everyday person (ignoring irrelevant things like atheism-religion debates). There are lots of applications based on an understanding of DNA, but DNA is a “real” thing. (This would make me sympathetic to a claim that rationality research would give us useful intuitions that lead us to discover “real” things that would then be important, but I don’t think that’s the claim.) My underlying model is that when you talk about something so “real” that you can make extremely precise predictions about it, you can create towers of abstractions upon it, without worrying that they might leak. You can’t do this with “non-real” things.So I’d rephrase the sentence as: (ETA: changed the sentence a bit to talk about fitness instead of evolution)
Which you could round off to “biologists don’t need to know about reproductive fitness”, in the sense that it is not the best use of their time.
ETA: I also have a model of you being less convinced by realism about rationality than others in the “MIRI crowd”; in particular, selection vs. control seems decidedly less “realist” than mesa-optimizers (which didn’t have to be “realist”, but was quite “realist” the way it was written, especially in its focus on search).
Huh? A lot of these points about evolution register to me as straightforwardly false. Understanding the theory of evolution moved us from “Why are there all these weird living things? Why do they exist? What is going on?” to “Each part of these organisms has been designed by a local hill-climbing process to maximise reproduction.” If I looked into it, I expect I’d find out that early medicine found it very helpful to understand how the system was built. This is like me handing you a massive amount of code that has a bunch of weird outputs and telling you to make it work better and more efficiently, and the same thing but where I tell you what company made the code, why they made it, and how they made it, and loads of examples of other pieces of code they made in this fashion.
If I knew how to operationalise it I would take a pretty strong bet that the theory of natural selection has been revolutionary in the history of medicine.
I don’t know which particular points you mean. The only one that it sounds like you’re arguing against is
Were there others?
I think the mathematical theory of natural selection + the theory of DNA / genes were probably very influential in both medicine and biology, because they make very precise predictions and the real world is a very good fit for the models they propose. (That is, they are “real”, in the sense that “real” is meant in the OP.) I don’t think that an improved mathematical understanding of what makes particular animals more fit has had that much of an impact on anything.
Separately, I also think the general insight of “each part of these organisms has been designed by a local hill-climbing process to maximise reproduction” would not have been very influential in either medicine or biology, had it not been accompanied by the math (and assuming no one ever developed the math).
On reflection, my original comment was quite unclear about this, I’ll add a note to it to clarify.
I do still stand by the thing that I meant in my original comment, which is that to the extent that you think rationality is like reproductive fitness (the claim made in the OP that Abram seems to agree with), where it is a very complicated mess of a function that we don’t hope to capture in a simple equation; I don’t think that improved understanding of that sort of thing has made much of an impact on our ability to do “big things” (as a proxy, things that affect normal people).
Within evolution, the claim would be that there has not been much impact from gaining an improved mathematical understanding of the reproductive fitness of some organism, or the “reproductive fitness” of some meme for memetic evolution.
But surely you wouldn’t get the mathematics of natural selection without the general insight, and so I think the general insight deserves to get a bunch of the credit. And both the mathematics of natural selection and the general insight seem pretty tied up to the notion of ‘reproductive fitness’.
Here is my understanding of what Abram thinks:
Rationality is like “reproductive fitness”, in that it is hard to formalize and turn into hard math. Regardless of how much theoretical progress we make on understanding rationality, it is never going to turn into something that can make very precise, accurate predictions about real systems. Nonetheless, qualitative understanding of rationality, of the sort that can make rough predictions about real systems, is useful for AI safety.
Hopefully that makes it clear why I’m trying to imagine a counterfactual where the math was never developed.
It’s possible that I’m misunderstanding Abram and he actually thinks that we will be able to make precise, accurate predictions about real systems; but if that’s the case I think he in fact is “realist about rationality” and this post is in fact pointing at a crux between him and Richard (or him and me), though not as well as he would like.
This seems like the closest fit, but my view has some commonalities with points 1-3 nonetheless.
It sounds like our potential cruxes are closer to point 3 and to the question of how doomed current approaches are. Given that, do you still think rationality realism seems super relevant (to your attempted steelman of my view)?
I guess my position is something like this. I think it may be quite possible to make capabilities “blindly”—basically the processing-power heavy type of AI progress (applying enough tricks so you’re not literally recapitulating evolution, but you’re sorta in that direction on a spectrum). Or possibly that approach will hit a wall at some point. But in either case, better understanding would be essentially necessary for aligning systems with high confidence. But that same knowledge could potentially accelerate capabilities progress.
So I believe in some kind of knowledge to be had (ie, point #1).
Yeah, so, taking stock of the discussion again, it seems like:
There’s a thing-I-believe-which-is-kind-of-like-rationality-realism.
Points 1 and 2 together seem more in line with that thing than “rationality realism” as I understood it from the OP.
You already believe #1, and somewhat believe #2.
We are both pessimistic about #3, but I’m so pessimistic about doing things without #3 that I work under the assumption anyway (plus I think my comparative advantage is contributing to those worlds).
We probably do have some disagreement about something like “how real is rationality?”—but I continue to strongly suspect it isn’t that cruxy.
I checked whether I thought the analogy was right with “reproductive fitness” and decided that evolution was a better analogy for this specific point. In claiming that rationality is as real as reproductive fitness, I’m claiming that there’s a theory of evolution out there.
Sorry it resulted in a confusing mixed metaphor overall.
But, separately, I don’t get how you’re seeing reproductive fitness and evolution as having radically different realness, such that you wanted to systematically correct. I agree they’re separate questions, but in fact I see the realness of reproductive fitness as largely a matter of the realness of evolution—without the overarching theory, reproductive fitness functions would be a kind of irrelevant abstraction and therefore less real.
I think this is due more to stuff like the relevant timescale than the degree of real-ness. I agree real-ness is relevant, but it seems to me that the rest of biology is roughly as real as reproductive fitness (ie, it’s all very messy compared to physics) but has far more practical consequences (thinking of medicine). On the other side, astronomy is very real but has few industry applications. There are other aspects to point at, but one relevant factor is that evolution and astronomy study things on long timescales.
Reproductive fitness would become very relevant if we were sending out seed ships to terraform nearby planets over geological time periods, in the hope that our descendants might one day benefit. (Because we would be in for some surprises if we didn’t understand how organisms seeded on those planets would likely evolve.)
So—it seems to me—the question should not be whether an abstract theory of rationality is the sort of thing which on-outside-view has few or many economic consequences, but whether it seems like the sort of thing that applies to building intelligent machines in particular!
Reproductive fitness does seem to me like the kind of abstraction you can build on, though. For example, the theory of kin selection is a significant theory built on top of it.
As for reaching high confidence, yeah, there needs to be a different model of how you reach high confidence.
The security mindset model of reaching high confidence is not that you have a model whose overall predictive accuracy is high enough, but rather that you have an argument for security which depends on few assumptions, each of which is individually very likely. E.G., in computer security you don’t usually need exact models of attackers, and a system which relies on those is less likely to be secure.
I think we disagree primarily on 2 (and also how doomy the default case is, but let’s set that aside).
I think that’s a crux between you and me. I’m no longer sure if it’s a crux between you and Richard. (ETA: I shouldn’t call this a crux, I wouldn’t change my mind on whether MIRI work is on-the-margin more valuable if I changed my mind on this, but it would be a pretty significant update.)
Yeah, I was ignoring that sort of stuff. I do think this post would be better without the evolutionary fitness example because of this confusion. I was imagining the “unreal rationality” world to be similar to what Daniel mentions below:
Yeah, I’m going to try to give a different explanation that doesn’t involve “realness”.
When groups of humans try to build complicated stuff, they tend to do so using abstraction. The most complicated stuff is built on a tower of many abstractions, each sitting on top of lower-level abstractions. This is most evident (to me) in software development, where the abstraction hierarchy is staggeringly large, but it applies elsewhere, too: the low-level abstractions of mechanical engineering are “levers”, “gears”, “nails”, etc.
A pretty key requirement for abstractions to work is that they need to be as non-leaky as possible, so that you do not have to think about them as much. When I code in Python and I write “x + y”, I can assume that the result will be the sum of the two values, and this is basically always right. Notably, I don’t have to think about the machine code that deals with the fact that overflow might happen. When I write in C, I do have to think about overflow, but I don’t have to think about how to implement addition at the bitwise level. This becomes even more important at the group level, because communication is expensive, slow, and low-bandwidth relative to thought, and so you need non-leaky abstractions so that you don’t need to communicate all the caveats and intuitions that would accompany a leaky abstraction.
One way to operationalize this is that to be built on, an abstraction must give extremely precise (and accurate) predictions.
It’s fine if there’s some complicated input to the abstraction, as long as that input can be estimated well in practice. This is what I imagine is going on with evolution and reproductive fitness—if you can estimate reproductive fitness, then you can get very precise and accurate predictions, as with e.g. the Price equation that Daniel mentioned. (And you can estimate fitness, either by using things like the Price equation + real data, or by controlling the environment where you set up the conditions that make something reproductively fit.)
If a thing cannot provide extremely precise and accurate predictions, then I claim that humans mostly can’t build on top of it. We can use it to make intuitive arguments about things very directly related to it, but can’t generalize it to something more far-off. Some examples from these comment threads of what “inferences about directly related things” looks like:
Note that in all of these examples, you can more or less explain the conclusion in terms of the thing it depends on. E.g. You can say “overuse of antibiotics might weaken the effect of antibiotics because the bacteria will evolve / be selected to be resistant to the antibiotic”.
In contrast, for abstractions like “logic gates”, “assembly language”, “levers”, etc, we have built things like rockets and search engines that certainly could not have been built without those abstractions, but nonetheless you’d be hard pressed to explain e.g. how a search engine works if you were only allowed to talk with abstractions at the level of logic gates. This is because the precision afforded by those abstractions allows us to build huge hierarchies of better abstractions.
So now I’d go back and state our crux as:
I would guess not. It sounds like you would guess yes.
I think this is upstream of 2. When I say I somewhat agree with 2, I mean that you can probably get a theory of rationality that makes imprecise predictions, which allows you to say things about “directly relevant things”, which will probably let you say some interesting things about AI systems, just not very much. I’d expect that, to really affect ML systems, given how far away from regular ML research MIRI research is, you would need a theory that’s precise enough to build hierarchies with.
(I think I’d also expect that you need to directly use the results of the research to build an AI system, rather than using it to inform existing efforts to build AI.)
(You might wonder why I’m optimistic about conceptual ML safety work, which is also not precise enough to build hierarchies of abstraction. The basic reason is that ML safety is “directly relevant” to existing ML systems, and so you don’t need to build hierarchies of abstraction—just the first imprecise layer is plausibly enough. You can see this in the fact that there are already imprecise concepts that are directly talking about safety.)
Your few assumptions need to talk about the system you actually build. On the model I’m outlining, it’s hard to state the assumptions for the system you actually build, and near-impossible to be very confident in those assumptions, because they are (at least) one level of hierarchy higher than the (assumed imprecise) theory of rationality.
I generally like the re-framing here, and agree with the proposed crux.
I may try to reply more at the object level later.
Abram, did you reply to that crux somewhere?
Just a quick reply to this part for now (but thanks for the extensive comment, I’ll try to get to it at some point).
It makes sense. My recent series on myopia also fits this theme. But I don’t get much* push-back on these things. Some others seem even less realist than I am. I see myself as trying to carefully deconstruct my notions of “agency” into component parts that are less fake. I guess I do feel confused why other people seem less interested in directly deconstructing agency the way I am. I feel somewhat like others kind of nod along to distinctions like selection vs control but then go back to using a unitary notion of “optimization”. (This applies to people at MIRI and also people outside MIRI.)
*The one person who has given me push-back is Scott.
For what it’s worth, I think I disagree with this even when “non-real” means “as real as the theory of liberalism”. One example is companies—my understanding is that people have fake theories about how companies should be arranged, that these theories can be better or worse (and evaluated as so without looking at how their implementations turn out), that one can maybe learn these theories in business school, and that implementing them creates more valuable companies (at least in expectation). At the very least, my understanding is that providing management advice to companies in developing countries significantly raises their productivity, and found this study to support this half-baked memory.
(next paragraph is super political, but it’s important to my point)
I live in what I honestly, straightforwardly believe is the greatest country in the world (where greatness doesn’t exactly mean ‘moral goodness’ but does imply the ability to support moral goodness—think some combination of wealth and geo-strategic dominance), whose government was founded after a long series of discussions about how best to use the state to secure individual liberty. If I think about other wealthy countries, it seems to me that ones whose governments built upon this tradition of the interaction between liberty and governance are over-represented (e.g. Switzerland, Singapore, Hong Kong). The theory of liberalism wasn’t complete or real enough to build a perfect government, or even a government reliable enough to keep to its founding principles (see complaints American constitutionalists have about how things are done today), but it was something that can be built upon.
At any rate, I think it’s the case that the things that can be built off of these fake theories aren’t reliable enough to satisfy a strict Yudkowsky-style security mindset. But I do think it’s possible to productively build off of them.
On the model proposed in this comment, I think of these as examples of using things / abstractions / theories with imprecise predictions to reason about things that are “directly relevant”.
If I agreed with the political example (and while I wouldn’t say that myself, it’s within the realm of plausibility), I’d consider that a particularly impressive version of this.
I’m confused how my examples don’t count as ‘building on’ the relevant theories—it sure seems like people reasoned in the relevant theories and then built things in the real world based on the results of that reasoning, and if that’s true (and if the things in the real world actually successfully fulfilled their purpose), then I’d think that spending time and effort developing the relevant theories was worth it. This argument has some weak points (the US government is not highly reliable at preserving liberty, very few individual businesses are highly reliable at delivering their products, the theories of management and liberalism were informed by a lot of experimentation), but you seem to be pointing at something else.
Agreed. I’d say they built things in the real world that were “one level above” their theories.
Agreed.
Agreed.
Overall I think these relatively-imprecise theories let you build things “one level above”, which I think your examples fit into. My claim is that it’s very hard to use them to build things “2+ levels above”.
Separately, I claim that:
“real AGI systems” are “2+ levels above” the sorts of theories that MIRI works on.
MIRI’s theories will always be the relatively-imprecise theories that can’t scale to “2+ levels above”.
(All of this with weak confidence.)
I think you disagree with the underlying model, but assuming you granted that, you would disagree with the second claim; I don’t know what you’d think of the first.
OK, I think I understand you now.
I think that I sort of agree if ‘levels above’ means levels of abstraction, where one system uses an abstraction of another and requires the mesa-system to satisfy some properties. In this case, the more layers of abstraction you have, the more requirements you’re demanding which can independently break, which exponentially reduces the chance that you’ll have no failure.
But also, to the extent that your theory is mathematisable and comes with ‘error bars’, you have a shot at coming up with a theory of abstractions that is robust to failure of your base-level theory. So some transistors on my computer can fail, evidencing the imprecision of the simple theory of logic gates, but my computer can still work fine because the abstractions on top of logic gates accounted for some amount of failure of logic gates. Similarly, even if you have some uncorrelated failures of individual economic rationality, you can still potentially have a pretty good model of a market. I’d say that the lesson is that the more levels of abstraction you have to go up, the more difficult it is to make each level robust to failures of the previous level, and as such the more you’d prefer the initial levels be ‘exact’.
I’d say that they’re some number of levels above (of abstraction) and also levels below (of implementation). So for an unrealistic example, if you develop logical induction decision theory, you have your theory of logical induction, then you depend on that theory to have your decision theory (first level of abstraction), and then you depend on your decision theory to have multiple LIDT agents behave well together (second level of abstraction). Separately, you need to actually implement your logical inductor by some machine learning algorithm (first level of implementation), which is going to depend on numpy and floating point arithmetic and such (second and third (?) levels of implementation), which depends on computing hardware and firmware (I don’t know how many levels of implementation that is).
When I read a MIRI paper, it typically seems to me that the theories discussed are pretty abstract, and as such there are more levels below than above. The levels below seem mostly unproblematic (except for machine learning, which in the form of deep learning is often under-theorised). They are also mathematised enough that I’m optimistic about upwards abstraction having the possibility of robustness. There are some exceptions (e.g. the mesa-optimisers paper), but they seem like they’re on the path to greater mathematisability.
I’m not sure about this, but I disagree with the version that replaces ‘MIRI’s theories’ with ‘mathematical theories of embedded rationality’, basically for the reasons that Vanessa discusses.
Yeah, I think this is the sense in which realism about rationality is an important disagreement.
Yeah, I agree that this would make it easier to build multiple levels of abstractions “on top”. I also would be surprised if mathematical theories of embedded rationality came with tight error bounds (where “tight” means “not so wide as to be useless”). For example, current theories of generalization in deep learning do not provide tight error bounds to my knowledge, except in special cases that don’t apply to the main successes of deep learning.
Agreed.
I am basically only concerned about machine learning, when I say that you can’t build on the theories. My understanding of MIRI’s mainline story of impact is that they develop some theory that AI researchers use to change the way they do machine learning that leads to safe AI. This sounds to me like there are multiple levels of inference: “MIRI’s theory” → “machine learning” → “AGI”. This isn’t exactly layers of abstraction, but I think the same principle applies, and this seems like too many layers.
You could imagine other stories of impact, and I’d have other questions about those, e.g. if the story was “MIRI’s theory will tell us how to build aligned AGI without machine learning”, I’d be asking when the theory was going to include computational complexity.
I’m not sure what exactly you mean, but examples that come to mind:
Crops and domestic animals that have been artificially selected for various qualities.
The medical community encouraging people to not use antibiotics unnecessarily.
[Inheritance but not selection] The fact that your kids will probably turn out like you without specific intervention on your part to make that happen.
I feel fairly confident this was done before we understood evolution.
Also seems like a thing we knew before we understood evolution.
That one seems plausible; though I’d want to know more about the history of how this came up. It also seems like the sort of thing that we’d have figured out even if we didn’t understand evolution, though it would have taken longer, and would have involved more deaths.
Going back to the AI case, my takeaway from this example is that understanding non-real things can still help if you need to get everything right the first time. And in fact, I do think that if you posit a discontinuity, such that we have to get everything right before that discontinuity, then the non-MIRI strategy looks worse because you can’t gather as much empirical evidence (though I still wouldn’t be convinced that the MIRI strategy is the right one).
Ah, I didn’t quite realise you meant to talk about “human understanding of the theory of evolution” rather than evolution itself. I still suspect that the theory of evolution is so fundamental to our understanding of biology, and our understanding of biology so useful to humanity, that if human understanding of evolution doesn’t contribute much to human welfare it’s just because most applications deal with pretty long time-scales.
(Also I don’t get why this discussion is treating evolution as ‘non-real’: stuff like the Price equation seems pretty formal to me. To me it seems like a pretty mathematisable theory with some hard-to-specify inputs like fitness.)
Yeah, I agree, see my edits to the original comment and also my reply to Ben. Abram’s comment was talking about reproductive fitness the entire time and then suddenly switched to evolution at the end; I didn’t notice this and kept thinking of evolution as reproductive fitness in my head, and then wrote a comment based on that where I used the word evolution despite thinking about reproductive fitness and the general idea of “there is a local hill-climbing search on reproductive fitness” while ignoring the hard math.
The most obvious thing is understanding why overuse of antibiotics might weaken the effect of antibiotics.
See response to Daniel below; I find this one a little compelling (but not that much).
Evolutionary psychology?
How does evolutionary psychology help us during our everyday life? We already know that people like having sex and that they execute all these sorts of weird social behaviors. Why does providing the ultimate explanation for our behavior provide more than a satisfaction of our curiosity?
+1, it seems like some people with direct knowledge of evolutionary psychology get something out of it, but not everyone else.
Sorry, how is this not saying “people who don’t know evo-psych don’t get anything out of knowing evo-psych”?