Communications @ MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s. (Though we agree about an awful lot.)
Rob Bensinger
Karma: 22,111
The problem was that you first seemed to belittle questions about word meanings (“self”) as being “just” about “definitions” that are “purely verbal”.
I did no such thing!
Luckily now you concede that the question about the meaning of “I” isn’t just about (arbitrary) “definitions”
Read the blog post at the top of this page! It’s my attempt to answer the question of when a mind is “me”, and you’ll notice it’s not talking about definitions.
But we already know all the empirical facts: Someone goes into the teleporter, a bit later someone comes out at the other end and sees something. So the issue can only be about the semantic interpretation of that question, about what we mean with expressions like “I will see x”.
Nope!
There are two perspectives here:
“I don’t want to upload myself, because I wouldn’t get to experience that uploads’ experiences. When I die, this stream of consciousness will end, rather than continuing in another body. Physically dying and then being being copied elsewhere is not phenomenologically indistinguishable from stepping through a doorway.”
“I do want to upload myself, because I would get to experience that uploads’ experiences. Physically dying and then being copied myself is phenomenologically indistinguishable from stepping through a doorway.”
The disagreement between these two perspectives isn’t about word definitions at all; a fear that “when my body dies, there will be nothing but oblivion” is a very real fear about anticipated experiences (and anticipated absences of experience), not a verbal quibble about how we ought to define a specific word.
But it’s also a bit confusing to call the disagreement between these two perspectives “empirical”, because “empirical” here is conflating “third-person empirical” with “first-person empirical”.
The disagreement here is about whether a stream of consciousness can “continue” across temporal and spatial gaps, in the same way that it continues when there are no obvious gaps. It’s about whether there’s a subjective, experiential difference between stepping through a doorway and using a teleporter.
The thing I’m arguing in the OP is that there can’t be an experiential difference here, because there’s no physical difference that could be underlying the supposed experiential difference. So the disagreement about the first-person facts, I claim, stems from a cognitive error, which I characterize as “making predictions as though you believed yourself to be a Cartesian Ghost (even if you don’t on-reflection endorse the claim that Cartesian Ghosts exist)”. This is, again, a very different error from “defining a word in a nonstandard way”.
You’re also free to define “I” however you want in your values.
Sort of!
It’s true that no law of nature will stop you from using “I” in a nonstandard way; your head will not explode if you redefine “table” to mean “penguin”.
And it’s true that there are possible minds in abstract mindspace that have all sorts of values, including strict preferences about whether they want their brain to be made of silicon vs. carbon.
But it’s not true that humans alive today have full and complete control over their own preferences.
And it’s not true that humans can never be mistaken in their beliefs about their own preferences.
In the case of teleportation, I think teleportation-phobic people are mostly making an implicit error of the form “mistakenly modeling situations as though you are a Cartesian Ghost who is observing experiences from outside the universe”, not making a mistake about what their preferences are per se. (Though once you realize that you’re not a Cartesian Ghost, that will have some implications for what experiences you expect to see next in some cases, and implications for what physical world-states you prefer relative to other world-states.)
FWIW, I typically use “alignment research” to mean “AI research aimed at making it possible to safely do ambitious things with sufficiently-capable AI” (with an emphasis on “safely”). So I’d include things like Chris Olah’s interpretability research, even if the proximate impact of this is just “we understand what’s going on better, so we may be more able to predict and finely control future systems” and the proximate impact is not “the AI is now less inclined to kill you”.
Some examples: I wouldn’t necessarily think of “figure out how we want to airgap the AI” as “alignment research”, since it’s less about designing the AI, shaping its mind, predicting and controlling it, etc., and more about designing the environment around the AI.
But I would think of things like “figure out how to make this AI system too socially-dumb to come up with ideas like ‘maybe I should deceive my operators’, while keeping it superhumanly smart at nanotech research” as central examples of “alignment research”, even though it’s about controlling capabilities (‘make the AI dumb in this particular way’) rather than about instilling a particular goal into the AI.And I’d also think of “we know this AI is trying to kill us; let’s figure out how to constrain its capabilities so that it keeps wanting that, but is too dumb to find a way to succeed in killing us, thereby forcing it to work with us rather than against us in order to achieve more of what it wants” as a pretty central example of alignment research, albeit not the sort of alignment research I feel optimistic about. The way I think about the field, you don’t have to specifically attack the “is it trying to kill you?” part of the system in order to be doing alignment research; there are other paths, and alignment researchers should consider all of them and focus on results rather than marrying a specific methodology.
But that isn’t an experience. It’s two experiences. You will not have an experience of having two experiences. Two experiences will experience having been one person.
Sure; from my perspective, you’re saying the same thing as me.
Are you going to care about 1000 different copies equally?
How am I supposed to choose between them?
Why? If “I” is arbitrary definition, then “When I step through this doorway, will I have another experience?” depends on this arbitrary definition and so is also arbitrary.
Which things count as “I” isn’t an arbitrary definition; it’s just a fuzzy natural-language concept.
(I guess you can call that “arbitrary” if you want, but then all the other words in the sentence, like “doorway” and “step”, are also “arbitrary”.)
Analogy: When you’re writing in your personal diary, you’re free to define “table” however you want. But in ordinary English-language discourse, if you call all penguins “tables” you’ll just be wrong. And this fact isn’t changed at all by the fact that “table” lacks a perfectly formal physics-level definition.
The same holds for “Will Rob Bensinger’s next experience be of sitting in his bedroom writing a LessWrong comment, or will it be of him grabbing some tomatoes in a supermarket in Beijing?”
Terms like ‘Rob Bensinger’ and ‘I’ aren’t perfectly physically crisp — there may be cases where the answer is “ehh, maybe?” rather than a clear yes or no. And if we live in a Big Universe and we allow that there can be many Beijings out there in space, then we’ll have to give a more nuanced quantitative answer, like “a lot more of Rob’s immediate futures are in his bedroom than in Beijing”.
But if we restrict our attention to this Beijing, then all that complexity goes away and we can pretty much rule out that anyone in Beijing will happen to momentarily exhibit exactly the right brain state to look like “Rob Bensinger plus one time step”.
The nuances and wrinkles don’t bleed over and make it a totally meaningless or arbitrary question; and indeed, if I thought I were likely to spontaneously teleport to Beijing in the next minute, I’d rightly be making very different life-choices! “Will I experience myself spontaneously teleporting to Beijing in the next second?” is a substantive (and easy) question, not a deep philosophical riddle.
So you always anticipate all possible experiences, because of multiverse?
Not all possible experiences; just all experiences of brains that have the same kinds of structural similarities to your current brain as, e.g., “me after I step through a doorway” has to “me before I stepped through the doorway”.
When is a mind me?
The problem is another way to phrase this is a superintelligent weapon system—“ending a risk period” by “reliably, and efficiently doing a small number of specific concrete tasks” means using physical force to impose your will on others.
The pivotal acts I usually think about actually don’t route through physically messing with anyone else. I’m usually thinking about using aligned AGI to bootstrap to fast human whole-brain emulation, then using the ems to bootstrap to fully aligned CEV AI.
If someone pushes a “destroy the world” button then the ems or CEV AI would need to stop the world from being destroyed, but that won’t necessarily happen if the developers have enough of a lead, if they get the job done quickly enough, and if CEV AI is able to persuade the world to step back from the precipice voluntarily (using superhumanly good persuasion that isn’t mind-control-y, deceptive, or otherwise consent-violating). It’s a big ask, but not as big as CEV itself, I expect.
From my current perspective this is all somewhat of a moot point, however, because I don’t think alignment is tractable enough that humanity should be trying to use aligned AI to prevent human extinction. I think we should instead hit the brakes on AI and shift efforts toward human enhancement, until some future generation is in a better position to handle the alignment problem.
If and only if that fails it may be appropriate to consider less consensual options.
It’s not clear to me that we disagree in any action-relevant way, since I also don’t think AI-enabled pivotal acts are the best path forward anymore. I think the path forward is via international agreements banning dangerous tech, and technical research to improve humanity’s ability to wield such tech someday.
That said, it’s not clear to me how your “if that fails, then try X instead” works in practice. How do you know when it’s failed? Isn’t it likely to be too late by the time we’re sure that we’ve failed on that front? Indeed, it’s plausibly already too late for humanity to seriously pivot to ‘aligned AGI’. If I thought humanity’s last best scrap of hope for survival lay in an AI-empowered pivotal act, I’d certainly want more details on when it’s OK to start trying to figure out have humanity not die via this last desperate path.
To pick out a couple of specific examples from your list, Wei Dai:
14. Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can’t be “balanced out” later) prior to reaching moral/philosophical maturity
This is a serious long-term concern if we don’t kill ourselves first, but it’s not something I see as a premise for “the priority is for governments around the world to form an international agreement to halt AI progress”. If AI were easy to use for concrete tasks like “build nanotechnology” but hard to use for things like CEV, I’d instead see the priority as “use AI to prevent anyone else from destroying the world with AI”, and I wouldn’t want to trade off probability of that plan working in exchange for (e.g.) more probability of the US and the EU agreeing in advance to centralize and monitor large computing clusters.
After someone has done a pivotal act like that, you might then want to move more slowly insofar as you’re worried about subtle moral errors creeping in to precursors to CEV.
30. AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else)
I currently assign very low probability to humans being able to control the first ASI systems, and redirecting governments’ attention away from “rogue AI” and toward “rogue humans using AI” seems very risky to me, insofar as it causes governments to misunderstand the situation, and to specifically misunderstand it in a way that encourages racing.
If you think rogue actors can use ASI to achieve their ends, then you should probably also think that you could use ASI to achieve your own ends; misuse risk tends to go hand-in-hand with “we’re the Good Guys, let’s try to outrace the Bad Guys so AI ends up in the right hands”. This could maybe be justified if it were true, but when it’s not even true it strikes me as an especially bad argument to make.
Yep, before I saw orthonormal’s response I had a draft-reply written that says almost literally the same thing:
we just call ‘em like we see ’em
[...]
insofar as we make bad predictions, we should get penalized for it. and insofar as we think alignment difficulty is the crux for ‘why we need to shut it all down’, we’d rather directly argue against illusory alignment progress (and directly acknowledge real major alignment progress as a real reason to be less confident of shutdown as a strategy) rather than redirect to something less cruxy
I’ll also add: Nate (unlike Eliezer, AFAIK?) hasn’t flatly said ‘alignment is extremely difficult’. Quoting from Nate’s “sharp left turn” post:
Many people wrongly believe that I’m pessimistic because I think the alignment problem is extraordinarily difficult on a purely technical level. That’s flatly false, and is pretty high up there on my list of least favorite misconceptions of my views.
I think the problem is a normal problem of mastering some scientific field, as humanity has done many times before. Maybe it’s somewhat trickier, on account of (e.g.) intelligence being more complicated than, say, physics; maybe it’s somewhat easier on account of how we have more introspective access to a working mind than we have to the low-level physical fields; but on the whole, I doubt it’s all that qualitatively different than the sorts of summits humanity has surmounted before.
It’s made trickier by the fact that we probably have to attain mastery of general intelligence before we spend a bunch of time working with general intelligences (on account of how we seem likely to kill ourselves by accident within a few years, once we have AGIs on hand, if no pivotal act occurs), but that alone is not enough to undermine my hope.
What undermines my hope is that nobody seems to be working on the hard bits, and I don’t currently expect most people to become convinced that they need to solve those hard bits until it’s too late.
So it may be that Nate’s models would be less surprised by alignment breakthroughs than Eliezer’s models. And some other MIRI folks are much more optimistic than Nate, FWIW.
My own view is that I don’t feel nervous leaning on “we won’t crack open alignment in time” as a premise, and absent that premise I’d indeed be much less gung-ho about government intervention.
why put all your argumentative eggs in the “alignment is hard” basket? (If you’re right, then policymakers can’t tell that you’re right.)
The short answer is “we don’t put all our eggs in the basket” (e.g., Eliezer’s TED talk and TIME article emphasize that alignment is an open problem, but they emphasize other things too, and they don’t go into detail on exactly how hard Eliezer thinks the problem is), plus “we very much want at least some eggs in that basket because it’s true, it’s honest, it’s cruxy for us, etc.” And it’s easier for policymakers to acquire strong Bayesian evidence for “the problem is currently unsolved” and “there’s no consensus about how to solve it” and “most leaders in the field seem to think there’s a serious chance we won’t solve it in time” than to acquire strong Bayesian evidence for “we’re very likely generations away from solving alignment”, so the difficulty of communicating the latter isn’t a strong reason to de-emphasize all the former points.
The longer answer is a lot more complicated. We’re still figuring out how best to communicate our views to different audiences, and “it’s hard for policymakers to evaluate all the local arguments or know whether Yann LeCun is making more sense than Yoshua Bengio” is a serious constraint. If there’s a specific argument (or e.g. a specific three arguments) you think we should be emphasizing alongside “alignment is unsolved and looks hard”, I’d be interested to hear your suggestion and your reasoning. https://www.lesswrong.com/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk is a very long list and isn’t optimized for policymakers, so I’m not sure what specific changes you have in mind here.
I expect it makes it easier, but I don’t think it’s solved.
Suppose you want to synthesize a lot of diamonds. Instead of giving an AI some lofty goal “maximize diamonds in an aligned way”, why not a bunch of small grounded ones.
“Plan the factory layout of the diamond synthesis plant with these requirements”.
“Order the equipment needed, here’s the payment credentials”.
“Supervise construction this workday comparing to original plans”
“Given this step of the plan, do it”
(Once the factory is built) “remove the output from diamond synthesis machine A53 and clean it”.
That is how MIRI imagines a sane developer using just-barely-aligned AI to save the world. You don’t build an open-ended maximizer and unleash it on the world to maximize some quantity that sounds good to you; that sounds insanely difficult. You carve out as many tasks as you can into concrete, verifiable chunks, and you build the weakest and most limited possible AI you can to complete each chunk, to minimize risk. (Though per faul_sname, you’re likely to be pretty limited in how much you can carve up the task, given time will be a major constraint and there may be parts of the task you don’t fully understand at the outset.)
Cf. The Rocket Alignment Problem. The point of solving the diamond maximizer problem isn’t to go build the thing; it’s that solving it is an indication that we’ve become less conceptually confused about real-world optimization and about aimable cognitive work. Being less conceptually confused about very basic aspects of problem-solving and goal-oriented reasoning means that you might be able to build some of your powerful AI systems out of building blocks that are relatively easy to analyze, test, design, predict, separate out into discrete modules, measure and limit the capabilities of, etc., etc.
That seems acceptable, industrial equipment causes accidents all the time, the main thing is to limit the damage. Fences to limit the robots operating area, timers that shut down control after a timeout, etc.
If everyone in the world chooses to permanently use very weak systems because they’re scared of AI killing them, then yes, the impact of any given system failing will stay low. But that’s not what’s going to actually happen; many people will use more powerful systems, once they can, because they misunderstand the risks or have galaxy-brained their way into not caring about them (e.g. ‘maybe humans don’t deserve to live’, ‘if I don’t do it someone else will anyway’, ‘if it’s that easy to destroy the world then we’re fucked anyway so I should just do the Modest thing of assuming nothing I do is that important’...).
The world needs some solution to the problem “if AI keeps advancing and more-powerful AI keeps proliferating, eventually someone will destroy the world with it”. I don’t know of a way to leverage AI to solve that problem without the AI being pretty dangerously powerful, so I don’t think AI is going to get us out of this mess unless we make a shocking amount of progress on figuring out how to align more powerful systems. (Where “aligning” includes things like being able to predict in advance how pragmatically powerful your system is, and being able to carefully limit the ways in which it’s powerful.)
To be clear: The diamond maximizer problem is about getting specific intended content into the AI’s goals (“diamonds” as opposed to some random physical structure it’s maximizing), not just about building a stable maximizer.
From briefly talking to Eliezer about this the other day, I think the story from MIRI’s perspective is more like:
Back in 2001, we defined “Friendly AI” as “The field of study concerned with the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.”
We could have defined the goal more narrowly or generically than that, but that just seemed like an invitation to take your eye off the ball: if we aren’t going to think about the question of how to get good long-run outcomes from powerful AI systems, who will?
And many of the technical and philosophical problems seemed particular to CEV, which seemed like an obvious sort of solution to shoot for: just find some way to leverage the AI’s intelligence to solve the problem of extrapolating everyone’s preferences in a reasonable way, and of aggregating those preferences fairly.
Come 2014, Stuart Russell and MIRI were both looking for a new term to replace “the Friendly AI problem”, now that the field was starting to become a Real Thing. Both parties disliked Bostrom’s “the control problem”. In conversation, Russell proposed “the alignment problem”, and MIRI liked it, so Russell and MIRI both started using the term in public.
Unfortunately, it gradually came to light that Russell and MIRI had understood “Friendly AI” to mean two moderately different things, and this disconnect now turned into a split between how MIRI used “(AI) alignment” and how Russell used “(value) alignment”. (Which I think also influenced the split between Paul Christiano’s “(intent) alignment” and MIRI’s “(outcome) alignment”.)
Russell’s version of “friendliness/alignment” was about making the AI have good, human-deferential goals. But Creating Friendly AI 1.0 had been very explicit that “friendliness” was about good behavior, regardless of how that’s achieved. MIRI’s conception of “the alignment problem” (like Bostrom’s “control problem”) included tools like capability constraint and boxing, because the thing we wanted researchers to focus on was the goal of leveraging AI capabilities to get actually-good outcomes, whatever technical work that requires, not some proxy goal that might turn out to be surprisingly irrelevant.
Again, we wanted a field of people keeping their eye on the ball and looking for clever technical ways to get the job done, rather than a field that neglects some actually-useful technique because it doesn’t fit their narrow definition of “alignment”.
Meanwhile, developments like the rise of deep learning had updated MIRI that CEV was not going to be a realistic thing to shoot for with your first AI. We were still thinking of some version of CEV as the ultimate goal, but it now seemed clear that capabilities were progressing too quickly for humanity to have time to nail down all the details of CEV, and it was also clear that the approaches to AI that were winning out would be far harder to analyze, predict, and “aim” than 2001-Eliezer had expected. It seemed clear that if AI was going to help make the future go well, the first order of business would be to do the minimal thing to prevent other AIs from destroying the world six months later, with other parts of alignment/friendliness deferred to later.
I think considerations like this eventually trickled in to how MIRI used the term “alignment”. Our first public writing reflecting the switch from “Friendly AI” to “alignment”, our Dec. 2014 agent foundations research agenda, said:
We call a smarter-than-human system that reliably pursues beneficial goals “aligned with human interests” or simply “aligned.”
Whereas by July 2016, when we released a new research agenda that was more ML-focused, “aligned” was shorthand for “aligned with the interests of the operators”.
In practice, we started using “aligned” to mean something more like “aimable” (where aimability includes things like corrigibility, limiting side-effects, monitoring and limiting capabilities, etc., not just “getting the AI to predictably tile the universe with smiley faces rather than paperclips”). Focusing on CEV-ish systems mostly seemed like a distraction, and an invitation to get caught up in moral philosophy and pie-in-the-sky abstractions, when “do a pivotal act” is legitimately a hugely more philosophically shallow topic than “implement CEV”. Instead, we went out of our way to frame the challenge of alignment in a way that seemed almost comically simple and “un-philosophical”, but that successfully captured all of the key obstacles: ‘explain how to use an AI to cause there two exist two strawberries that are identical at the cellular level, without causing anything weird or disruptive to happen in the process’.
Since realistic pivotal acts still seemed pretty outside the Overton window (and since we were mostly focused on our own research at the time), we wrote up our basic thoughts about the topic on Arbital but didn’t try to super-popularize the topic among rationalists or EAs at the time. (Which unfortunately, I think, exacerbated a situation where the larger communities had very fuzzy models of the strategic situation, and fuzzy models of what the point even was of this “alignment research” thing; alignment research just become a thing-that-was-good-because-it-was-a-good, not a concrete part of a plan backchained from concrete real-world goals.)
I don’t think MIRI wants to stop using “aligned” in the context of pivotal acts, and I also don’t think MIRI wants to totally divorce the term from the original long-term goal of friendliness/alignment.
Turning “alignment” purely into a matter of “get the AI to do what a particular stakeholder wants” is good in some ways—e.g., it clarifies that the level of alignment needed for pivotal acts could also be used to do bad things.
But from Eliezer’s perspective, this move would also be sending a message to all the young Eliezers “Alignment Research is what you do if you’re a serious sober person who thinks it’s naive to care about Doing The Right Thing and is instead just trying to make AI Useful To Powerful People; if you want to aim for the obvious desideratum of making AI friendly and beneficial to the world, go join e/acc or something”. Which does not seem ideal.
So I think my proposed solution would be to just acknowledge that ‘the alignment problem’ is ambiguous between three different (overlapping) efforts to figure out how to get good and/or intended outcomes from powerful AI systems:
intent alignment, which is about getting AIs to try to do what the AI thinks the user wants, and in practice seems to be most interested in ‘how do we get AIs to be generically trying-to-be-helpful’.
“strawberry problem” alignment, which is about getting AIs to safely, reliably, and efficiently do a small number of specific concrete tasks that are very difficult, for the sake of ending the acute existential risk period.
CEV-style alignment, which is about getting AIs to fully figure out how to make the future good.
Plausibly it would help to have better names for the latter two things. The distinction is similar to “narrow value learning vs. ambitious value learning”, but both problems (as MIRI thinks about them) are a lot more general than just “value learning”, and there’s a lot more content to the strawberry problem than to “narrow alignment”, and more content to CEV than to “ambitious value learning” (e.g., CEV cares about aggregation across people, not just about extrapolation).
(Note: Take the above summary of MIRI’s history with a grain of salt; I had Nate Soares look at this comment and he said “on a skim, it doesn’t seem to quite line up with my recollections nor cut things along the joints I would currently cut them along, but maybe it’s better than nothing”.)
- 's comment on Paul Christiano named as US AI Safety Institute Head of AI Safety by (Apr 18, 2024, 10:53 PM; 20 points)
- Definition of alignment science I like by (Jan 6, 2025, 8:40 PM; 19 points)
- 's comment on Why I no longer identify as transhumanist by (Feb 7, 2024, 4:18 AM; 5 points)
- 's comment on TruePath’s Shortform by (Dec 31, 2023, 2:48 PM; 2 points)
- 's comment on Matthew Barnett’s Shortform by (Jun 18, 2024, 3:43 AM; 2 points)
- 's comment on The Gemini Incident by (Feb 23, 2024, 10:27 PM; 2 points)
In the context of a conversation with Balaji Srinivasan about my AI views snapshot, I asked Nate Soares what sorts of alignment results would impress him, and he said:
example thing that would be relatively impressive to me: specific, comprehensive understanding of models (with the caveat that that knowledge may lend itself more (and sooner) to capabilities before alignment). demonstrated e.g. by the ability to precisely predict the capabilities and quirks of the next generation (before running it)
i’d also still be impressed by simple theories of aimable cognition (i mostly don’t expect that sort of thing to have time to play out any more, but if someone was able to come up with one after staring at LLMs for a while, i would at least be impressed)
fwiw i don’t myself really know how to answer the question “technical research is more useful than policy research”; like that question sounds to me like it’s generated from a place of “enough of either of these will save you” whereas my model is more like “you need both”
tho i’m more like “to get the requisite technical research, aim for uploads” at this juncture
if this was gonna be blasted outwards, i’d maybe also caveat that, while a bunch of this is a type of interpretability work, i also expect a bunch of interpretability work to strike me as fake, shallow, or far short of the bar i consider impressive/hopeful
(which is not itself supposed to be any kind of sideswipe; i applaud interpretability efforts even while thinking it’s moving too slowly etc.)
AI Views Snapshots
I can come up with plans for destroying the world without wanting to do it, and other cognitive systems probably can too.
You’re changing the topic to “can you do X without wanting Y?”, when the original question was “can you do X without wanting anything at all?”.
Nate’s answer to nearly all questions of the form “can you do X without wanting Y?” is “yes”, hence his second claim in the OP: “the wanting-like behavior required to pursue a particular training target X, does not need to involve the AI wanting X in particular”.
I do need to answer that question using in a goal-oriented search process. But my goal would be “answer Paul’s question”, not “destroy the world”.
Your ultimate goal would be neither of those things; you’re a human, and if you’re answering Paul’s question it’s probably because you have other goals that are served by answering.
In the same way, an AI that’s sufficiently good at answering sufficiently hard and varied questions would probably also have goals, and it’s unlikely by default that “answer questions” will be the AI’s primary goal.
The idea that an area of study is less scientific because the subject is inelegant is a blinkered view of what science is.
See my reply to Bogdan here. The issue isn’t “inelegance”; we also lack an inelegant ability to predict or explain how particular ML systems do what they do.
Modern ML is less like modern chemistry, and more like ancient culinary arts and medicine. (Or “ancient culinary arts and medicine shortly after a cultural reboot”, such that we have a relatively small number of recently-developed shallow heuristics and facts to draw on, rather than centuries of hard-earned experience.)
The opening sounds a lot like saying “aerodynamics used to be a science until people started building planes.”
The reason this analogy doesn’t land for me is that I don’t think our epistemic position regarding LLMs is similar to, e.g., the Wright brothers’ epistemic position regarding heavier-than-air flight.
The point Nate was trying to make with “ML is no longer a science” wasn’t “boo current ML that actually works, yay GOFAI that didn’t work”. The point was exactly to draw a contrast between, e.g., our understanding of heavier-than-air flight and our understanding of how the human brain works. The invention of useful tech that interfaces with the brain doesn’t entail that we understand the brain’s workings in the way we’ve long understood flight; it depends on what the (actual or hypothetical) tech is.
Maybe a clearer way of phrasing it is “AI used to be failed science; now it’s (mostly, outside of a few small oases) a not-even-attempted science”. “Failed science” maybe makes it clearer that the point here isn’t to praise the old approaches that didn’t work; there’s a more nuanced point being made.
Some of Nate’s quick thoughts (paraphrased), after chatting with him:
Nate isn’t trying to say that we have literally zero understanding of deep nets. What he’s trying to do is qualitatively point to the kind of high-level situation we’re in, in part because he thinks there is real interpretability progress, and when you’re working in the interpretability mines and seeing real advances it can be easy to miss the forest for the trees and forget how far we are from understanding what LLMs are doing. (Compared to, e.g., how well we can predict or post-facto-mechanistically-explain a typical system humans have engineered.)
Nobody’s been able to call the specific capabilities of systems in advance. Nobody’s been able to call the specific exploits in advance. Nobody’s been able to build better cognitive algorithms by hand after understanding how the AI does things we can’t yet code by hand. There is clearly some other level of understanding that is possible that we lack, and that we once sought, and that only the interpretability folks continue to seek.
E.g., think of that time Neel Nanda figured out how a small transformer does modular arithmetic (AXRP episode). If nobody had ever thought of that algorithm for an adder, we would have thereby learned a new algorithm for an adder. There are things that these AI systems are doing that aren’t just lots of stuff we know; there are levels of organization of understanding that give you the ability to predict how things work outside of the bands where we’ve observed them.
It seems trendy to declare that they never existed in the first place and that that’s all white tower stuff, but Nate thinks this point of view is missing a pretty important and central thread.
The missing thread isn’t trivial to put into words, but it includes things like:
This sounds like the same sort of thing some people would say if they were staring at computer binary for the first time and didn’t know about the code behind the scenes: “We have plenty of understanding beyond just how the CPU handles instructions; we understand how memory caching works and we have recognized patterns like the stack and the heap; talking as if there’s some deeper level of organization is talking like a theorist when in fact this is an engineering problem.” Those types of understanding aren’t false, but they aren’t the sort of understanding of someone who has comprehended the codebase they’re looking at.
There are, predictably, things to learn here; the messiness and complexity of the real world doesn’t mean we already know the relevant principles. You don’t need to understand everything about how a bird works in order to build an airplane; there are compressible principles behind how birds fly; if you understand what’s going on you can build flying devices that have significantly more carrying capacity than a bird, and this holds true even if the practical engineering of an airplane requires a bunch of trial and error and messy engineering work.
A mind’s causal structure is allowed to be complicated; we can see the weights, but we don’t thereby have a mastery of the high-level patterns. In the case of humans, neuroscience hasn’t actually worked to give us a mastery of the high-level patterns the human brain is implementing.
Mystery is in the map, not in the territory; reductionism works. Not all sciences that can exist, already exist today.
Possibly the above pointers are only useful if you already grok the point we’re trying to make, and isn’t so useful for communicating a new idea; but perhaps not.
I read and responded to some pieces of that post when it came out; I don’t know whether Eliezer, Nate, etc. read it, and I’m guessing it didn’t shift MIRI, except as one of many data points “person X is now loudly in favor of a pause (and other people seem receptive), so maybe this is more politically tractable than we thought”.
I’d say that Kerry Vaughan was the main person who started smashing this Overton window, and this started in April/May/June of 2022. By late December my recollection is that this public conversation was already fully in swing and MIRI had already added our voices to the “stop building toward AGI” chorus. (Though at that stage I think we were mostly doing this on general principle, for lack of any better ideas than “share our actual long-standing views and hope that helps somehow”. Our increased optimism about policy solutions mostly came later, in 2023.)
That said, I bet Katja’s post had tons of relevant positive effects even if it didn’t directly shift MIRI’s views.
Yep! Ask someone with this view whether the current stream of consciousness continues from their pre-uploaded self to their post-uploaded self, like it continues when they pass through a doorway. The typical claim is some version of “this stream of consciousness will end, what comes next is only oblivion”, not “oh sure, the stream of consciousness is going to continue in the same way it always does, but I prefer not to use the English word ‘me’ to refer to the later parts of that stream of consciousness”.
This is why the disagreement here has policy implications: people with different views of personal identity have different beliefs about the desirability of mind uploading. They aren’t just disagreeing about how to use words, and if they were, you’d be forced into the equally “uncharitable” perspective that someone here is very confused about how relevant word choice is to the desirability of uploading.
I didn’t say that the relevant people endorse a belief in ghosts or magic. (Some may do so, but many explicitly don’t!)
It’s a bit darkly funny that you’ve reached for a clearly false and super-uncharitable interpretation of what I said, in the same sentence you’re chastising me for being uncharitable! But also, “charity” is a bad approach to trying to understand other people, and bad epistemology can get in the way of a lot of stuff.