A podcast interview (posted 2023-06-29) with noted AI researcher Douglas Hofstadter discusses his career and current views on AI (via Edward Kmett), and amplified to David Brooks.
Hofstadter has previously energetically criticized GPT-2/3 models (and deep learning and compute-heavy GOFAI). These criticisms were widely circulated & cited, and apparently many people found Hofstadter a convincing & trustworthy authority when he was negative on deep learning capabilities & prospects, and so I found his most-recent comments (which amplify things he has been saying in private since at least 2014) of considerable interest.
This interview (EDIT: and earlier material, it turns out), appears to have gone under the radar, perhaps because it’s a video, so below I excerpt from the second half where he discusses DL progress & AI risk:
Q: …Which ideas from GEB are most relevant today?
Douglas Hofstadter: …In my book, I Am a Strange Loop, I tried to set forth what it is that really makes a self or a soul. I like to use the word “soul”, not in the religious sense, but as a synonym for “I”, a human “I”, capital letter “I.” So, what is it that makes a human being able to validly say “I”? What justifies the use of that word? When can a computer say “I” and we feel that there is a genuine “I” behind the scenes?
I don’t mean like when you call up the drugstore and the chatbot, or whatever you want to call it, on the phone says, “Tell me what you want. I know you want to talk to a human being, but first, in a few words, tell me what you want. I can understand full sentences.” And then you say something and it says, “Do you want to refill a prescription?” And then when I say yes, it says, “Gotcha”, meaning “I got you.” So it acts as if there is an “I” there, but I don’t have any sense whatsoever that there is an “I” there. It doesn’t feel like an “I” to me, it feels like a very mechanical process.
But in the case of more advanced things like ChatGPT-3 or GPT-4, it feels like there is something more there that merits the word “I.” The question is, when will we feel that those things actually deserve to be thought of as being full-fledged, or at least partly fledged, “I”s?
I personally worry that this is happening right now. But it’s not only happening right now. It’s not just that certain things that are coming about are similar to human consciousness or human selves. They are also very different, and in one way, it is extremely frightening to me. They are extraordinarily much more knowledgeable and they are extraordinarily much faster. So that if I were to take an hour in doing something, the ChatGPT-4 might take one second, maybe not even a second, to do exactly the same thing.
And that suggests that these entities, whatever you want to think of them, are going to be very soon, right now they still make so many mistakes that we can’t call them more intelligent than us, but very soon they’re going to be, they may very well be more intelligent than us and far more intelligent than us. And at that point, we will be receding into the background in some sense. We will have handed the baton over to our successors, for better or for worse.
And I can understand that if this were to happen over a long period of time, like hundreds of years, that might be okay. But it’s happening over a period of a few years. It’s like a tidal wave that is washing over us at unprecedented and unimagined speeds. And to me, it’s quite terrifying because it suggests that everything that I used to believe was the case is being overturned.
Q: What are some things specifically that terrify you? What are some issues that you’re really...
D. Hofstadter: When I started out studying cognitive science and thinking about the mind and computation, you know, this was many years ago, around 1960, and I knew how computers worked and I knew how extraordinarily rigid they were. You made the slightest typing error and it completely ruined your program. Debugging was a very difficult art and you might have to run your program many times in order to just get the bugs out. And then when it ran, it would be very rigid and it might not do exactly what you wanted it to do because you hadn’t told it exactly what you wanted to do correctly, and you had to change your program, and on and on.
Computers were very rigid and I grew up with a certain feeling about what computers can or cannot do. And I thought that artificial intelligence, when I heard about it, was a very fascinating goal, which is to make rigid systems act fluid. But to me, that was a very long, remote goal. It seemed infinitely far away. It felt as if artificial intelligence was the art of trying to make very rigid systems behave as if they were fluid. And I felt that would take enormous amounts of time. I felt it would be hundreds of years before anything even remotely like a human mind would be asymptotically approaching the level of the human mind, but from beneath.
I never imagined that computers would rival, let alone surpass, human intelligence. And in principle, I thought they could rival human intelligence. I didn’t see any reason that they couldn’t. But it seemed to me like it was a goal that was so far away, I wasn’t worried about it. But when certain systems started appearing, maybe 20 years ago, they gave me pause. And then this started happening at an accelerating pace, where unreachable goals and things that computers shouldn’t be able to do started toppling. The defeat of Gary Kasparov by Deep Blue, and then going on to Go systems, Go programs, well, systems that could defeat some of the best Go players in the world. And then systems got better and better at translation between languages, and then at producing intelligible responses to difficult questions in natural language, and even writing poetry.
And my whole intellectual edifice, my system of beliefs… It’s a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon. People ask me, “What do you mean by ‘soon’?” And I don’t know what I really mean. I don’t have any way of knowing. But some part of me says 5 years, some part of me says 20 years, some part of me says, “I don’t know, I have no idea.” But the progress, the accelerating progress, has been so unexpected, so completely caught me off guard, not only myself but many, many people, that there is a certain kind of terror of an oncoming tsunami that is going to catch all humanity off guard.
It’s not clear whether that will mean the end of humanity in the sense of the systems we’ve created destroying us. It’s not clear if that’s the case, but it’s certainly conceivable. If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches.
Q: That’s an interesting thought. [nervous laughter]
Hofstadter: Well, I don’t think it’s interesting. I think it’s terrifying. I hate it. I think about it practically all the time, every single day. [Q: Wow.] And it overwhelms me and depresses me in a way that I haven’t been depressed for a very long time.
Q: Wow, that’s really intense. You have a unique perspective, so knowing you feel that way is very powerful.
Q: How have LLMs, large language models, impacted your view of how human thought and creativity works?
D H: Of course, it reinforces the idea that human creativity and so forth come from the brain’s hardware. There is nothing else than the brain’s hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It’s like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn’t make sense to me, but that just shows that I’m naive.
It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn’t think it was going to happen, you know, within a very short time.
And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don’t want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we’re so imperfect and so fallible. We forget things all the time, we confuse things all the time, we contradict ourselves all the time. You know, it may very well be that that just shows how limited we are.
Q: Wow. So let me keep going through the questions. Is there a time in our history as human beings when there was something analogous that terrified a lot of smart people?
D H: Fire.
Q: You didn’t even hesitate, did you? So what can we learn from that?
D H: No, I don’t know. Caution, but you know, we may have already gone too far. We may have already set the forest on fire. I mean, it seems to me that we’ve already done that. I don’t think there’s any way of going back.
When I saw an interview with Geoff Hinton, who was probably the most central person in the development of all of these kinds of systems, he said something striking. He said he might regret his life’s work. He said, “Part of me regrets all of my life’s work.” The interviewer then asked him how important these developments are. “Are they as important as the Industrial Revolution? Is there something analogous in history that terrified people?” Hinton thought for a second and he said, “Well, maybe as important as the wheel.”
(YouTube transcript cleaned up by GPT-4 & checked against audio.)
It is beautiful to see that many of our greatest minds are willing to Say Oops, even about their most famous works. It may not score that many winning-points, but it does restore quite a lot of dignity-points I think.
I felt exactly the same, until I had read this June 2020 paper: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.
It turns out that using Transformers in the autoregressive mode (with output tokens being added back to the input by concatenating the previous input and the new output token, and sending the new versions of the input through the model again and again) results in them emulating dynamics of recurrent neural networks, and that clarifies things a lot...
Yeah, there’s obviously SOME recursion there but it’s still surprising that such a relatively low bandwidth recursion can still work so well. It’s more akin to me writing down my thoughts and then rereading them to gather my ideas than the kind of loops I imagine our neurons might have.
That said, who knows, maybe the loops in our brain are superfluous, or only useful for learning feedback purposes, and so a neural network trained by an external system doesn’t need them.
In a sense, that is what is happening when you think in words. It’s called the phonological loop.
I think it seems that way, in your conscious thoughts, but actually there’s a lot more inter-brain-region communication going on simultaneously. I think that without this, you’d see far worse human outputs. And I think once we add something like higher-bandwidth-recursive-thought into language models, we’re going to see a capabilities jump.
It sounds a lot like what we do when we write (as opposed to talk). I recall Kurt Vonnegut once said something like (can’t find cite sry)
‘The reason an author can sound intelligent is because they have the advantage of time. My brain is so slow, people have thought me stupid. But as a writer, I can think at my own speed.’
Think of it this way: how would it feel to chat with someone whose perception of time is 10X slower? Or 100X or 1000X—or, imagine playing chess where your clock was running orders of mag faster than your opponent’s.
Pondering this particular recursion, I noticed that it looks like things change not too much from iteration to iteration of this autoregressive dynamics, because we just add one token each time.
The key property of those artificial recurrent architectures which successfully fight the vanishing gradient problem is that a single iteration of recurrence looks like Identity + epsilon (so, X → X + deltaX for a small deltaX on each iteration, see, for example, this 2018 paper, Overcoming the vanishing gradient problem in plain recurrent networks which explains how this is the case for LSTMs and such, and explains how to achieve this for plain recurrent networks; for a brief explanation see my review of the first version of this paper, Understanding Recurrent Identity Networks).
So, I strongly suspect that it is also the case for the recurrence which is happening in Transformers used in the autoregressive mode (because the input is changing mildly from iteration to iteration).
But I don’t know to which extent this is also true for biological recurrent networks. On one hand, our perceptions seem to change smoothly with time, and that seems to be an argument for gradual change of the X → X + deltaX nature in the biological case as well. But we don’t understand the biological case all that well...
I think recurrence is actually quite important for LLMs. Cf. Janus’ Simulator theory which is now relatively well developed (see e.g. the original Simulators or brief notes I took on the recent status of that theory May-23-2023-status-update). The fact that this is an autoregressive simulation is playing the key role.
But we indeed don’t know whether complexity of biological recurrences vs. relative simplicity of artificial recurrent networks matters much...
I’d speculate that our perceptions just seem to change smoothly because we encode second-order (or even third-order) dynamics in our tokens. From what I layman-understand of consciousness, I’d be surprised if it wasn’t discrete.
Can you explain what you mean by second or third order dynamics? That sounds interesting. Do you mean e.g. the order of the differential equation or something else?
I just mean like, if we see an object move we have a qualia of position but also of velocity/vector and maybe acceleration. So when we see for instance a sphere rolling down an incline, we may have a discrete conscious “frame” where the marble has a velocity of 0 but a positive acceleration, so despite the fact that the next frame is discontinuous with the last one looking only at position, we perceive them as one smooth sequence because the predicted end position of the motion in the first frame is continuous with the start point in the second.
This seems to me the opposite of a low bandwidth recursion. Having access the the entire context window of the previous iteration minus the first token, it should be pretty obvious that most of the relevant information encoded by the values of the nodes in that iteration could in principal be reconstructed, excepting the unlikely event that first token turns out to be extremely important. And it would be pretty weird if much if that information wasn’t actually reconstructed in some sense in the current iteration. An inefficient way to get information from one iteration to the next, if that is your only goal, but plausibly very high bandwidth.
Which is why asking an LLM to give an answer that starts with “Yes” or “No” and then gives an explanation is the worst possible way to do it.
This was thought provoking. While I believe what you said is currently true for the LLMs I’ve used, a sufficiently expensive decoding strategy would overcome it. Might be neat to try this for the specific case you describe. Ask it a question that it would answer correctly with a good prompt style, but use the bad prompt style (asking to give an answer that starts with Yes or No), and watch how the ratio of the cumulative probabilities of Yes* and No* sequences changes as you explore the token sequence tree.
I’d say it’s pretty low bandwidth compared to the wealth of information that must exist in the intermediate layers. Even just the distribution of logits gets collapsed into a single returned value. You could definitely send back more than just that, but the question is whether it’s workable or if it just adds confusion.
The loops in our neurons can’t be that great, otherwise I wouldn’t benefit so much from writing down my thoughts and then rereading them. :P
(Not a serious disagreement with you, I think I agree overall)
It could also be that LLMs don’t do it like we do it and simply offer a computationally sufficient platform.
In what sense do they emulate these dynamics?
The formulas and a brief discussion are in Section 3.4 (page 5) of https://arxiv.org/abs/2006.16236
Thanks!
Further discussion on Twitter of feedforward vs recurrent.
Thanks!
Being an autoregressive language model is like having a strange form of amnesia, where you forget everything you thought about so far as soon as you utter a new word, and you can remember only what you said before.
that paper is one of many claiming some linear attention mechanism that’s as good as full self attention. in practice they’re all sufficiently much worse that nobody uses them except the original authors in the original paper, usually not even the original authors in subsequent papers.
the one exception is flash attention, which is basically just a very fancy fused kernel for the same computation (actually the same, up to numerical error, unlike all these “linear attention” papers).
>It turns out that using Transformers in the autoregressive mode (with output tokens being added back to the input by concatenating the previous input and the new output token, and sending the new versions of the input through the model again and again) results in them emulating dynamics of recurrent neural networks, and that clarifies things a lot...
I’ll bite: Could you dumb down the implications of the paper a little bit, what is the difference between a Transformer emulating a RNN and some pre-Transformer RNNs and/or not-RNN?
My much more novice-level answer to Hofstadter’s intuition would have been: it’s not the feedforward firing, but it is the gradient descent training of the model on massive scale (both in data and in computation). But apparently you think that something RNN-like about the model structure itself is important?
I think that gradient descent in computation is super-important (this is, apparently, the key mechanism responsible for the phenomenon of few-shot learning).
And, moreover, massive linear combinations of vectors (“artificial attention”) seem to be super-important (the starting point in this sense was adding this kind of artificial attention mechanism to the RNN architecture in 2014).
Yes, this might be related to my personal history, which is that I have been focusing on whether one can express algorithms as neural machines, and whether one can meaningfully speak about continuously deformable programs.
And, then, for Turing completeness one would want both unlimited number of steps and unbounded memory, and there has been a rather involved debate on whether RNNs are more like Turing complete programs, or are they, in practice, only similar to finite automata. (It’s a long topic, on which there is more to say.)
So, from this viewpoint, a machine with a fixed finite number of steps seems very limited.
But autoregressive Transformers are not machines with a fixed finite number of steps, they just commit to emitting a token after a fixed number of steps, but they can continue in an unbounded fashion, so they are very similar to RNNs in this sense.
I’ll bite even further, and ask for the concept of “recurrence” itself to be dumbed down. What is “recurrence”, why is it important, and in what sense does e.g. a feedforward network hooked up to something like MCTS not qualify as relevantly “recurrent”?
“Hooked up to something” might make a difference.
(To me one important aspect is whether computation is fundamentally limited to a fixed number of steps vs. having a potentially unbounded loop.
The autoregressive version is an interesting compromise: it’s a fixed number of steps per token, but the answer can unfold in an unbounded fashion.
An interesting tid-bit here is that for traditional RNNs it is one loop iteration per an input token, but in autoregressive Transformers it is one loop iteration per an output token.)
I don’t think I’ve ever seen a better description of how I feel about the coming creation of artificial superintelligence. I find myself returning over and over again to that post by benkuhn about “Staring into the abyss as a core life skill” I think that is going to become a necessary core life skill for almost everyone in the coming years.
It has been morbidly gratifying to see more and more people develop the same feelings about AI as I have had for about a year now. Like validation in the worst possible way. I think if people actually understood what was coming there would be a near total call to ban improvements in this technology and only allow advancement under very strict conditions. But almost no one has really thought through the consequences of making a general purpose replacement for human beings.
Yeah. I particularly hate the handwavium which makes this sound like it’s super simple, just make the ASI, have it churn out labour for us, surely human society will adapt just nicely to the new state of things and chill. It’s easy to say this if you think you’re going to be the one in charge of the ASI because you’re a CEO or big shot in some company (you may still be vastly overestimating your chances of controlling the ASI of course but that’s just plain old hubris). But not so easy to believe it if instead you’re more the kind of person who usually gets the short end of the stick. Like, as much as we may celebrate automation, it DOES often have hugely disruptive effects. There’s a lot of pain hidden in that “some jobs are destroyed but others are created so it evens out”. And that’s not even a fraction of the pain that would be possible if ALL jobs were destroyed and never replaced and we would somehow have to find a way to deal with that.
And that’s, again, just the most optimistic scenario. The more pessimistic one is more along the lines of “you’re a megatherium and these strange new hairless apes with sticks have started flooding in from the north”.
The clip of the most touching part of the interview: “Well, I don’t think it’s interesting. I think it’s terrifying. I hate it. I think about it practically all the time, every single day. And it overwhelms me and depresses me in a way that I haven’t been depressed for a very long time.”
At the time of Hofstadter’s Singularity Summit talk , I wondered why he wasn’t “getting with the program”, and it became clear he was a mysterian: He believed—without being a dualist -- that some things, like the mind, are ultimately, basically, essentially, impossible to understand or describe.
This 2023 interview shows that the new generation of AI has done more than chagne his mind about the potential of AI: it has struck at the core of his mysterianism
He was only a de facto mysterian: thought mind is so complicated that it may as well be mysterious (but ofc he believed it’s ultimately just physics). This position is updateable, and he clearly updated.
Gwern’s comment makes it clear to me that Hofstadter has never been a mysterian.
It has become slightly more plausible that Melanie Mitchell could come around.
But only slightly. It appears that Hofstadter’s doubts have been building for a long time in private, even to organizing informal conferences/meetings about it, to an extent that his op-eds don’t convey (compare his comments in OP to his comments published in the Atlantic just a week before! they are so drastically different I was wondering if this was some sort of bizarre deepfake prank, but some cursory searching made it seemed legit and no one like Mitchell was saying it was fake and the text sounds like Hofstadter). On Twitter, John Teets helpfully notes that Mitchell has a 2019 book Artificial Intelligence: A Guide for Thinking Humans where she records some private Hofstadter material I was unfamiliar with:
That is, whatever the snarky “don’t worry, it can’t happen” tone of his public writings about DL has been since ~2010, Hofstadter has been saying these things in private for at least a decade*, starting somewhere around Deep Blue which clearly falsified a major prediction of his, and his worries about the scaling paradigm intensifying ever since; what has happened is that only one of two paradigms can be true, and Hofstadter has finally flipped to the other paradigm (with ChatGPT-3.5 and then GPT-4 apparently being the straws that broke the camel’s back). Mitchell, however, has heard all of this firsthand long before this podcast and appears to be completely immune to Hofstadter’s concerns (publicly), so I wouldn’t expect it to change her mind.
* I wonder what other experts & elites have different views on AI than their public statements would lead you to believe?
† Hofstadter’s semi-mysterian/irreducible-complexity view:
So the question becomes, why the front of optimism, even after this conversation?
Also on Twitter, Experimental Learning highlights a 2022-11-01 podcast by Melanie Mitchell which confirms the book description, “Increments Podcast: #45---4 Central Fallacies of AI Research (with Melanie Mitchell)”:
(The DARPA cat example is a weird one. If I’d heard of it, I’d long since forgotten it, and I’m not sure why she’d put so much weight on it; $5m was a drop in the bucket then for chip development - $5m often doesn’t even cover simple NREs when it comes to chip designing/fabbing—especially compared to Blue Brain, and it’s not like one could train useful spiking networks in the first place. I hope she doesn’t really put as much weight on that as a reason to dismiss DL scaling as she seems to.)
I think the conferences Mitchell refers to are the same ones mentioned by Chalmers 2010:
(The 1 April 2000 conference was covered at length by Ellen Ullman; it’s an interesting piece, if only for showing how far the AI zeitgeist was then from now in 2003.)
Another Mitchell followup: https://www.science.org/doi/10.1126/science.adj5957 tldr: argues that LLMs still aren’t intelligent and explains away everything they do as dataset contamination, bad benchmarks and sloppy evaluation, or shallow heuristics.
I heard something like this might be true for Yann also; like, allegedly being more worried about extinction-risk-from-AI in private, but then publicly doing the same snarky tweets.
This seems doubtful to me; if Yan truly believed that AI was an imminent extinction risk, or even thought it was credible, what would Yann be hoping to do or gain by ridiculing people who are similarly worried?
It often crosses my mind that public discourse about AI safety might not be useful. Tell men that AGI is powerful and they’ll start trying harder to acquire it. Tell legislators and, perhaps Yann thinks they’ll just start an arms race and complicate the work and not do much else.
I wonder if that’s what he’s thinking.
That’s also my confusion, yes.
I could imagine someone suppressing their alignment fears temporarily, to work their way up to a position of power in a capabilities lab and then steer outcomes from there.
But that doesn’t seem to work, since:
The top AI capabilities labs (OpenAI, DeepMind, Anthropic) are more vocal about capabilities. Meta AI is a follow-the-leader lab anyway.
I don’t think “bringing up concerns later, instead of now” is a strategically great way to do this. I don’t know a ton about the politics of historical programs for e.g. atomic weapons and bioweapons. But based on my cursory knowledge, I don’t think “be worried in secret” is anything like a slam-dunk for those situations.
Yann, specifically, is already the Chief AI Person at Meta/Facebook! Unless Meta is really quick to fire people (or Yann is angling for Zuckerberg’s position), what more career capital could he gain at this stage?
I wonder how many AI experts hold back their thoughts because they remember what happened to Copernicus when he presented that the Earth was not the center of the universe. Thank you for your post. I’m new here and am, therefore, not permitted to upvote it, but I would, if I could.
What was the argument that being feed-forward limited the potential for deep thought in principle? It makes sense that multi-directional nets could do more with fewer neurons but Hofstader seemed to think there were things that feed-forward system fundamentally couldn’t do.
He explained a bunch of his position on this in Godel, Escher, Bach. If I remember correctly, it describes the limits of primitive recursive and general recursive functions this in chapter XIII. The basic idea (again, if I remember), is that a proof system can only reason about itself if its general recursive, and will always be able to reason about itself if its general recursive. Lots of what we see that makes humanity special compared to computers has to do with people having feelings and emotions and self-concepts, and reflection about past situations & thoughts. All things that really seem to require deep levels of recursion (this is a far shallower statement than what’s actually written in the book). Its strange to us then that ChatGPT can mimic those same outputs with the only recursive element of its thought being that it can pass 16 bits to its next running.
I would name activations for all previous tokens as the relevant “element of thought” here that gets passed, and this can be gigabytes.
From how the quote looks, I think his gripe is with the possibility of in-context learning, where human-like learning happens without anything about how the network works (neither its weights nor previous token states) being ostensibly updated.
I don’t understand this. Something is being updated when humans or LLMs learn, no?
For every token, model activations are computed once when the token is encountered and then never explicitly revised → “only [seems like it] goes in one direction”
David Brooks’s 2023-07-13 NYT column covers this post’s excerpts and the original podcast, and includes quotes/paraphrases of Brooks’s phonecall interview with Hofstadter about it. “‘Human Beings Are Soon Going to Be Eclipsed’”:
Hofstadter’s long-time associate/friend/co-author, Daniel Dennett, has discussed Hofstadter’s change of heart in a recent (December 2023?) Theories of Everything podcast/interview: https://www.youtube.com/watch?v=bH553zzjQlI&t=7195s
I have not watched it myself but AI Safety Memes has excerpted it as follows:
HN comments.
Ben Goertzel:
It’s interesting that he seems so in despair over this now. To the extent that he’s worried about existential/catastrophic risks, I wonder if he is unaware of efforts to mitigate those, or if he is aware but thinks they are hopeless (or at least not guaranteed to succeed, which—fair enough). To the extent that he’s more broadly worried about human obsolescence (or anyway something more metaphysical), well, there are people trying to slow/stop AI, and others trying to enhance human capabilities—maybe he’s pessimistic about those efforts, too.
I am working on human capability enhancement via genetics. I think it’s quite plausible that we could create humans smarter than any that have ever lived within a decade. But even I think that digital intelligence wins in the end.
Like it just seems obvious to me. The only reason I’m even working in the field is because I think that enhanced humans could play an extremely critical role in the development of aligned AI. Of course this requires time for them to grow up and do research, which we are increasingly short of. But in case AGI takes longer than projected or we get our act together and implement a ban on AI capabilities improvements until alignment is solved, it still seems worth continuing the work to me.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?