the arguments for short timelines are definitely weaker than their proponents usually assume, but they aren’t totally vibes based
Each person with short timelines can repeat sentences that were generated by a legitimate reason to expect short timelines, but many of them did not generate any of those sentences themselves as the result of trying to figure out when AGI would come; their repeating those sentences is downstream of their timelines. In that sense, for many such people, short timelines actually are totally vibes based.
In that sense, for many such people, short timelines actually are totally vibes based.
I dispute this characterization. It’s normal and appropriate for people’s views to update in response to the arguments produced by others.
Sure, sometimes people most parrot other people’s views, without either developing them independently or even doing evaluatory checks to see if those views seem correct. But most of the time, I think people are doing those checks?
Speaking for myself, most of my views on timelines are downstream of ideas that I didn’t generate myself. But I did think about those ideas, and evaluate if they seemed true.
No. You can tell because they can’t have an interesting conversation about it, because they don’t have surrounding mental content (such as analyses of examples that stand up to interrogation, or open questions, or cruxes that aren’t stupid). (This is in contrast to several people who can have an interesting conversation about, even if I think they’re wrong and making mistakes and so on.)
But I did think about those ideas, and evaluate if they seemed true.
Of course I can’t tell from this sentence, but I’m pretty skeptical both of you in particular and of other people in the broad reference class, that most of them have done this in a manner that really does greatly attenuate the dangers of deference.
Why? Forecasting the future is hard, and I expect surprises that deviate from my model of how things will go. But o1 and o3, seem like pretty blatant evidence that reduced my uncertainty a lot. On pretty simple heuristics, it looks like earth now knows how to make a science and engineering superintelligence: by scaling reasoning modes in a self-play-ish regime.
I would take a bet with you about what we expect to see in the next 5 years. But more than that, what kind of epistemology do you think I should be doing that I’m not?
o1/o3/R1/R1-Zero seem to me like evidence that “scaling reasoning models in a self-play-ish regime” can reach superhuman performance on some class of tasks, with properties like {short horizons, cheap objective verifiability, at most shallow conceptual innovation needed} or maybe some subset thereof. This is important! But, for reasons similar to this part of Tsvi’s post, it’s a lot less apparent to me that it can get to superintelligence at all science and engineering tasks.
I can’t tell what you mean by much of this (e.g. idk what you mean by “pretty simple heuristics” or “science + engineering SI” or “self-play-ish regime”). (Not especially asking you to elaborate.) Most of my thoughts are here, including the comments:
I would take a bet with you about what we expect to see in the next 5 years.
Not really into formal betting, but what are a couple Pareto[impressive, you’re confident we’ll see within 5 years] things?
But more than that, what kind of epistemology do you think I should be doing that I’m not?
Come on, you know. Actually doubt, and then think it through.
I mean, I don’t know. Maybe you really did truly doubt a bunch. Maybe you could argue me from 5% omnicide in next ten years to 50%. Go ahead. I’m speaking from informed priors and impressions.
I think this is true but also that “most people’s reasons for believing X are vibes-based” is true for almost any X that is not trivially verifiable. And also that this way of forming beliefs works reasonably well in many cases. This doesn’t contradict anything you’re saying but feels worth adding, like I don’t think AI timelines are an unusual topic in that regard.
E.g. there are many activities that many people engage in frequently—eating, walking around, reading, etc etc. Knowledge and skill related to those activities is usually not vibes-based, or only half vibes-based, or something, even if not trivially verifiable. For example, after a few times accidentally growing mold on some wet clothes or under a sink, very many people learn not to leave areas wet.
E.g. anyone who studies math seriously must learn to verify many very non-trivial things themselves. (There will also be many things they will believe partly based on vibes.)
I don’t think AI timelines are an unusual topic in that regard.
In that regard, technically, yes, but it’s not very comparable. It’s unusual in that it’s a crucial question that affects very many people’s decisions. (IIRC, EVERY SINGLE ONE of the >5 EA / LW / X-derisking adjacent funder people that I’ve talked to about human intelligence enhancement says “eh, doesn’t matter, timelines short”.) And it’s in an especially uncertain field, where consensus should much less strongly be expected to be correct. And it’s subject to especially strong deference and hype dynamics and disinformation. For comparison, you can probably easily find entire communities in which the vibe is very strongly “COVID came from the wet market” and others where it’s very strongly “COVID came from the lab”. You can also find communities that say “AGI a century away”. There are some questions where the consensus is right for the right reasons and it’s reasonable to trust the consensus on some class of beliefs. But vibes-based reasoning is just not robust, and nearly all the resources supposedly aimed at X-derisking in general are captured by a largely vibes-based consensus.
I definitely agree with this, but in their defense, this is to be expected, especially in fast growing fields.
Model building is hard, and specialization generally beats trying to deeply understand something in general, so it’s not that surprising that many people won’t understand why, and this will be the case regardless of the truth value of timelines claims.
Each person with short timelines can repeat sentences that were generated by a legitimate reason to expect short timelines, but many of them did not generate any of those sentences themselves as the result of trying to figure out when AGI would come; their repeating those sentences is downstream of their timelines. In that sense, for many such people, short timelines actually are totally vibes based.
I dispute this characterization. It’s normal and appropriate for people’s views to update in response to the arguments produced by others.
Sure, sometimes people most parrot other people’s views, without either developing them independently or even doing evaluatory checks to see if those views seem correct. But most of the time, I think people are doing those checks?
Speaking for myself, most of my views on timelines are downstream of ideas that I didn’t generate myself. But I did think about those ideas, and evaluate if they seemed true.
No. You can tell because they can’t have an interesting conversation about it, because they don’t have surrounding mental content (such as analyses of examples that stand up to interrogation, or open questions, or cruxes that aren’t stupid). (This is in contrast to several people who can have an interesting conversation about, even if I think they’re wrong and making mistakes and so on.)
Of course I can’t tell from this sentence, but I’m pretty skeptical both of you in particular and of other people in the broad reference class, that most of them have done this in a manner that really does greatly attenuate the dangers of deference.
(I endorse personal call outs like this one.)
Why? Forecasting the future is hard, and I expect surprises that deviate from my model of how things will go. But o1 and o3, seem like pretty blatant evidence that reduced my uncertainty a lot. On pretty simple heuristics, it looks like earth now knows how to make a science and engineering superintelligence: by scaling reasoning modes in a self-play-ish regime.
I would take a bet with you about what we expect to see in the next 5 years. But more than that, what kind of epistemology do you think I should be doing that I’m not?
To be more object-level than Tsvi:
o1/o3/R1/R1-Zero seem to me like evidence that “scaling reasoning models in a self-play-ish regime” can reach superhuman performance on some class of tasks, with properties like {short horizons, cheap objective verifiability, at most shallow conceptual innovation needed} or maybe some subset thereof. This is important! But, for reasons similar to this part of Tsvi’s post, it’s a lot less apparent to me that it can get to superintelligence at all science and engineering tasks.
I can’t tell what you mean by much of this (e.g. idk what you mean by “pretty simple heuristics” or “science + engineering SI” or “self-play-ish regime”). (Not especially asking you to elaborate.) Most of my thoughts are here, including the comments:
https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce
Not really into formal betting, but what are a couple Pareto[impressive, you’re confident we’ll see within 5 years] things?
Come on, you know. Actually doubt, and then think it through.
I mean, I don’t know. Maybe you really did truly doubt a bunch. Maybe you could argue me from 5% omnicide in next ten years to 50%. Go ahead. I’m speaking from informed priors and impressions.
I think this is true but also that “most people’s reasons for believing X are vibes-based” is true for almost any X that is not trivially verifiable. And also that this way of forming beliefs works reasonably well in many cases. This doesn’t contradict anything you’re saying but feels worth adding, like I don’t think AI timelines are an unusual topic in that regard.
Broadly true, I think.
I’d probably quibble a lot with this.
E.g. there are many activities that many people engage in frequently—eating, walking around, reading, etc etc. Knowledge and skill related to those activities is usually not vibes-based, or only half vibes-based, or something, even if not trivially verifiable. For example, after a few times accidentally growing mold on some wet clothes or under a sink, very many people learn not to leave areas wet.
E.g. anyone who studies math seriously must learn to verify many very non-trivial things themselves. (There will also be many things they will believe partly based on vibes.)
In that regard, technically, yes, but it’s not very comparable. It’s unusual in that it’s a crucial question that affects very many people’s decisions. (IIRC, EVERY SINGLE ONE of the >5 EA / LW / X-derisking adjacent funder people that I’ve talked to about human intelligence enhancement says “eh, doesn’t matter, timelines short”.) And it’s in an especially uncertain field, where consensus should much less strongly be expected to be correct. And it’s subject to especially strong deference and hype dynamics and disinformation. For comparison, you can probably easily find entire communities in which the vibe is very strongly “COVID came from the wet market” and others where it’s very strongly “COVID came from the lab”. You can also find communities that say “AGI a century away”. There are some questions where the consensus is right for the right reasons and it’s reasonable to trust the consensus on some class of beliefs. But vibes-based reasoning is just not robust, and nearly all the resources supposedly aimed at X-derisking in general are captured by a largely vibes-based consensus.
I definitely agree with this, but in their defense, this is to be expected, especially in fast growing fields.
Model building is hard, and specialization generally beats trying to deeply understand something in general, so it’s not that surprising that many people won’t understand why, and this will be the case regardless of the truth value of timelines claims.