However, ‘timelines’ is not the interesting question.
Well, they are the decision-relevant question. At some point timelines get short enough that it’s pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children. At some point timelines get short enough that it becomes worthwhile to engage in risky activities like skydiving, motorcycling, or taking drugs (because the less time you have left the fewer expected minutes of life you lose from risking your life). Etc.
But yes, the thing where you discard your model is not great.
(Also, I personally have never generated a timeline using a model; I do something like Focusing on the felt senses of which numbers feel right. This is the final step in Eliezer’s “do the math, then burn the math and go with your gut” thing.)
I don’t mean “Discussion of timelines is not useful”. I mean it is not the central point nor should it be the main part of conversation.
Here’s three quick variables that would naturally go into a timelines model, but in fact are more important because of their strategic implications.
Can ML produce human-level AGI?
Timelines implications: If the current research paradigm will continue to AGI, then we can do fairly basic extrapolation to determine timelines.
Strategic implications: If the current research paradigm will continue to AGI, this tells us important things about what alignment strategies to pursue, what sorts of places to look for alignment researchers in, and what sort of relationship to build with academia.
How flexible is are the key governments (US, Russia, China, etc)?
Timelines implications: This will let us know how much speed-up they can give to timelines (e.g. by funding, by pushing on race dynamics).
Strategic implications: This has a lot of impact in terms of how much we should start collaborating with governments, what information we should actively try to propagate through government, whether some of us should take governmental roles, etc.
Will an intelligence explosion be local or dispersed?
Timelines implications: If intelligence explosions can be highly local it could be that a take-off is happening right now that we just can’t see, and so our timelines should be shorter.
Strategic Implications: The main reason I might want to know about local v dispersed is because I need to know what sorts of information flows to set up between government, industry, and academia.
The word I’d object to in the sentence “Well, they are the decision-relevant question” is the word ‘the’. They are ‘a’ decision-relevant question, but not at all obviously ‘the’, nor even obviously one of the first five most important ones (I don’t have a particular list in mind, but I expect the top five are mostly questions about alignment agendas and the structure of intelligence).
---
(Also, I personally have never generated a timeline using a model; I do something like Focusing on the felt senses of which numbers feel right. This is the final step in Eliezer’s “do the math, then burn the math and go with your gut” thing.)
Yeah, I agree that it’s hard to make the most subtle intuitions explicit, and that nonetheless you should trust them. I also want to say that the do-the-math part first is pretty useful ;-)
Also, I personally have never generated a timeline using a model; I do something like Focusing on the felt senses of which numbers feel right. This is the final step in Eliezer’s “do the math, then burn the math and go with your gut” thing.
This does not seem like the sort of process that would generate an output suitable for shared map-making (i.e. epistemics). It seems like a process designed to produce an answer optimized for your local incentives, which will of course be mostly social ones, since your S1 is going to be tracking those pretty intently.
I’m also confused by this response. I don’t mean to imply that the model-building version of the thing is bad, but I did want to contribute as a data point that I’ve never done it.
I don’t have full introspective access to what my gut is tracking but it mostly doesn’t feel to me like it’s tracking social incentives; what I’m doing is closer to an inner sim kind of thing along the lines of “how surprised would I be if 30 years passed and we were all still alive?” Of course I haven’t provided any evidence for this other than claiming it, and I’m not claiming that people should take my gut timelines very seriously, but I can still pay attention to them.
That seems substantially better than Focusing directly on the numbers. Thanks for explaining. I’m still pretty surprised if your anticipations are (a) being produced by a model that’s primarily epistemic rather than socially motivated, and (b) hard to unpack into a structured model without a lot more work than what’s required to generate them in the first place.
Well, they are the decision-relevant question. At some point timelines get short enough that it’s pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children. At some point timelines get short enough that it becomes worthwhile to engage in risky activities like skydiving, motorcycling, or taking drugs (because the less time you have left the fewer expected minutes of life you lose from risking your life). Etc.
This helped me understand why people interpret the Rationalists as just another apocalypse cult. The decision-relevant question is not, apparently, what’s going to happen as increasing amounts of cognitive labor are delegated to machines and what sort of strategies are available to make this transition go well and what role one might personally play in instantiating these strategies, but instead, simply how soon the inevitable apocalypse is going to happen and accordingly what degree of time preference we should have.
I’m confused by this response. I didn’t mention this to imply that I’m not interested in the model-building and strategic discussions—they seem clearly good and important—but I do think there’s a tendency for people to not connect their timelines with their actual daily lives, which are in fact nontrivially affected by their timelines, and I track this level of connection to see who’s putting their money where their mouth is in this particular way. E.g. I’ve been asking people for the last few years whether they’re saving for retirement for this reason.
Calling them the decision-relevant question implies that you think this sort of thing is more important than the other questions. This is very surprising—as is the broader emphasis on timelines over other strategic considerations that might help us favor some interventions over others—if you take the AI safety narrative literally. It’s a lot less surprising if you think explicit narratives are often cover stories people don’t mean literally, and notice that one of the main ways factions within a society become strategically distinct is by coordinating around distinct time preferences.
In particular, inducing high time preference in people already disposed to trusting you as a leader seems like a pretty generally applicable strategy for getting them to defer to your authority. “Emergency powers,” wartime authority and rallying behind the flag, doomsday cults, etc.
Yeah, “the” was much too strong and that is definitely not a thing I think. I don’t appreciate the indirect accusation that I’m trying to get people to defer to my authority by inducing high time preference in them.
For what it’s worth, I read Benquo as saying not “Qiaochu is trying to do this” but something more like “People who see rationalism as a cult are likely to think the cult leaders are trying to do this”. Though I can see arguments for reading it the other way.
I anticipated a third thing, which is that “at least some people who are talking about short timelines are at least somewhat unconsciously motivated to do so for this reason, Qiaochu is embedded in the social web that may be playing a role in shaping his intuitions.”
Well, they are the decision-relevant question. At some point timelines get short enough that it’s pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children. At some point timelines get short enough that it becomes worthwhile to engage in risky activities like skydiving, motorcycling, or taking drugs (because the less time you have left the fewer expected minutes of life you lose from risking your life). Etc.
But yes, the thing where you discard your model is not great.
(Also, I personally have never generated a timeline using a model; I do something like Focusing on the felt senses of which numbers feel right. This is the final step in Eliezer’s “do the math, then burn the math and go with your gut” thing.)
I don’t mean “Discussion of timelines is not useful”. I mean it is not the central point nor should it be the main part of conversation.
Here’s three quick variables that would naturally go into a timelines model, but in fact are more important because of their strategic implications.
Can ML produce human-level AGI?
Timelines implications: If the current research paradigm will continue to AGI, then we can do fairly basic extrapolation to determine timelines.
Strategic implications: If the current research paradigm will continue to AGI, this tells us important things about what alignment strategies to pursue, what sorts of places to look for alignment researchers in, and what sort of relationship to build with academia.
How flexible is are the key governments (US, Russia, China, etc)?
Timelines implications: This will let us know how much speed-up they can give to timelines (e.g. by funding, by pushing on race dynamics).
Strategic implications: This has a lot of impact in terms of how much we should start collaborating with governments, what information we should actively try to propagate through government, whether some of us should take governmental roles, etc.
Will an intelligence explosion be local or dispersed?
Timelines implications: If intelligence explosions can be highly local it could be that a take-off is happening right now that we just can’t see, and so our timelines should be shorter.
Strategic Implications: The main reason I might want to know about local v dispersed is because I need to know what sorts of information flows to set up between government, industry, and academia.
The word I’d object to in the sentence “Well, they are the decision-relevant question” is the word ‘the’. They are ‘a’ decision-relevant question, but not at all obviously ‘the’, nor even obviously one of the first five most important ones (I don’t have a particular list in mind, but I expect the top five are mostly questions about alignment agendas and the structure of intelligence).
---
Yeah, I agree that it’s hard to make the most subtle intuitions explicit, and that nonetheless you should trust them. I also want to say that the do-the-math part first is pretty useful ;-)
This does not seem like the sort of process that would generate an output suitable for shared map-making (i.e. epistemics). It seems like a process designed to produce an answer optimized for your local incentives, which will of course be mostly social ones, since your S1 is going to be tracking those pretty intently.
I’m also confused by this response. I don’t mean to imply that the model-building version of the thing is bad, but I did want to contribute as a data point that I’ve never done it.
I don’t have full introspective access to what my gut is tracking but it mostly doesn’t feel to me like it’s tracking social incentives; what I’m doing is closer to an inner sim kind of thing along the lines of “how surprised would I be if 30 years passed and we were all still alive?” Of course I haven’t provided any evidence for this other than claiming it, and I’m not claiming that people should take my gut timelines very seriously, but I can still pay attention to them.
That seems substantially better than Focusing directly on the numbers. Thanks for explaining. I’m still pretty surprised if your anticipations are (a) being produced by a model that’s primarily epistemic rather than socially motivated, and (b) hard to unpack into a structured model without a lot more work than what’s required to generate them in the first place.
This helped me understand why people interpret the Rationalists as just another apocalypse cult. The decision-relevant question is not, apparently, what’s going to happen as increasing amounts of cognitive labor are delegated to machines and what sort of strategies are available to make this transition go well and what role one might personally play in instantiating these strategies, but instead, simply how soon the inevitable apocalypse is going to happen and accordingly what degree of time preference we should have.
I’m confused by this response. I didn’t mention this to imply that I’m not interested in the model-building and strategic discussions—they seem clearly good and important—but I do think there’s a tendency for people to not connect their timelines with their actual daily lives, which are in fact nontrivially affected by their timelines, and I track this level of connection to see who’s putting their money where their mouth is in this particular way. E.g. I’ve been asking people for the last few years whether they’re saving for retirement for this reason.
Calling them the decision-relevant question implies that you think this sort of thing is more important than the other questions. This is very surprising—as is the broader emphasis on timelines over other strategic considerations that might help us favor some interventions over others—if you take the AI safety narrative literally. It’s a lot less surprising if you think explicit narratives are often cover stories people don’t mean literally, and notice that one of the main ways factions within a society become strategically distinct is by coordinating around distinct time preferences.
In particular, inducing high time preference in people already disposed to trusting you as a leader seems like a pretty generally applicable strategy for getting them to defer to your authority. “Emergency powers,” wartime authority and rallying behind the flag, doomsday cults, etc.
Yeah, “the” was much too strong and that is definitely not a thing I think. I don’t appreciate the indirect accusation that I’m trying to get people to defer to my authority by inducing high time preference in them.
For what it’s worth, I read Benquo as saying not “Qiaochu is trying to do this” but something more like “People who see rationalism as a cult are likely to think the cult leaders are trying to do this”. Though I can see arguments for reading it the other way.
I anticipated a third thing, which is that “at least some people who are talking about short timelines are at least somewhat unconsciously motivated to do so for this reason, Qiaochu is embedded in the social web that may be playing a role in shaping his intuitions.”
This is what I meant.