I feel a bit confused about where you think we meta-disagree here, meta-policy-wise. If you have a thesis about the sort of things I’m liable to disagree with you about, because you think you’re more familiar with the facts on the ground, can’t you write up Paul’s View of the Next Five Years and then if I disagree with it better yet, but if not, you still get to be right and collect Bayes points for the Next Five Years?
I mean, it feels to me like this should be a case similar to where, for example, I think I know more about macroeconomics than your typical EA; so if I wanted to expend the time/stamina points, I could say a bunch of things I consider obvious and that contradict hot takes on Twitter and many EAs would go “whoa wait really” and then I could collect Bayes points later and have performed a public service, even if nobody showed up to disagree with me about that. (The reason I don’t actually do this… is that I tried; I keep trying to write a book about basic macro, only it’s the correct version explained correctly, and have a bunch of isolated chapters and unfinished drafts.) I’m also trying to write up my version of The Next Five Years assuming the world starts to end in 2025, since this is not excluded by my model; but writing in long-form requires stamina and I’ve been tired of late which is part of why I’ve been having Discord conversations instead.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do, and so I should be able to state my own prediction about that topic and bet with you about that; or, alternatively, that I should retract some thing I said recently which implies that. And so, you shouldn’t need to have to do all the work to write up your forecasts generally, and it’s unfair that I’m trying to make you do all that work. Check? If so, I don’t yet see the derivation chain on this meta-level point.
I think the Hansonian viewpoint—which I consider another gradualist viewpoint, and whose effects were influential on early EA and which I think are still lingering around in EA—seemed surprised by AlphaGo and Alpha Zero, when you contrast its actual advance language with what actually happened. Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then; and I also think that “there’s always a smooth abstraction in hindsight, so what, there’ll be one of those when the world ends too”, is a huge big deal in practice with respect to the future being unpredictable. From this, you seem to derive that I should already know what to bet with you about, and are annoyed by how I’m playing coy; because if I don’t bet with you right now, I should retract the statement that I think gradualists were surprised; but to me I’m not following the sequitur there.
Or maybe I’m just entirely misinterpreting the flow of your thoughts here.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do.
I’ve said I’m happy to bet about anything, and listed some particular questions I’d bet about where I expect you to be wronger. If you had issued the same challenge to me, I would have picked one of the things and we would have already made some bets. So that’s why I feel like the ball is in your court to say what things you’re willing to make forecasts about.
That said, I don’t know if making bets is at all a good use of time. I’m inclined to do it because I feel like your view really should be making different predictions (and I feel like you are participating in good faith and in fact would end up making different predictions). And I think it’s probably more promising than trying to hash out the arguments since at this point I feel like I mostly know your position and it’s incredibly slow going. But it seems very plausible that the right move is just to agree to disagree and not spend time on this. In that case it was particularly bad of me to try to claim the epistemic high ground. I can’t really defend myself there, but can explain by saying that I found your vitriolic reading of takeoff speeds pretty condescending and frustrating and, given that I think you are more wrong than right, wanted a nice way to demonstrate that.
I’ve mentioned the kinds of things I think your model will forecast badly, and suggested that we bet about them in particular:
I think you generally overestimate the rate of trend breaks on measurable trends. So let’s pick some trends and estimate probability of trend breaks.
I think you don’t understand in which domains trend-breaks are surprising and where they aren’t surprising, so you will be sometimes underconfident and sometimes overconfident on any given forecast. Same bet as last time.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I think you are underconfident about continuing scale-up in AI. So we can bet about future spending, size of particular labs, size of the ML field.
I think you overestimate DeepMind’s advantage over the rest of the field and so will make bad forecasts about where any given piece of progress comes from.
I think your AI timelines are generally too short. You can point to cool stuff happening as a vindication for your view, and there will certainly be some cool stuff happening, but I think if we actually get concrete you are just going to make worse predictions.
My uncharitable read on many of these domains is that you are saying “Sure, I think that Paul might have somewhat better forecasts than me on those questions, but why is that relevant to AGI?”
In that case it seems like the situation is pretty asymmetrical. I’m claiming that my view of AGI is related to beliefs and models that also bear on near-term questions, and I expect to make better forecasts than you in those domains because I have more accurate beliefs/models. If your view of AGI is unrelated to any near-term questions where we disagree, then that seems like an important asymmetry.
I suspect that indeed EY’s model has a limited ability to make near-term predictions, so that yes, the situation is asymmetrical. But I suspect his view is similar to my view, so I don’t think EY is wrong. But I am confused about why EY (i) hasn’t replied himself and (ii) in general, doesn’t communicate more clearly on this topic.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I wouldn’t be especially surprised by waifutechnology or machine translation jumping to newly accessible domains (the thing I care about and you shrug about (until the world ends)), but is that likely to exhibit a visible economic discontinuity in profits (which you care about and I shrug about (until the world ends))? There’s apparently already mass-scale deployment of waifutech in China to forlorn male teenagers, so maybe you’ll say the profits were already precedented. Google offers machine translation now, even though they don’t make much obvious measurable profit on that, but maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue. Or perhaps you want to say that “almost all AI profits” will come from robotics over the same period. Or maybe I misunderstand your viewpoint, and if you said something concrete about the stuff you care about, I would manage to disagree with that; or maybe you think that waifutech suddenly getting much more charming with the next generation of text transformers is something you already know enough to rule out; or maybe you think that 2024′s waifutech should definitely be able to do some observable surface-level thing it can’t do now.
I’d be happy to disagree about romantic chatbots or machine translation. I’d have to look into it more to get a detailed sense in either, but I can guess. I’m not sure what “wouldn’t be especially surprised” means, I think to actually get disagreements we need way more resolution than that so one question is whether you are willing to play ball (since presumably you’d also have to looking into to get a more detailed sense). Maybe we could save labor if people would point out the empirical facts we’re missing and we can revise in light of that, but we’d still need more resolution. (That said: what’s up for grabs here are predictions about the future, not present.)
I’d guess that machine translation is currently something like $100M/year in value, and will scale up more like 2x/year than 10x/year as DL improves (e.g. most of the total log increase will be in years with <3x increase rather than >3x increase, and 3 is like the 60th percentile of the number for which that inequality is tight).
I’d guess that increasing deployment of romantic chatbots will end up with technical change happening first followed by social change second, so the speed of deployment and change will depend on the speed of social change. At early stages of the social change you will likely see much large investment in fine-tuning for this use case, and the results will be impressive as you shift from random folks doing it to actual serious efforts. The fact that it’s driven by social rather than technical change means it could proceed at very different paces in different countries. I don’t expect anyone to make a lot of profit from this before self-driving cars, for example I’d be pretty surprised if this surpassed $1B/year of revenue before self-driving cars passed $10B/year of revenue. I have no idea what’s happening in China. It would be fairly surprising to me if there was currently an actually-compelling version of the technology—which we could try operationalize as something like how bad your best available romantic relationship with humans has to be, or how lonely you’d have to be, or how short-sighted you’d have to be, before it’s appealing. I don’t have strong views about a mediocre product with low activation energy that’s nevertheless used by many (e.g. in the same way we see lots of games with mediocre hedonic value and high uptake, or lots of passive gambling).
Thanks for continuing to try on this! Without having spent a lot of labor myself on looking into self-driving cars, I think my sheer impression would be that we’ll get $1B/yr waifutech before we get AI freedom-of-the-road; though I do note again that current self-driving tech would be more than sufficient for $10B/yr revenue if people built new cities around the AI tech level, so I worry a bit about some restricted use-case of self-driving tech that is basically possible with current tech finding some less regulated niche worth a trivial $10B/yr. I also remark that I wouldn’t be surprised to hear that waifutech is already past $1B/yr in China, but I haven’t looked into things there. I don’t expect the waifutech to transcend my own standards for mediocrity, but something has to be pretty good before I call it more than mediocre; do you think there’s particular things that waifutech won’t be able to do?
My model permits large jumps in ML translation adoption; it is much less clear about whether anyone will be able to build a market moat and charge big prices for it. Do you have a similar intuition about # of users increasing gradually, not just revenue increasing gradually?
I think we’re still at the level of just drawing images about the future, so that anybody who came back in 5 years could try to figure out who sounded right, at all, rather than assembling a decent portfolio of bets; but I also think that just having images versus no images is a lot of progress.
Yes, I think that value added by automated translation will follow a similar pattern. Number of words translated is more sensitive to how you count and random nonsense, as is number of “users” which has even more definitional issues.
You can state a prediction about self-driving cars in any way you want. The obvious thing is to talk about programs similar to the existing self-driving taxi pilots (e.g. Waymo One) and ask when they do $X of revenue per year, or when $X of self-driving trucking is done per year. (I don’t know what AI freedom-of-the-road means, do you mean something significantly more ambitious than self-driving trucks or taxis?)
Man, the problem is that you say the “jump to newly accessible domains” will be the thing that lets you take over the world. So what’s up for dispute is the prototype being enough to take over the world rather than years of progress by a giant lab on top of the prototype. It doesn’t help if you say “I expect new things to sometimes become possible” if you don’t further say something about the impact of the very early versions of the product.
Maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue
If e.g. people were spending $1B/year developing a technology, and then after a while it jumps from 0/year to $1B/year of profit, I’m not that surprised. (Note that machine translation is radically smaller than this, I don’t know the numbers.)
I do suspect they could have rolled out a crappy version earlier, perhaps by significantly changing their project. But why would they necessarily bother doing that? For me this isn’t violating any of the principles that make your stories sound so crazy. The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
(Note: it is surprising if an industry is spending $10T/year on R&D and then jumps from $1T --> $10T of revenue in one year in a world that isn’t yet growing crazily. The surprising depends a lot on the numbers involved, and in particular on how valuable it would have been to deploy a worse version earlier and how hard it is to raise money at different scales.)
The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
Would you say that this is a good description of Suddenly Hominids but you don’t expect that to happen again, or that this is a bad description of hominids?
It’s not a description of hominids at all, no one spent any money on R&D.
I think there are analogies where this would be analogous to hominids (which I think are silly, as we discuss in the next part of this transcript). And there are analogies where this is a bad description of hominids (which I prefer).
Spending money on R&D is essentially the expenditure of resources in order to explore and optimize over a promising design space, right? That seems like a good description of what natural selection did in the case of hominids. I imagine this still sounds silly to you, but I’m not sure why. My guess is that you think natural selection isn’t relevantly similar because it didn’t deliberately plan to allocate resources as part of a long bet that it would pay off big.
I think natural selection has lots of similarities to R&D, but (i) there are lots of ways of drawing the analogy, (ii) some important features of R&D are missing in evolution, including some really important ones for fast takeoff arguments (like the existence of actors who think ahead).
If someones wants to spell out why they think evolution of hominids means takeoff is fast then I’m usually happy to explain why I disagree with their particular analogy. I think this happens in the next discord log between me and Eliezer.
Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then
It seems like you are saying that there is some measure that was continuous all along, but that it’s not obvious in advance which measure was continuous. That seems to suggest that there are a bunch of plausible measures you could suggest in advance, and lots of interesting action will be from changes that are discontinuous changes on some of those measures. Is that right?
If so, don’t we get out a ton of predictions? Like, for every particular line someone thinks might be smooth, the gradualist has a higher probability on it being smooth than you would? So why can’t I just start naming some smooth lines (like any of the things I listed in the grandparent) and then we can play ball?
If not, what’s your position? Is it that you literally can’t think of the possible abstractions that would later make the graph smooth? (This sounds insane to me.)
I feel a bit confused about where you think we meta-disagree here, meta-policy-wise. If you have a thesis about the sort of things I’m liable to disagree with you about, because you think you’re more familiar with the facts on the ground, can’t you write up Paul’s View of the Next Five Years and then if I disagree with it better yet, but if not, you still get to be right and collect Bayes points for the Next Five Years?
I mean, it feels to me like this should be a case similar to where, for example, I think I know more about macroeconomics than your typical EA; so if I wanted to expend the time/stamina points, I could say a bunch of things I consider obvious and that contradict hot takes on Twitter and many EAs would go “whoa wait really” and then I could collect Bayes points later and have performed a public service, even if nobody showed up to disagree with me about that. (The reason I don’t actually do this… is that I tried; I keep trying to write a book about basic macro, only it’s the correct version explained correctly, and have a bunch of isolated chapters and unfinished drafts.) I’m also trying to write up my version of The Next Five Years assuming the world starts to end in 2025, since this is not excluded by my model; but writing in long-form requires stamina and I’ve been tired of late which is part of why I’ve been having Discord conversations instead.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do, and so I should be able to state my own prediction about that topic and bet with you about that; or, alternatively, that I should retract some thing I said recently which implies that. And so, you shouldn’t need to have to do all the work to write up your forecasts generally, and it’s unfair that I’m trying to make you do all that work. Check? If so, I don’t yet see the derivation chain on this meta-level point.
I think the Hansonian viewpoint—which I consider another gradualist viewpoint, and whose effects were influential on early EA and which I think are still lingering around in EA—seemed surprised by AlphaGo and Alpha Zero, when you contrast its actual advance language with what actually happened. Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then; and I also think that “there’s always a smooth abstraction in hindsight, so what, there’ll be one of those when the world ends too”, is a huge big deal in practice with respect to the future being unpredictable. From this, you seem to derive that I should already know what to bet with you about, and are annoyed by how I’m playing coy; because if I don’t bet with you right now, I should retract the statement that I think gradualists were surprised; but to me I’m not following the sequitur there.
Or maybe I’m just entirely misinterpreting the flow of your thoughts here.
I’ve said I’m happy to bet about anything, and listed some particular questions I’d bet about where I expect you to be wronger. If you had issued the same challenge to me, I would have picked one of the things and we would have already made some bets. So that’s why I feel like the ball is in your court to say what things you’re willing to make forecasts about.
That said, I don’t know if making bets is at all a good use of time. I’m inclined to do it because I feel like your view really should be making different predictions (and I feel like you are participating in good faith and in fact would end up making different predictions). And I think it’s probably more promising than trying to hash out the arguments since at this point I feel like I mostly know your position and it’s incredibly slow going. But it seems very plausible that the right move is just to agree to disagree and not spend time on this. In that case it was particularly bad of me to try to claim the epistemic high ground. I can’t really defend myself there, but can explain by saying that I found your vitriolic reading of takeoff speeds pretty condescending and frustrating and, given that I think you are more wrong than right, wanted a nice way to demonstrate that.
I’ve mentioned the kinds of things I think your model will forecast badly, and suggested that we bet about them in particular:
I think you generally overestimate the rate of trend breaks on measurable trends. So let’s pick some trends and estimate probability of trend breaks.
I think you don’t understand in which domains trend-breaks are surprising and where they aren’t surprising, so you will be sometimes underconfident and sometimes overconfident on any given forecast. Same bet as last time.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I think you are underconfident about continuing scale-up in AI. So we can bet about future spending, size of particular labs, size of the ML field.
I think you overestimate DeepMind’s advantage over the rest of the field and so will make bad forecasts about where any given piece of progress comes from.
I think your AI timelines are generally too short. You can point to cool stuff happening as a vindication for your view, and there will certainly be some cool stuff happening, but I think if we actually get concrete you are just going to make worse predictions.
My uncharitable read on many of these domains is that you are saying “Sure, I think that Paul might have somewhat better forecasts than me on those questions, but why is that relevant to AGI?”
In that case it seems like the situation is pretty asymmetrical. I’m claiming that my view of AGI is related to beliefs and models that also bear on near-term questions, and I expect to make better forecasts than you in those domains because I have more accurate beliefs/models. If your view of AGI is unrelated to any near-term questions where we disagree, then that seems like an important asymmetry.
I suspect that indeed EY’s model has a limited ability to make near-term predictions, so that yes, the situation is asymmetrical. But I suspect his view is similar to my view, so I don’t think EY is wrong. But I am confused about why EY (i) hasn’t replied himself and (ii) in general, doesn’t communicate more clearly on this topic.
I wouldn’t be especially surprised by waifutechnology or machine translation jumping to newly accessible domains (the thing I care about and you shrug about (until the world ends)), but is that likely to exhibit a visible economic discontinuity in profits (which you care about and I shrug about (until the world ends))? There’s apparently already mass-scale deployment of waifutech in China to forlorn male teenagers, so maybe you’ll say the profits were already precedented. Google offers machine translation now, even though they don’t make much obvious measurable profit on that, but maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue. Or perhaps you want to say that “almost all AI profits” will come from robotics over the same period. Or maybe I misunderstand your viewpoint, and if you said something concrete about the stuff you care about, I would manage to disagree with that; or maybe you think that waifutech suddenly getting much more charming with the next generation of text transformers is something you already know enough to rule out; or maybe you think that 2024′s waifutech should definitely be able to do some observable surface-level thing it can’t do now.
I’d be happy to disagree about romantic chatbots or machine translation. I’d have to look into it more to get a detailed sense in either, but I can guess. I’m not sure what “wouldn’t be especially surprised” means, I think to actually get disagreements we need way more resolution than that so one question is whether you are willing to play ball (since presumably you’d also have to looking into to get a more detailed sense). Maybe we could save labor if people would point out the empirical facts we’re missing and we can revise in light of that, but we’d still need more resolution. (That said: what’s up for grabs here are predictions about the future, not present.)
I’d guess that machine translation is currently something like $100M/year in value, and will scale up more like 2x/year than 10x/year as DL improves (e.g. most of the total log increase will be in years with <3x increase rather than >3x increase, and 3 is like the 60th percentile of the number for which that inequality is tight).
I’d guess that increasing deployment of romantic chatbots will end up with technical change happening first followed by social change second, so the speed of deployment and change will depend on the speed of social change. At early stages of the social change you will likely see much large investment in fine-tuning for this use case, and the results will be impressive as you shift from random folks doing it to actual serious efforts. The fact that it’s driven by social rather than technical change means it could proceed at very different paces in different countries. I don’t expect anyone to make a lot of profit from this before self-driving cars, for example I’d be pretty surprised if this surpassed $1B/year of revenue before self-driving cars passed $10B/year of revenue. I have no idea what’s happening in China. It would be fairly surprising to me if there was currently an actually-compelling version of the technology—which we could try operationalize as something like how bad your best available romantic relationship with humans has to be, or how lonely you’d have to be, or how short-sighted you’d have to be, before it’s appealing. I don’t have strong views about a mediocre product with low activation energy that’s nevertheless used by many (e.g. in the same way we see lots of games with mediocre hedonic value and high uptake, or lots of passive gambling).
Thanks for continuing to try on this! Without having spent a lot of labor myself on looking into self-driving cars, I think my sheer impression would be that we’ll get $1B/yr waifutech before we get AI freedom-of-the-road; though I do note again that current self-driving tech would be more than sufficient for $10B/yr revenue if people built new cities around the AI tech level, so I worry a bit about some restricted use-case of self-driving tech that is basically possible with current tech finding some less regulated niche worth a trivial $10B/yr. I also remark that I wouldn’t be surprised to hear that waifutech is already past $1B/yr in China, but I haven’t looked into things there. I don’t expect the waifutech to transcend my own standards for mediocrity, but something has to be pretty good before I call it more than mediocre; do you think there’s particular things that waifutech won’t be able to do?
My model permits large jumps in ML translation adoption; it is much less clear about whether anyone will be able to build a market moat and charge big prices for it. Do you have a similar intuition about # of users increasing gradually, not just revenue increasing gradually?
I think we’re still at the level of just drawing images about the future, so that anybody who came back in 5 years could try to figure out who sounded right, at all, rather than assembling a decent portfolio of bets; but I also think that just having images versus no images is a lot of progress.
Yes, I think that value added by automated translation will follow a similar pattern. Number of words translated is more sensitive to how you count and random nonsense, as is number of “users” which has even more definitional issues.
You can state a prediction about self-driving cars in any way you want. The obvious thing is to talk about programs similar to the existing self-driving taxi pilots (e.g. Waymo One) and ask when they do $X of revenue per year, or when $X of self-driving trucking is done per year. (I don’t know what AI freedom-of-the-road means, do you mean something significantly more ambitious than self-driving trucks or taxis?)
Man, the problem is that you say the “jump to newly accessible domains” will be the thing that lets you take over the world. So what’s up for dispute is the prototype being enough to take over the world rather than years of progress by a giant lab on top of the prototype. It doesn’t help if you say “I expect new things to sometimes become possible” if you don’t further say something about the impact of the very early versions of the product.
If e.g. people were spending $1B/year developing a technology, and then after a while it jumps from 0/year to $1B/year of profit, I’m not that surprised. (Note that machine translation is radically smaller than this, I don’t know the numbers.)
I do suspect they could have rolled out a crappy version earlier, perhaps by significantly changing their project. But why would they necessarily bother doing that? For me this isn’t violating any of the principles that make your stories sound so crazy. The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
(Note: it is surprising if an industry is spending $10T/year on R&D and then jumps from $1T --> $10T of revenue in one year in a world that isn’t yet growing crazily. The surprising depends a lot on the numbers involved, and in particular on how valuable it would have been to deploy a worse version earlier and how hard it is to raise money at different scales.)
Would you say that this is a good description of Suddenly Hominids but you don’t expect that to happen again, or that this is a bad description of hominids?
It’s not a description of hominids at all, no one spent any money on R&D.
I think there are analogies where this would be analogous to hominids (which I think are silly, as we discuss in the next part of this transcript). And there are analogies where this is a bad description of hominids (which I prefer).
Spending money on R&D is essentially the expenditure of resources in order to explore and optimize over a promising design space, right? That seems like a good description of what natural selection did in the case of hominids. I imagine this still sounds silly to you, but I’m not sure why. My guess is that you think natural selection isn’t relevantly similar because it didn’t deliberately plan to allocate resources as part of a long bet that it would pay off big.
I think natural selection has lots of similarities to R&D, but (i) there are lots of ways of drawing the analogy, (ii) some important features of R&D are missing in evolution, including some really important ones for fast takeoff arguments (like the existence of actors who think ahead).
If someones wants to spell out why they think evolution of hominids means takeoff is fast then I’m usually happy to explain why I disagree with their particular analogy. I think this happens in the next discord log between me and Eliezer.
It seems like you are saying that there is some measure that was continuous all along, but that it’s not obvious in advance which measure was continuous. That seems to suggest that there are a bunch of plausible measures you could suggest in advance, and lots of interesting action will be from changes that are discontinuous changes on some of those measures. Is that right?
If so, don’t we get out a ton of predictions? Like, for every particular line someone thinks might be smooth, the gradualist has a higher probability on it being smooth than you would? So why can’t I just start naming some smooth lines (like any of the things I listed in the grandparent) and then we can play ball?
If not, what’s your position? Is it that you literally can’t think of the possible abstractions that would later make the graph smooth? (This sounds insane to me.)