I do wish to note that we spent a fair amount of time on Discord trying to nail down what earlier points we might disagree on, before the world started to end, and these Discord logs should be going up later.
From my perspective, the basic problem is that Eliezer’s story looks a lot like “business as usual until the world starts to end sharply”, and Paul’s story looks like “things continue smoothly until their smooth growth ends the world smoothly”, and both of us have ever heard of superforecasting and both of us are liable to predict near-term initial segments by extrapolating straight lines while those are available. Another basic problem, as I’d see it, is that we tend to tell stories about very different subject matters—I care a lot less than Paul about the quantitative monetary amount invested into Intel, to the point of not really trying to develop expertise about that.
I claim that I came off better than Robin Hanson in our FOOM debate compared to the way that history went. I’d claim that my early judgments of the probable importance of AGI, at all, stood up generally better than early non-Yudkowskian EA talking about that. Other people I’ve noticed ever making correct bold predictions in this field include Demis Hassabis, for predicting that deep learning would work at all, and then for predicting that he could take the field of Go and taking it; and Dario Amodei, for predicting that brute-forcing stacking more layers would be able to produce GPT-2 and GPT-3 instead of just asymptoting and petering out. I think Paul doesn’t need to bet against me to start producing a track record like this; I think he can already start to accumulate reputation by saying what he thinks is bold and predictable about the next 5 years; and if it overlaps “things that interest Eliezer” enough for me to disagree with some of it, better yet.
From my perspective, the basic problem is that Eliezer’s story looks a lot like “business as usual until the world starts to end sharply”, and Paul’s story looks like “things continue smoothly until their smooth growth ends the world smoothly”, and both of have ever heard of superforecasting and both of us are liable to predict near-term initial segments by extrapolating straight lines while those are available.
I agree that it’s plausible that we both make the same predictions about the near future. I think we probably don’t, and there are plenty of disagreements about all kinds of stuff. But if in fact we agree, then in 5 years you shouldn’t say “and see how much the world looked like I said?”
It feels to me like it goes: you say AGI will look crazy. Then I say that sounds unlike the world of today. Then you say “no, the world actually always looks discontinuous in the ways I’m predicting and your model is constantly surprised by real stuff that happens, e.g. see transformers or AlphaGo” and then I say “OK, let’s bet about literally anything at all, you pick.”
I think it’s pretty likely that we actually do disagree about how much the world of today is boring and continuous, where my error theory is that you spend too much time reading papers and press releases that paint a misleading picture and just aren’t that familiar with what’s happening on the ground. So I expect if we stake out any random quantity we’ll disagree somewhat.
Most things just aren’t bold and predictable, they are modest disagreements. I’m not saying I have some deep secret about the world, just that you are wrong in this case.
I feel a bit confused about where you think we meta-disagree here, meta-policy-wise. If you have a thesis about the sort of things I’m liable to disagree with you about, because you think you’re more familiar with the facts on the ground, can’t you write up Paul’s View of the Next Five Years and then if I disagree with it better yet, but if not, you still get to be right and collect Bayes points for the Next Five Years?
I mean, it feels to me like this should be a case similar to where, for example, I think I know more about macroeconomics than your typical EA; so if I wanted to expend the time/stamina points, I could say a bunch of things I consider obvious and that contradict hot takes on Twitter and many EAs would go “whoa wait really” and then I could collect Bayes points later and have performed a public service, even if nobody showed up to disagree with me about that. (The reason I don’t actually do this… is that I tried; I keep trying to write a book about basic macro, only it’s the correct version explained correctly, and have a bunch of isolated chapters and unfinished drafts.) I’m also trying to write up my version of The Next Five Years assuming the world starts to end in 2025, since this is not excluded by my model; but writing in long-form requires stamina and I’ve been tired of late which is part of why I’ve been having Discord conversations instead.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do, and so I should be able to state my own prediction about that topic and bet with you about that; or, alternatively, that I should retract some thing I said recently which implies that. And so, you shouldn’t need to have to do all the work to write up your forecasts generally, and it’s unfair that I’m trying to make you do all that work. Check? If so, I don’t yet see the derivation chain on this meta-level point.
I think the Hansonian viewpoint—which I consider another gradualist viewpoint, and whose effects were influential on early EA and which I think are still lingering around in EA—seemed surprised by AlphaGo and Alpha Zero, when you contrast its actual advance language with what actually happened. Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then; and I also think that “there’s always a smooth abstraction in hindsight, so what, there’ll be one of those when the world ends too”, is a huge big deal in practice with respect to the future being unpredictable. From this, you seem to derive that I should already know what to bet with you about, and are annoyed by how I’m playing coy; because if I don’t bet with you right now, I should retract the statement that I think gradualists were surprised; but to me I’m not following the sequitur there.
Or maybe I’m just entirely misinterpreting the flow of your thoughts here.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do.
I’ve said I’m happy to bet about anything, and listed some particular questions I’d bet about where I expect you to be wronger. If you had issued the same challenge to me, I would have picked one of the things and we would have already made some bets. So that’s why I feel like the ball is in your court to say what things you’re willing to make forecasts about.
That said, I don’t know if making bets is at all a good use of time. I’m inclined to do it because I feel like your view really should be making different predictions (and I feel like you are participating in good faith and in fact would end up making different predictions). And I think it’s probably more promising than trying to hash out the arguments since at this point I feel like I mostly know your position and it’s incredibly slow going. But it seems very plausible that the right move is just to agree to disagree and not spend time on this. In that case it was particularly bad of me to try to claim the epistemic high ground. I can’t really defend myself there, but can explain by saying that I found your vitriolic reading of takeoff speeds pretty condescending and frustrating and, given that I think you are more wrong than right, wanted a nice way to demonstrate that.
I’ve mentioned the kinds of things I think your model will forecast badly, and suggested that we bet about them in particular:
I think you generally overestimate the rate of trend breaks on measurable trends. So let’s pick some trends and estimate probability of trend breaks.
I think you don’t understand in which domains trend-breaks are surprising and where they aren’t surprising, so you will be sometimes underconfident and sometimes overconfident on any given forecast. Same bet as last time.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I think you are underconfident about continuing scale-up in AI. So we can bet about future spending, size of particular labs, size of the ML field.
I think you overestimate DeepMind’s advantage over the rest of the field and so will make bad forecasts about where any given piece of progress comes from.
I think your AI timelines are generally too short. You can point to cool stuff happening as a vindication for your view, and there will certainly be some cool stuff happening, but I think if we actually get concrete you are just going to make worse predictions.
My uncharitable read on many of these domains is that you are saying “Sure, I think that Paul might have somewhat better forecasts than me on those questions, but why is that relevant to AGI?”
In that case it seems like the situation is pretty asymmetrical. I’m claiming that my view of AGI is related to beliefs and models that also bear on near-term questions, and I expect to make better forecasts than you in those domains because I have more accurate beliefs/models. If your view of AGI is unrelated to any near-term questions where we disagree, then that seems like an important asymmetry.
I suspect that indeed EY’s model has a limited ability to make near-term predictions, so that yes, the situation is asymmetrical. But I suspect his view is similar to my view, so I don’t think EY is wrong. But I am confused about why EY (i) hasn’t replied himself and (ii) in general, doesn’t communicate more clearly on this topic.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I wouldn’t be especially surprised by waifutechnology or machine translation jumping to newly accessible domains (the thing I care about and you shrug about (until the world ends)), but is that likely to exhibit a visible economic discontinuity in profits (which you care about and I shrug about (until the world ends))? There’s apparently already mass-scale deployment of waifutech in China to forlorn male teenagers, so maybe you’ll say the profits were already precedented. Google offers machine translation now, even though they don’t make much obvious measurable profit on that, but maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue. Or perhaps you want to say that “almost all AI profits” will come from robotics over the same period. Or maybe I misunderstand your viewpoint, and if you said something concrete about the stuff you care about, I would manage to disagree with that; or maybe you think that waifutech suddenly getting much more charming with the next generation of text transformers is something you already know enough to rule out; or maybe you think that 2024′s waifutech should definitely be able to do some observable surface-level thing it can’t do now.
I’d be happy to disagree about romantic chatbots or machine translation. I’d have to look into it more to get a detailed sense in either, but I can guess. I’m not sure what “wouldn’t be especially surprised” means, I think to actually get disagreements we need way more resolution than that so one question is whether you are willing to play ball (since presumably you’d also have to looking into to get a more detailed sense). Maybe we could save labor if people would point out the empirical facts we’re missing and we can revise in light of that, but we’d still need more resolution. (That said: what’s up for grabs here are predictions about the future, not present.)
I’d guess that machine translation is currently something like $100M/year in value, and will scale up more like 2x/year than 10x/year as DL improves (e.g. most of the total log increase will be in years with <3x increase rather than >3x increase, and 3 is like the 60th percentile of the number for which that inequality is tight).
I’d guess that increasing deployment of romantic chatbots will end up with technical change happening first followed by social change second, so the speed of deployment and change will depend on the speed of social change. At early stages of the social change you will likely see much large investment in fine-tuning for this use case, and the results will be impressive as you shift from random folks doing it to actual serious efforts. The fact that it’s driven by social rather than technical change means it could proceed at very different paces in different countries. I don’t expect anyone to make a lot of profit from this before self-driving cars, for example I’d be pretty surprised if this surpassed $1B/year of revenue before self-driving cars passed $10B/year of revenue. I have no idea what’s happening in China. It would be fairly surprising to me if there was currently an actually-compelling version of the technology—which we could try operationalize as something like how bad your best available romantic relationship with humans has to be, or how lonely you’d have to be, or how short-sighted you’d have to be, before it’s appealing. I don’t have strong views about a mediocre product with low activation energy that’s nevertheless used by many (e.g. in the same way we see lots of games with mediocre hedonic value and high uptake, or lots of passive gambling).
Thanks for continuing to try on this! Without having spent a lot of labor myself on looking into self-driving cars, I think my sheer impression would be that we’ll get $1B/yr waifutech before we get AI freedom-of-the-road; though I do note again that current self-driving tech would be more than sufficient for $10B/yr revenue if people built new cities around the AI tech level, so I worry a bit about some restricted use-case of self-driving tech that is basically possible with current tech finding some less regulated niche worth a trivial $10B/yr. I also remark that I wouldn’t be surprised to hear that waifutech is already past $1B/yr in China, but I haven’t looked into things there. I don’t expect the waifutech to transcend my own standards for mediocrity, but something has to be pretty good before I call it more than mediocre; do you think there’s particular things that waifutech won’t be able to do?
My model permits large jumps in ML translation adoption; it is much less clear about whether anyone will be able to build a market moat and charge big prices for it. Do you have a similar intuition about # of users increasing gradually, not just revenue increasing gradually?
I think we’re still at the level of just drawing images about the future, so that anybody who came back in 5 years could try to figure out who sounded right, at all, rather than assembling a decent portfolio of bets; but I also think that just having images versus no images is a lot of progress.
Yes, I think that value added by automated translation will follow a similar pattern. Number of words translated is more sensitive to how you count and random nonsense, as is number of “users” which has even more definitional issues.
You can state a prediction about self-driving cars in any way you want. The obvious thing is to talk about programs similar to the existing self-driving taxi pilots (e.g. Waymo One) and ask when they do $X of revenue per year, or when $X of self-driving trucking is done per year. (I don’t know what AI freedom-of-the-road means, do you mean something significantly more ambitious than self-driving trucks or taxis?)
Man, the problem is that you say the “jump to newly accessible domains” will be the thing that lets you take over the world. So what’s up for dispute is the prototype being enough to take over the world rather than years of progress by a giant lab on top of the prototype. It doesn’t help if you say “I expect new things to sometimes become possible” if you don’t further say something about the impact of the very early versions of the product.
Maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue
If e.g. people were spending $1B/year developing a technology, and then after a while it jumps from 0/year to $1B/year of profit, I’m not that surprised. (Note that machine translation is radically smaller than this, I don’t know the numbers.)
I do suspect they could have rolled out a crappy version earlier, perhaps by significantly changing their project. But why would they necessarily bother doing that? For me this isn’t violating any of the principles that make your stories sound so crazy. The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
(Note: it is surprising if an industry is spending $10T/year on R&D and then jumps from $1T --> $10T of revenue in one year in a world that isn’t yet growing crazily. The surprising depends a lot on the numbers involved, and in particular on how valuable it would have been to deploy a worse version earlier and how hard it is to raise money at different scales.)
The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
Would you say that this is a good description of Suddenly Hominids but you don’t expect that to happen again, or that this is a bad description of hominids?
It’s not a description of hominids at all, no one spent any money on R&D.
I think there are analogies where this would be analogous to hominids (which I think are silly, as we discuss in the next part of this transcript). And there are analogies where this is a bad description of hominids (which I prefer).
Spending money on R&D is essentially the expenditure of resources in order to explore and optimize over a promising design space, right? That seems like a good description of what natural selection did in the case of hominids. I imagine this still sounds silly to you, but I’m not sure why. My guess is that you think natural selection isn’t relevantly similar because it didn’t deliberately plan to allocate resources as part of a long bet that it would pay off big.
I think natural selection has lots of similarities to R&D, but (i) there are lots of ways of drawing the analogy, (ii) some important features of R&D are missing in evolution, including some really important ones for fast takeoff arguments (like the existence of actors who think ahead).
If someones wants to spell out why they think evolution of hominids means takeoff is fast then I’m usually happy to explain why I disagree with their particular analogy. I think this happens in the next discord log between me and Eliezer.
Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then
It seems like you are saying that there is some measure that was continuous all along, but that it’s not obvious in advance which measure was continuous. That seems to suggest that there are a bunch of plausible measures you could suggest in advance, and lots of interesting action will be from changes that are discontinuous changes on some of those measures. Is that right?
If so, don’t we get out a ton of predictions? Like, for every particular line someone thinks might be smooth, the gradualist has a higher probability on it being smooth than you would? So why can’t I just start naming some smooth lines (like any of the things I listed in the grandparent) and then we can play ball?
If not, what’s your position? Is it that you literally can’t think of the possible abstractions that would later make the graph smooth? (This sounds insane to me.)
I disagree that this is a meaningful forecasting track record. Massive degrees of freedom, and the mentioned events seem unresolvable, and it’s highly ambiguous how these things particularly prove the degree of error unless they were properly disambiguated in advance. Log score or it didn’t happen.
I want to register a gripe, re your follow-up post: when Eliezer says that he, Demis Hassabis, and Dario Amodei have a good “track record” because of their qualitative prediction successes, you object that the phrase “track record” should be reserved for things like Metaculus forecasts.
But when Ben Garfinkel says that Eliezer has a bad “track record” because he made various qualitative predictions Ben disagrees with, you slam the retweet button.
I already thought this narrowing of the term “track record” was weird. If you’re saying that we shouldn’t count Linus Pauling’s achievements in chemistry, or his bad arguments for Vitamin C megadosing, as part of Pauling’s “track record”, because they aren’t full probability distributions over concrete future events, then I worry a lot that this new word usage will cause confusion and lend itself to misuse. As long as it’s used even-handedly, though, it’s ultimately just a word.
(On my model, the main consequence of this is just that “track records” matter a lot less, because they become a much smaller slice of the evidence we have about a lot of people’s epistemics, expertise, etc.)
But if you’re going to complain about “track record” talk when the track record is alleged to be good but not when it’s alleged to be bad, then I have a genuine gripe with this terminology proposal. It already sounded a heck of a lot like an isolated demand for rigor to me, but if you’re going to redefine “track record” to refer to a narrow slice of the evidence, you at least need to do this consistently, and not crow some variant of ‘Aha! His track record is terrible after all!’ as soon as you find equally qualitative evidence that you like.
This was already a thing I worried would happen if we adopted this terminological convention, and it happened immediately.
I see what you’re saying, but it looks like you’re strawmanning me yet again with a more extreme version of my position. You’ve done that several times and you need to stop that.
What you’ve argued here prevents me from questioning the forecasting performance of every pundit who I can’t formally score, which is ~all of them.
Yes, it’s not a real forecasting track record unless it meets the sort of criteria that are fairly well understood in Tetlockian research. And neither is Ben Garfinkel’s post, that doesn’t give us a forecasting track record, like on Metaculus.
But if a non-track-recorded person suggests they’ve been doing a good job anticipating things, it’s quite reasonable to point out non-scorable things they said that seem incorrect, even with no way to score it.
In an earlier draft of my essay, I considered getting into bets he’s made (several of which he’s lost). I ended up not including those things. Partly because my focus was waning and it was more attainable to stick to the meta-level point. And partly because I thought the essay might be better if it was more focused. I don’t think there is literally zero information about his forecasting performance (that’s not plausible), but it seemed like it would be more of a distraction from my epistemic point. Bets are not as informative as Metaculus-style forecasts, but they are better than nothing. This stuff is a spectrum, even Metaculus doesn’t retain some kinds of information about the forecaster. Still, I didn’t get into it, though I could have.
But I ended up later editing in a link to one of Paul’s comments, where he describes some reasons that Robin looks pretty bad in hindsight, but also includes several things Eliezer said that seem quite off. None of those are scorable. But I added in a link to that, because Eliezer explicitly claimed he came across better in that debate, which overall he may have, but it’s actually more mixed than that, and that’s relevant to my meta-point that one can obfuscate these things without a proper track record. And Ben Garfinkel’s post is similarly relevant.
If the community felt more ambivalently about Eliezer’s forecasts, or even if Eliezer was more ambivalent about his own forecasts? And then there was some guy trying to convince people he has made bad forecasts? Then your objection of one-sidedness would make much more sense to me. That’s not what this is.
Eliezer actively tells people he’s anticipating things well, but he deliberately prevents his forecasts from being scorable. Pundits do that too, and you bet I would eagerly criticize vague non-scorable stuff they said that seems wrong. And yes, I would retweet someone criticizing those things too. Does that also bother you?
IMO that’s a much more defensible position, and is what the discussion should have initially focused on. From my perspective, the way the debate largely went is:
Jotto: Eliezer claims to have a relatively successful forecasting track record, along with Dario and Demis; but this is clearly dissembling, because a forecasting track record needs to look like a long series of Metaculus predictions.
Other people: (repeat without qualification the claim that Eliezer is falsely claiming to have a “forecasting track record”; simultaneously claims that Eliezer has a subpar “forecasting track record”, based on evidence that wouldn’t meet Jotto’s stated bar)
Jotto: (signal-boosts the inconsistent claims other people are making, without noting that this is equivocating between two senses of “track record” and therefore selectively applying two different standards)
Rob B: (gripes and complains)
Whereas the way the debate should have gone is:
Jotto: I personally disagree with Eliezer that the AI Foom debate is easy to understand and cash out into rough predictions about how the field has progressed since 2009, or how it is likely to progress in the future. Also, I wish that all of Eliezer, Robin, Demis, Dario, and Paul had made way more Metaculus-style forecasts back in 2010, so it would be easier to compare their prediction performance. I find it frustrating that nobody did this, and think we should start doing this way more now. Also, I think this sharper comparison would probably have shown that Eliezer is significantly worse at thinking about this topic than Paul, and maybe than Robin, Demis, and Dario.
Rob B: I disagree with your last sentence, and I disagree quantitatively that stuff like the Foom debate is as hard-to-interpret as you suggest. But I otherwise agree with you, and think it would have been useful if the circa-2010 discussions had included more explicit probability distributions, scenario breakdowns, quantitative estimates, etc. (suitably flagged as unstable, spitballed ass-numbers). Even where these aren’t cruxy and don’t provide clear evidence about people’s quality of reasoning about AGI, it’s still just helpful to have a more precise sense of what people’s actual beliefs at the time were. “X is unlikely” is way less useful than knowing whether it’s more like 30%, or more like 5%, or more like 0.1%, etc.
I think the whole ‘X isn’t a real track record’ thing was confusing, and made your argument sound more forceful than it should have.
Plus maybe some disagreements about how possible it is in general to form good models of people and of topics like AGI in the absence of Metaculus-ish forecasts, and disagreement about exactly how informative it would be to have a hundred examples of narrow-AI benchmark predictions over the last ten years from all the influential EAs?
(I think it would be useful, but more like ‘1% to 10% of the overall evidence for weighing people’s reasoning and correctness about AGI’, not ’90% to 100% of the evidence’.)
(An exception would be if, e.g., it turned out that ML progress is way more predictable than Eliezer or I believe. ML’s predictability is a genuine crux for us, so seeing someone else do amazing at this prediction task for a bunch of years, with foresight rather than hindsight, would genuinely update us a bunch. But we don’t expect to learn much from Eliezer or Rob trying to predict stuff, because while someone else may have secret insight that lets them predict the future of narrow-AI advances very narrowly, we are pretty sure we don’t know how to do that.)
Part of what I object to is that you’re a Metaculus radical, whose Twitter bio says “Replace opinions with forecasts.”
This is a view almost no one in the field currently agrees with or tries to live up to.
Which is fine, on its own. I like radicals, and want to hear their views argued for and hashed out in conversation.
But then you selectively accuse Eliezer of lying about having a “track record”, without noting how many other people are also expressing non-forecast “opinions” (and updating on these), and while using language in ways that make it sound like Eliezer is doing something more unusual than he is, and making it sound like your critique is more independent of your nonstandard views on track records and “opinions” than it actually is.
That’s the part that bugs me. If you have an extreme proposal for changing EA’s norms, argue for that proposal. Don’t just selectively take potshots at views or people you dislike more, while going easy on everyone else.
That’s the part that bugs me. If you have an extreme proposal for changing EA’s norms, argue for that proposal. Don’t just selectively take potshots at views or people you dislike more, while going easy on everyone else.
I think Jotto has argued for the proposal in the past. Whether he did it in that particular comment is not very important, so long as he holds everyone to the same standards.
As for his standards: I think he sees Eliezer as an easy target because he’s high status in this community and has explicitly said that he thinks his track record is good (in fact, better than other people). On its own, therefore, it’s not surprising that Eliezer would get singled out.
I no longer see exchanges with you as a good use of energy, unless you’re able to describe some of the strawmanning of me you’ve done and come clean about that.
EDIT: Since this is being downvoted, here is a comment chain where Rob Besinger interpreted me in ways that are bizarre, such as suggesting that I think Eliezer is saying he has “a crystal ball”, or that “if you record any prediction anywhere other than Metaculus (that doesn’t have similarly good tools for representing probability distributions), you’re a con artist”. Things that sound thematically similar to what I was saying, but were weird, persistent extremes that I don’t see as good-faith readings of me. It kept happening over Twitter, then again on LW. At no point have I felt he’s trying to understand what I actually think. So I don’t see the point of continuing with him.
This is a strawman. Ben Garfinkel never says that Yudkowsky has a bad track record. In fact the only time the phrase “bad track record” comes up in Garfinkel’s post is when you mention it in one of your comments.
The most Ben Garfinkel says about Yudkowsky’s track record is that it’s “at least pretty mixed”, which I think the content of the post supports, especially the clear-cut examples. He even emphasizes that he is deliberately cherry-picking bad examples from Eliezer’s track record in order to make a point, e.g. about Eliezer never having addressed his own bad predictions from the past.
It’s not enough to say “my world model was bad in such and such ways and I’ve changed it” to address your mistakes; you have to say “I made this specific prediction and it later turned out to be wrong”. Can you cite any instance of Eliezer ever doing that?
This is a strawman. Ben Garfinkel never says that Yudkowsky has a bad track record.
In the post, he says “his track record is at best fairly mixed” and “Yudkowsky may have a track record of overestimating or overstating the quality of his insights into AI”; and in the comments, he says “Yudkowsky’s track record suggests a substantial bias toward dramatic and overconfident predictions”.
What makes a track record “bad” is relative, but if Ben objects to my summarizing his view with the imprecise word “bad”, then I’ll avoid doing that. It doesn’t strike me as an important point for anything I said above.
The most Ben Garfinkel says about Yudkowsky’s track record is that it’s “at least pretty mixed”, which I think the content of the post supports, especially the clear-cut examples.
As long as we agree that “track record” includes the kind of stuff Jotto was saying it doesn’t include, I’m happy to say that Eliezer’s track record includes failures as well as successes. Indeed, I think that would make way more sense.
about Eliezer never having addressed his own bad predictions from the past.
Maybe worth mentioning in passing that this is of course false?
It’s not enough to say “my world model was bad in such and such ways and I’ve changed it” to address your mistakes; you have to say “I made this specific prediction and it later turned out to be wrong”. Can you cite any instance of Eliezer ever doing that?
Sure! “I wouldn’t have predicted AlphaGo and lost money betting against the speed of its capability gains”.
In the post, he says “his track record is at best fairly mixed” and “Yudkowsky may have a track record of overestimating or overstating the quality of his insights into AI”; and in the comments, he says “Yudkowsky’s track record suggests a substantial bias toward dramatic and overconfident predictions”.
Yes, I think all of that checks out. It’s hard to say, of course, because Eliezer rarely makes explicit predictions, but insofar as he does make them I think he clearly puts a lot of weight on his inside view into things.
That doesn’t make his track record “bad” but it’s something to keep in mind when he makes predictions.
Sure! “I wouldn’t have predicted AlphaGo and lost money betting against the speed of its capability gains”.
This counts as a mistake but I don’t think it’s important relative to the bad prediction about AI timelines Ben brings up in his post. If Eliezer explained why he had been wrong then it would make his position now more convincing, especially given his condescending attitude towards e.g. Metaculus forecasts.
I still think there’s something about the way Eliezer admits he was wrong that rubs me the wrong way but it’s hard to explain what that is right now. It’s not correct to say he doesn’t admit his mistakes per se, but there’s some other problem with how much he seems to “internalize” the fact that he was wrong.
I’ve retracted my original comment because of your example as it was not correct (despite having the right “vibe”, whatever that means).
I do wish to note that we spent a fair amount of time on Discord trying to nail down what earlier points we might disagree on, before the world started to end, and these Discord logs should be going up later.
From my perspective, the basic problem is that Eliezer’s story looks a lot like “business as usual until the world starts to end sharply”, and Paul’s story looks like “things continue smoothly until their smooth growth ends the world smoothly”, and both of us have ever heard of superforecasting and both of us are liable to predict near-term initial segments by extrapolating straight lines while those are available. Another basic problem, as I’d see it, is that we tend to tell stories about very different subject matters—I care a lot less than Paul about the quantitative monetary amount invested into Intel, to the point of not really trying to develop expertise about that.
I claim that I came off better than Robin Hanson in our FOOM debate compared to the way that history went. I’d claim that my early judgments of the probable importance of AGI, at all, stood up generally better than early non-Yudkowskian EA talking about that. Other people I’ve noticed ever making correct bold predictions in this field include Demis Hassabis, for predicting that deep learning would work at all, and then for predicting that he could take the field of Go and taking it; and Dario Amodei, for predicting that brute-forcing stacking more layers would be able to produce GPT-2 and GPT-3 instead of just asymptoting and petering out. I think Paul doesn’t need to bet against me to start producing a track record like this; I think he can already start to accumulate reputation by saying what he thinks is bold and predictable about the next 5 years; and if it overlaps “things that interest Eliezer” enough for me to disagree with some of it, better yet.
I agree that it’s plausible that we both make the same predictions about the near future. I think we probably don’t, and there are plenty of disagreements about all kinds of stuff. But if in fact we agree, then in 5 years you shouldn’t say “and see how much the world looked like I said?”
It feels to me like it goes: you say AGI will look crazy. Then I say that sounds unlike the world of today. Then you say “no, the world actually always looks discontinuous in the ways I’m predicting and your model is constantly surprised by real stuff that happens, e.g. see transformers or AlphaGo” and then I say “OK, let’s bet about literally anything at all, you pick.”
I think it’s pretty likely that we actually do disagree about how much the world of today is boring and continuous, where my error theory is that you spend too much time reading papers and press releases that paint a misleading picture and just aren’t that familiar with what’s happening on the ground. So I expect if we stake out any random quantity we’ll disagree somewhat.
Most things just aren’t bold and predictable, they are modest disagreements. I’m not saying I have some deep secret about the world, just that you are wrong in this case.
(Sorry for edits, accidentally posted early.)
I feel a bit confused about where you think we meta-disagree here, meta-policy-wise. If you have a thesis about the sort of things I’m liable to disagree with you about, because you think you’re more familiar with the facts on the ground, can’t you write up Paul’s View of the Next Five Years and then if I disagree with it better yet, but if not, you still get to be right and collect Bayes points for the Next Five Years?
I mean, it feels to me like this should be a case similar to where, for example, I think I know more about macroeconomics than your typical EA; so if I wanted to expend the time/stamina points, I could say a bunch of things I consider obvious and that contradict hot takes on Twitter and many EAs would go “whoa wait really” and then I could collect Bayes points later and have performed a public service, even if nobody showed up to disagree with me about that. (The reason I don’t actually do this… is that I tried; I keep trying to write a book about basic macro, only it’s the correct version explained correctly, and have a bunch of isolated chapters and unfinished drafts.) I’m also trying to write up my version of The Next Five Years assuming the world starts to end in 2025, since this is not excluded by my model; but writing in long-form requires stamina and I’ve been tired of late which is part of why I’ve been having Discord conversations instead.
I think you think there’s a particular thing I said which implies that the ball should be in my court to already know a topic where I make a different prediction from what you do, and so I should be able to state my own prediction about that topic and bet with you about that; or, alternatively, that I should retract some thing I said recently which implies that. And so, you shouldn’t need to have to do all the work to write up your forecasts generally, and it’s unfair that I’m trying to make you do all that work. Check? If so, I don’t yet see the derivation chain on this meta-level point.
I think the Hansonian viewpoint—which I consider another gradualist viewpoint, and whose effects were influential on early EA and which I think are still lingering around in EA—seemed surprised by AlphaGo and Alpha Zero, when you contrast its actual advance language with what actually happened. Inevitably, you can go back afterwards and claim it wasn’t really a surprise in terms of the abstractions that seem so clear and obvious now, but I think it was surprised then; and I also think that “there’s always a smooth abstraction in hindsight, so what, there’ll be one of those when the world ends too”, is a huge big deal in practice with respect to the future being unpredictable. From this, you seem to derive that I should already know what to bet with you about, and are annoyed by how I’m playing coy; because if I don’t bet with you right now, I should retract the statement that I think gradualists were surprised; but to me I’m not following the sequitur there.
Or maybe I’m just entirely misinterpreting the flow of your thoughts here.
I’ve said I’m happy to bet about anything, and listed some particular questions I’d bet about where I expect you to be wronger. If you had issued the same challenge to me, I would have picked one of the things and we would have already made some bets. So that’s why I feel like the ball is in your court to say what things you’re willing to make forecasts about.
That said, I don’t know if making bets is at all a good use of time. I’m inclined to do it because I feel like your view really should be making different predictions (and I feel like you are participating in good faith and in fact would end up making different predictions). And I think it’s probably more promising than trying to hash out the arguments since at this point I feel like I mostly know your position and it’s incredibly slow going. But it seems very plausible that the right move is just to agree to disagree and not spend time on this. In that case it was particularly bad of me to try to claim the epistemic high ground. I can’t really defend myself there, but can explain by saying that I found your vitriolic reading of takeoff speeds pretty condescending and frustrating and, given that I think you are more wrong than right, wanted a nice way to demonstrate that.
I’ve mentioned the kinds of things I think your model will forecast badly, and suggested that we bet about them in particular:
I think you generally overestimate the rate of trend breaks on measurable trends. So let’s pick some trends and estimate probability of trend breaks.
I think you don’t understand in which domains trend-breaks are surprising and where they aren’t surprising, so you will be sometimes underconfident and sometimes overconfident on any given forecast. Same bet as last time.
I think you are underconfident about the fact that almost all AI profits will come from areas that had almost-as-much profit in recent years. So we could bet about where AI profits are in the near term, or try to generalize this.
I think you are underconfident about continuing scale-up in AI. So we can bet about future spending, size of particular labs, size of the ML field.
I think you overestimate DeepMind’s advantage over the rest of the field and so will make bad forecasts about where any given piece of progress comes from.
I think your AI timelines are generally too short. You can point to cool stuff happening as a vindication for your view, and there will certainly be some cool stuff happening, but I think if we actually get concrete you are just going to make worse predictions.
My uncharitable read on many of these domains is that you are saying “Sure, I think that Paul might have somewhat better forecasts than me on those questions, but why is that relevant to AGI?”
In that case it seems like the situation is pretty asymmetrical. I’m claiming that my view of AGI is related to beliefs and models that also bear on near-term questions, and I expect to make better forecasts than you in those domains because I have more accurate beliefs/models. If your view of AGI is unrelated to any near-term questions where we disagree, then that seems like an important asymmetry.
I suspect that indeed EY’s model has a limited ability to make near-term predictions, so that yes, the situation is asymmetrical. But I suspect his view is similar to my view, so I don’t think EY is wrong. But I am confused about why EY (i) hasn’t replied himself and (ii) in general, doesn’t communicate more clearly on this topic.
I wouldn’t be especially surprised by waifutechnology or machine translation jumping to newly accessible domains (the thing I care about and you shrug about (until the world ends)), but is that likely to exhibit a visible economic discontinuity in profits (which you care about and I shrug about (until the world ends))? There’s apparently already mass-scale deployment of waifutech in China to forlorn male teenagers, so maybe you’ll say the profits were already precedented. Google offers machine translation now, even though they don’t make much obvious measurable profit on that, but maybe you’ll want to say that however much Google spends on that, they must rationally anticipate at least that much added revenue. Or perhaps you want to say that “almost all AI profits” will come from robotics over the same period. Or maybe I misunderstand your viewpoint, and if you said something concrete about the stuff you care about, I would manage to disagree with that; or maybe you think that waifutech suddenly getting much more charming with the next generation of text transformers is something you already know enough to rule out; or maybe you think that 2024′s waifutech should definitely be able to do some observable surface-level thing it can’t do now.
I’d be happy to disagree about romantic chatbots or machine translation. I’d have to look into it more to get a detailed sense in either, but I can guess. I’m not sure what “wouldn’t be especially surprised” means, I think to actually get disagreements we need way more resolution than that so one question is whether you are willing to play ball (since presumably you’d also have to looking into to get a more detailed sense). Maybe we could save labor if people would point out the empirical facts we’re missing and we can revise in light of that, but we’d still need more resolution. (That said: what’s up for grabs here are predictions about the future, not present.)
I’d guess that machine translation is currently something like $100M/year in value, and will scale up more like 2x/year than 10x/year as DL improves (e.g. most of the total log increase will be in years with <3x increase rather than >3x increase, and 3 is like the 60th percentile of the number for which that inequality is tight).
I’d guess that increasing deployment of romantic chatbots will end up with technical change happening first followed by social change second, so the speed of deployment and change will depend on the speed of social change. At early stages of the social change you will likely see much large investment in fine-tuning for this use case, and the results will be impressive as you shift from random folks doing it to actual serious efforts. The fact that it’s driven by social rather than technical change means it could proceed at very different paces in different countries. I don’t expect anyone to make a lot of profit from this before self-driving cars, for example I’d be pretty surprised if this surpassed $1B/year of revenue before self-driving cars passed $10B/year of revenue. I have no idea what’s happening in China. It would be fairly surprising to me if there was currently an actually-compelling version of the technology—which we could try operationalize as something like how bad your best available romantic relationship with humans has to be, or how lonely you’d have to be, or how short-sighted you’d have to be, before it’s appealing. I don’t have strong views about a mediocre product with low activation energy that’s nevertheless used by many (e.g. in the same way we see lots of games with mediocre hedonic value and high uptake, or lots of passive gambling).
Thanks for continuing to try on this! Without having spent a lot of labor myself on looking into self-driving cars, I think my sheer impression would be that we’ll get $1B/yr waifutech before we get AI freedom-of-the-road; though I do note again that current self-driving tech would be more than sufficient for $10B/yr revenue if people built new cities around the AI tech level, so I worry a bit about some restricted use-case of self-driving tech that is basically possible with current tech finding some less regulated niche worth a trivial $10B/yr. I also remark that I wouldn’t be surprised to hear that waifutech is already past $1B/yr in China, but I haven’t looked into things there. I don’t expect the waifutech to transcend my own standards for mediocrity, but something has to be pretty good before I call it more than mediocre; do you think there’s particular things that waifutech won’t be able to do?
My model permits large jumps in ML translation adoption; it is much less clear about whether anyone will be able to build a market moat and charge big prices for it. Do you have a similar intuition about # of users increasing gradually, not just revenue increasing gradually?
I think we’re still at the level of just drawing images about the future, so that anybody who came back in 5 years could try to figure out who sounded right, at all, rather than assembling a decent portfolio of bets; but I also think that just having images versus no images is a lot of progress.
Yes, I think that value added by automated translation will follow a similar pattern. Number of words translated is more sensitive to how you count and random nonsense, as is number of “users” which has even more definitional issues.
You can state a prediction about self-driving cars in any way you want. The obvious thing is to talk about programs similar to the existing self-driving taxi pilots (e.g. Waymo One) and ask when they do $X of revenue per year, or when $X of self-driving trucking is done per year. (I don’t know what AI freedom-of-the-road means, do you mean something significantly more ambitious than self-driving trucks or taxis?)
Man, the problem is that you say the “jump to newly accessible domains” will be the thing that lets you take over the world. So what’s up for dispute is the prototype being enough to take over the world rather than years of progress by a giant lab on top of the prototype. It doesn’t help if you say “I expect new things to sometimes become possible” if you don’t further say something about the impact of the very early versions of the product.
If e.g. people were spending $1B/year developing a technology, and then after a while it jumps from 0/year to $1B/year of profit, I’m not that surprised. (Note that machine translation is radically smaller than this, I don’t know the numbers.)
I do suspect they could have rolled out a crappy version earlier, perhaps by significantly changing their project. But why would they necessarily bother doing that? For me this isn’t violating any of the principles that make your stories sound so crazy. The crazy part is someone spending $1B and then generating $100B/year in revenue (much less $100M and then taking over the world).
(Note: it is surprising if an industry is spending $10T/year on R&D and then jumps from $1T --> $10T of revenue in one year in a world that isn’t yet growing crazily. The surprising depends a lot on the numbers involved, and in particular on how valuable it would have been to deploy a worse version earlier and how hard it is to raise money at different scales.)
Would you say that this is a good description of Suddenly Hominids but you don’t expect that to happen again, or that this is a bad description of hominids?
It’s not a description of hominids at all, no one spent any money on R&D.
I think there are analogies where this would be analogous to hominids (which I think are silly, as we discuss in the next part of this transcript). And there are analogies where this is a bad description of hominids (which I prefer).
Spending money on R&D is essentially the expenditure of resources in order to explore and optimize over a promising design space, right? That seems like a good description of what natural selection did in the case of hominids. I imagine this still sounds silly to you, but I’m not sure why. My guess is that you think natural selection isn’t relevantly similar because it didn’t deliberately plan to allocate resources as part of a long bet that it would pay off big.
I think natural selection has lots of similarities to R&D, but (i) there are lots of ways of drawing the analogy, (ii) some important features of R&D are missing in evolution, including some really important ones for fast takeoff arguments (like the existence of actors who think ahead).
If someones wants to spell out why they think evolution of hominids means takeoff is fast then I’m usually happy to explain why I disagree with their particular analogy. I think this happens in the next discord log between me and Eliezer.
It seems like you are saying that there is some measure that was continuous all along, but that it’s not obvious in advance which measure was continuous. That seems to suggest that there are a bunch of plausible measures you could suggest in advance, and lots of interesting action will be from changes that are discontinuous changes on some of those measures. Is that right?
If so, don’t we get out a ton of predictions? Like, for every particular line someone thinks might be smooth, the gradualist has a higher probability on it being smooth than you would? So why can’t I just start naming some smooth lines (like any of the things I listed in the grandparent) and then we can play ball?
If not, what’s your position? Is it that you literally can’t think of the possible abstractions that would later make the graph smooth? (This sounds insane to me.)
I disagree that this is a meaningful forecasting track record. Massive degrees of freedom, and the mentioned events seem unresolvable, and it’s highly ambiguous how these things particularly prove the degree of error unless they were properly disambiguated in advance. Log score or it didn’t happen.
(Slightly edited to try and sound less snarky)
I want to register a gripe, re your follow-up post: when Eliezer says that he, Demis Hassabis, and Dario Amodei have a good “track record” because of their qualitative prediction successes, you object that the phrase “track record” should be reserved for things like Metaculus forecasts.
But when Ben Garfinkel says that Eliezer has a bad “track record” because he made various qualitative predictions Ben disagrees with, you slam the retweet button.
I already thought this narrowing of the term “track record” was weird. If you’re saying that we shouldn’t count Linus Pauling’s achievements in chemistry, or his bad arguments for Vitamin C megadosing, as part of Pauling’s “track record”, because they aren’t full probability distributions over concrete future events, then I worry a lot that this new word usage will cause confusion and lend itself to misuse. As long as it’s used even-handedly, though, it’s ultimately just a word.
(On my model, the main consequence of this is just that “track records” matter a lot less, because they become a much smaller slice of the evidence we have about a lot of people’s epistemics, expertise, etc.)
But if you’re going to complain about “track record” talk when the track record is alleged to be good but not when it’s alleged to be bad, then I have a genuine gripe with this terminology proposal. It already sounded a heck of a lot like an isolated demand for rigor to me, but if you’re going to redefine “track record” to refer to a narrow slice of the evidence, you at least need to do this consistently, and not crow some variant of ‘Aha! His track record is terrible after all!’ as soon as you find equally qualitative evidence that you like.
This was already a thing I worried would happen if we adopted this terminological convention, and it happened immediately.
</end of gripe>
I see what you’re saying, but it looks like you’re strawmanning me yet again with a more extreme version of my position. You’ve done that several times and you need to stop that.
What you’ve argued here prevents me from questioning the forecasting performance of every pundit who I can’t formally score, which is ~all of them.
Yes, it’s not a real forecasting track record unless it meets the sort of criteria that are fairly well understood in Tetlockian research. And neither is Ben Garfinkel’s post, that doesn’t give us a forecasting track record, like on Metaculus.
But if a non-track-recorded person suggests they’ve been doing a good job anticipating things, it’s quite reasonable to point out non-scorable things they said that seem incorrect, even with no way to score it.
In an earlier draft of my essay, I considered getting into bets he’s made (several of which he’s lost). I ended up not including those things. Partly because my focus was waning and it was more attainable to stick to the meta-level point. And partly because I thought the essay might be better if it was more focused. I don’t think there is literally zero information about his forecasting performance (that’s not plausible), but it seemed like it would be more of a distraction from my epistemic point. Bets are not as informative as Metaculus-style forecasts, but they are better than nothing. This stuff is a spectrum, even Metaculus doesn’t retain some kinds of information about the forecaster. Still, I didn’t get into it, though I could have.
But I ended up later editing in a link to one of Paul’s comments, where he describes some reasons that Robin looks pretty bad in hindsight, but also includes several things Eliezer said that seem quite off. None of those are scorable. But I added in a link to that, because Eliezer explicitly claimed he came across better in that debate, which overall he may have, but it’s actually more mixed than that, and that’s relevant to my meta-point that one can obfuscate these things without a proper track record. And Ben Garfinkel’s post is similarly relevant.
If the community felt more ambivalently about Eliezer’s forecasts, or even if Eliezer was more ambivalent about his own forecasts? And then there was some guy trying to convince people he has made bad forecasts? Then your objection of one-sidedness would make much more sense to me. That’s not what this is.
Eliezer actively tells people he’s anticipating things well, but he deliberately prevents his forecasts from being scorable. Pundits do that too, and you bet I would eagerly criticize vague non-scorable stuff they said that seems wrong. And yes, I would retweet someone criticizing those things too. Does that also bother you?
IMO that’s a much more defensible position, and is what the discussion should have initially focused on. From my perspective, the way the debate largely went is:
Jotto: Eliezer claims to have a relatively successful forecasting track record, along with Dario and Demis; but this is clearly dissembling, because a forecasting track record needs to look like a long series of Metaculus predictions.
Other people: (repeat without qualification the claim that Eliezer is falsely claiming to have a “forecasting track record”; simultaneously claims that Eliezer has a subpar “forecasting track record”, based on evidence that wouldn’t meet Jotto’s stated bar)
Jotto: (signal-boosts the inconsistent claims other people are making, without noting that this is equivocating between two senses of “track record” and therefore selectively applying two different standards)
Rob B: (gripes and complains)
Whereas the way the debate should have gone is:
Jotto: I personally disagree with Eliezer that the AI Foom debate is easy to understand and cash out into rough predictions about how the field has progressed since 2009, or how it is likely to progress in the future. Also, I wish that all of Eliezer, Robin, Demis, Dario, and Paul had made way more Metaculus-style forecasts back in 2010, so it would be easier to compare their prediction performance. I find it frustrating that nobody did this, and think we should start doing this way more now. Also, I think this sharper comparison would probably have shown that Eliezer is significantly worse at thinking about this topic than Paul, and maybe than Robin, Demis, and Dario.
Rob B: I disagree with your last sentence, and I disagree quantitatively that stuff like the Foom debate is as hard-to-interpret as you suggest. But I otherwise agree with you, and think it would have been useful if the circa-2010 discussions had included more explicit probability distributions, scenario breakdowns, quantitative estimates, etc. (suitably flagged as unstable, spitballed ass-numbers). Even where these aren’t cruxy and don’t provide clear evidence about people’s quality of reasoning about AGI, it’s still just helpful to have a more precise sense of what people’s actual beliefs at the time were. “X is unlikely” is way less useful than knowing whether it’s more like 30%, or more like 5%, or more like 0.1%, etc.
I think the whole ‘X isn’t a real track record’ thing was confusing, and made your argument sound more forceful than it should have.
Plus maybe some disagreements about how possible it is in general to form good models of people and of topics like AGI in the absence of Metaculus-ish forecasts, and disagreement about exactly how informative it would be to have a hundred examples of narrow-AI benchmark predictions over the last ten years from all the influential EAs?
(I think it would be useful, but more like ‘1% to 10% of the overall evidence for weighing people’s reasoning and correctness about AGI’, not ’90% to 100% of the evidence’.)
(An exception would be if, e.g., it turned out that ML progress is way more predictable than Eliezer or I believe. ML’s predictability is a genuine crux for us, so seeing someone else do amazing at this prediction task for a bunch of years, with foresight rather than hindsight, would genuinely update us a bunch. But we don’t expect to learn much from Eliezer or Rob trying to predict stuff, because while someone else may have secret insight that lets them predict the future of narrow-AI advances very narrowly, we are pretty sure we don’t know how to do that.)
Part of what I object to is that you’re a Metaculus radical, whose Twitter bio says “Replace opinions with forecasts.”
This is a view almost no one in the field currently agrees with or tries to live up to.
Which is fine, on its own. I like radicals, and want to hear their views argued for and hashed out in conversation.
But then you selectively accuse Eliezer of lying about having a “track record”, without noting how many other people are also expressing non-forecast “opinions” (and updating on these), and while using language in ways that make it sound like Eliezer is doing something more unusual than he is, and making it sound like your critique is more independent of your nonstandard views on track records and “opinions” than it actually is.
That’s the part that bugs me. If you have an extreme proposal for changing EA’s norms, argue for that proposal. Don’t just selectively take potshots at views or people you dislike more, while going easy on everyone else.
I think Jotto has argued for the proposal in the past. Whether he did it in that particular comment is not very important, so long as he holds everyone to the same standards.
As for his standards: I think he sees Eliezer as an easy target because he’s high status in this community and has explicitly said that he thinks his track record is good (in fact, better than other people). On its own, therefore, it’s not surprising that Eliezer would get singled out.
I no longer see exchanges with you as a good use of energy, unless you’re able to describe some of the strawmanning of me you’ve done and come clean about that.
EDIT: Since this is being downvoted, here is a comment chain where Rob Besinger interpreted me in ways that are bizarre, such as suggesting that I think Eliezer is saying he has “a crystal ball”, or that “if you record any prediction anywhere other than Metaculus (that doesn’t have similarly good tools for representing probability distributions), you’re a con artist”. Things that sound thematically similar to what I was saying, but were weird, persistent extremes that I don’t see as good-faith readings of me. It kept happening over Twitter, then again on LW. At no point have I felt he’s trying to understand what I actually think. So I don’t see the point of continuing with him.
This is a strawman. Ben Garfinkel never says that Yudkowsky has a bad track record. In fact the only time the phrase “bad track record” comes up in Garfinkel’s post is when you mention it in one of your comments.
The most Ben Garfinkel says about Yudkowsky’s track record is that it’s “at least pretty mixed”, which I think the content of the post supports, especially the clear-cut examples. He even emphasizes that he is deliberately cherry-picking bad examples from Eliezer’s track record in order to make a point, e.g. about Eliezer never having addressed his own bad predictions from the past.
It’s not enough to say “my world model was bad in such and such ways and I’ve changed it” to address your mistakes; you have to say “I made this specific prediction and it later turned out to be wrong”. Can you cite any instance of Eliezer ever doing that?
In the post, he says “his track record is at best fairly mixed” and “Yudkowsky may have a track record of overestimating or overstating the quality of his insights into AI”; and in the comments, he says “Yudkowsky’s track record suggests a substantial bias toward dramatic and overconfident predictions”.
What makes a track record “bad” is relative, but if Ben objects to my summarizing his view with the imprecise word “bad”, then I’ll avoid doing that. It doesn’t strike me as an important point for anything I said above.
As long as we agree that “track record” includes the kind of stuff Jotto was saying it doesn’t include, I’m happy to say that Eliezer’s track record includes failures as well as successes. Indeed, I think that would make way more sense.
Maybe worth mentioning in passing that this is of course false?
Sure! “I wouldn’t have predicted AlphaGo and lost money betting against the speed of its capability gains”.
Extremely important failures and extremely important successes, no less.
Yes, I think all of that checks out. It’s hard to say, of course, because Eliezer rarely makes explicit predictions, but insofar as he does make them I think he clearly puts a lot of weight on his inside view into things.
That doesn’t make his track record “bad” but it’s something to keep in mind when he makes predictions.
This counts as a mistake but I don’t think it’s important relative to the bad prediction about AI timelines Ben brings up in his post. If Eliezer explained why he had been wrong then it would make his position now more convincing, especially given his condescending attitude towards e.g. Metaculus forecasts.
I still think there’s something about the way Eliezer admits he was wrong that rubs me the wrong way but it’s hard to explain what that is right now. It’s not correct to say he doesn’t admit his mistakes per se, but there’s some other problem with how much he seems to “internalize” the fact that he was wrong.
I’ve retracted my original comment because of your example as it was not correct (despite having the right “vibe”, whatever that means).