Knowingly make an epistemic error? Say “X is false but I believe it is true”?
That’s what the arguments you’ve given for this have mostly amounted to. You have said “I need to believe this to be motivated and do productive work” in response to questions about the probabilities in the past, while not giving solid reasons for the confidence.
Okay, so suppose we jump back 4 years and I’m saying that maybe I ought to write a Harry Potter fanfiction. And it’s going to be the most popular HP fanfiction on the whole Internet.
When did you predict that? Early on I did not hear you making such claims, with the tune changing after it became clear that demand for it was good.
4 years ago I did advocate getting Math Olympiad people, and said they could be gotten, and had empirical evidence of that from multiple angles. And I did recognize your writing and fiction were well-received, and had evidence from the reactions to “Staring into the Singularity” and OB/LW. You tried several methods, including the rationality book, distributing written rationality exercises/micromanaging CFAR content, and the fanfiction. Most of them wasted time and resources without producing results, and one succeeded.
And there is a larger context, that in addition to the successes you are highlighting the path includes: Flare, LOGI and associated research, pre-Vassar SI management issues, open-source singularity, commercial software, trying to create non-FAI before nanowars in 2015.
This made me update on Michael Vassar’s ability to discriminate Tomer Kagans in advance from within a rather large reference class of people trying to be Tomer Kagan.
Tomer is indeed pretty great, but I have heard Michael say things like that about a number of people and projects over the years. Most did not become like Quixey. And what’s the analogy here? That people with good ability to predict success in scientific research have indicated you will succeed taking into account how the world and the space of computer science and AI progress must be for that? That Michael has?
As a LessWrong reader, I notice that I am confused because this does not sound like something you would say, but I’m not sure I could explain the difference between this and “heroic epistemology.”
EDIT: for the benefit of other readers following the conversation, Eliezer gives a description of heroic epistemology here.
For the record, I don’t recall ever hearing you say something like this in my presence or online, and if somebody had told me in person that you had said this, I think I would’ve raised a skeptical eyebrow and said “Really? That doesn’t sound like something Eliezer would say. Have you read The Sequences?”
But also, I remain confused about the normative content of “heroic epistemology.”
Ask Anna about it, she was present on both occasions, at the tail end of the Singularity Summit workshop discussion in New York City, and at the roundtable meeting at the office with Anna, Paul, Luke, and Louie.
In a related vein, arguments like this are arguments that someone could do A, but not so much that you will do A (and B and C and...). My impression is of too many arguments like the former and enough of the latter. If you can remedy that, it would be great, but it is a fact about the responses I have seen.
Eliezer emailed me to ask me about it (per Carl’s request, above); I emailed him back with the email below, which Eliezer requested I paste into the LW thread. Pasting:
In the majority of cases, people do not straightforwardly say “X is false, but I need to believe X anyhow”. More often they wiggle, and polite conversation drops the subject.
You have made statements that I and at least some others interpreted as perhaps indicating such wiggles (i.e., as perhaps indicating a desire to hold onto false impressions by diverting conscious attention or discussion from a particular subject). You have never, to my knowledge, uttered a straight-forward endorsement of holding false beliefs. The wiggle-suggesting statements were not super-clear, and were not beyond the realm of possible misinterpretation.
Re: statements that seemed to me and some others to indicate possible wiggles: You have mentioned multiple times that, well, I forget, but something like, it’d be hard to do top focused research on FAI-like problems while estimating a 1% chance of success. You’ve also avoided giving probability estimates in a lot of contexts, and have sometimes discouraged conversations in which others did so. You seemed perhaps a bit upset or defensive at multiple points during the probability estimates conversation with Zvi in NYC (enough so that a couple people commented on it with surprise to me afterward (not Carl or Paul; I forget who; newcomers to our conversations)), but, to your credit, you commented on these difficulties and proceeded to use debiasing techniques (e.g., I think you might’ve mentioned leaving a line of retreat, and might’ve given yourself a minute to do so).
If you would like polite conversation not to drop the subject on future occasions on which your impressions look (to me and other non-telepathic observers) like they might possibly be wiggly, give me a straight-forward request and I can change my settings here. I have in fact been a bit afraid of disturbing your motivation, and also a bit afraid of reducing your good will toward me.
Michael Vassar might be another interesting one to probe, if you’re collecting opinions. Though note that Vassar, Carl, and I have all discussed this at least a bit, and so are not independent datapoints.
From my internal perspective, the truth-as-I-experience-it is that I’m annoyed when people raise the topic because it’s all wasted motion, the question sets up a trap that forces you into appearing arrogant, and I honestly think that “Screw all this, I’m just going to go ahead and do it and you can debate afterward what the probabilities were” is a perfectly reasonable response.
From the perspective of folks choosing between supporting multiple lines of AI risk reduction effort, of which MIRI is only one, such probability estimates are not wasted effort.
Though your point about appearing arrogant is well taken. It’s unfortunate that it isn’t socially okay to publicly estimate a high probability of success, or to publicly claim one’s own exceptionalism, when ones impressions point that way. It places a barrier toward honest conversation here.
From my internal perspective, the truth-as-I-experience-it is that I’m annoyed when people raise the topic [of MIRI’s success-odds] because [good reason].
I suspect this annoyance is easily misinterpreted, independent of its actual cause. Most humans respond with annoyance when their plans are criticized. Also, in situations where A has power over B, and where B then shares concerns or criticisms about A’s plans, and where A responds with annoyance or with avoidance of such conversation… B is apt to respond (as I did) by being a bit hesitant to bring the topic up, and by also wondering if A is being defensive.
I’m not saying I was correct here. I’m also not sure what the fix is. But it might be worth setting a 1-minute timer and brainstorming or something.
If you were anyone else, this is ordinarily the point where I tell you that I’m just going to ignore all this and go ahead do it, and then afterward you can explain why it was either predictable in retrospect or a fluke, according to your taste. Since it’s you: What’s the first next goal you think I can’t achieve, strongly enough that if I do it, you give up on non-heroic epistemology?
If you were anyone else, this is ordinarily the point where I tell you that I’m just going to ignore all this and go ahead do it
I’m familiar with this move. But you make it before failing too, so its evidentiary weight is limited, and insufficient for undertakings with low enough prior probability from all the other evidence besides the move.
What’s the first next goal you think I can’t achieve, strongly enough that if I do it, you give up on non-heroic epistemology?
I don’t buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality.
Two developments in AI as big as Pearl’s causal networks (as judged by Norvig types) by a small MIRI team would be a limited subset of the problems to be solved by a project trying to build AGI with a different and very safe architecture before the rest of the world, and wouldn’t address the question of the probability that such is needed in the counterfactual, but it would cause me to stop complaining and would powerfully support the model that MIRI can be more productive than the rest of the AI field when currently-available objective indicators put it as a small portion of the quality-adjusted capacity.
If we want a predictor for success that’s a lot better than the vast majority of quite successful entrepreneurs and pathbreaking researchers, making numerous major basic science discoveries and putting them together in a way that saves the world, then we need some evidence to distinguish the team and explain why it will make greater scientific contributions than any other ever with high reliability in a limited time.
A lot of intermediate outcomes would multiply my credence in and thus valuation of the “dozen people race ahead of the rest of the world in AI” scenario, but just being as productive as von Neumann or Turing or Pearl or Einstein would not result in high probability of FAI success, so the evidence has to be substantial.
I’m familiar with this move. But you make it before failing too
Sure, you try, sometimes you lose, sometimes you win. On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn’t be any impossible successes by anyone you know personally after you met them. They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.
I don’t buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality.
We disagree about how to use statistics in order to get really actually correct answers. Having such a low estimate of my rationality that you think that I know what correct statistics are, and am refusing to use them, is not good news from an Aumann perspective and fails the ideological Turing Test. In any case, surely if my predictions are correct you should update your belief about good frameworks (see the reasoning used in the Pascal’s Muggle post) - to do otherwise and go on insisting that your framework was nonetheless correct would be oblivious.
Two developments in AI as big as Pearl’s causal networks (as judged by Norvig types)
...should not have been disclosed to the general world, since proof well short of this should suffice for sufficient funding (Bayes nets were huge), though they might be disclosed to some particular Norvig type on a trusted oversight committee if there were some kind of reason for the risk. Major breakthroughs on the F side of FAI are not likely to be regarded as being as exciting as AGI-useful work like Bayes nets, though they may be equally mathematically impressive or mathematically difficult. Is there some kind of validation which you think MIRI should not be able to achieve on non-heroic premises, such that the results should be disclosed to the general world?
EDIT: Reading through the rest of the comment more carefully, I’m not sure we estimate the same order of magnitude of work for what it takes to build FAI under mildly good background settings of hidden variables. The reason why I don’t think the mainstream can build FAI isn’t that FAI is intrinsically huge a la the Cyc hypothesis. The mainstream is pretty good at building huge straightforward things. I just expect them to run afoul of one of the many instakill gotchas because they’re one or two orders of magnitude underneath the finite level of caring required.
EDIT 2: Also, is there a level short of 2 gigantic breakthroughs which causes you to question non-heroic epistemology? The condition is sufficient, but is it necessary? Do you start to doubt the framework after one giant breakthrough (leaving aside the translation question for now)? If not, what probability would you assign to that, on your framework? Standard Bayesian Judo applies—if you would, as I see it, play the role of the skeptic, then you must either be overly-credulous-for-the-role that we can do heroic things like one giant breakthrough, or else give up your skepticism at an earlier signal than the second. For you cannot say that something is strongly prohibited on your model and yet also refuse to update much if it happens, and this applies to every event which might lie along the way. (Evenhanded application: ’Tis why I updated on Quixey instead of saying “Ah, but blah”; Quixey getting this far just wasn’t supposed to happen on my previous background theory, and shouldn’t have happened even if Vassar had praised ten people to me instead of two.)
On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn’t be any impossible successes by anyone you know personally after you met them.
I don’t understand why you say this. Given Carl’s IQ and social circle (didn’t he used to work for a hedge fund run by Peter Thiel?) why would it be very surprising that someone he personally knows achieves your current level of success after he meets them?
They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.
Carl referenced “Staring Into the Singularity” as an early indicator of your extraordinary verbal abilities (which explains much if not all of your subsequent successes). It suggests that’s how you initially attracted his attention. The same is certainly true for me. I distinctly recall saying to myself “I should definitely keep track of this guy” when I read that, back in the extropian days. Is that enough for you to count as “people who you met because of their previous success”?
In any case, almost everyone who meets you now would count you as such. What arguments can you give to them that “heroic epistemology” is normative (and hence they are justified in donating to MIRI)?
To state my overall position on the topic being discussed, I think according to “non-heroic epistemology”, after someone achieves an “impossible success”, you update towards them being able to achieve further successes of roughly the same difficulty and in related fields that use similar skills, but the posterior probabilities of them solving much more difficult problems or in fields that use very different skills remain low (higher relative to the prior, but still low in an absolute sense). Given my understanding of the distribution of cognitive abilities in humans, I don’t see why I would ever “give up” this epistemology, unless you achieved a level of success that made me suspect that you’re an alien avatar or something.
In any case, almost everyone who meets you now would count you as such. What arguments can you give to them that “heroic epistemology” is normative (and hence they are justified in donating to MIRI)?
Yes, no matter how many impossible things you do, the next person you meet thinks that they only heard of you because of them, ergo selection bias. This is an interesting question purely on a philosophical level—it seems to me to have some of the flavor of quantum suicide experiments where you can’t communicate your evidence. In principle this shouldn’t happen without quantum suicide for logically omniscient entities who already know the exact fraction of people with various characteristics, i.e., agree on exact priors, but I think it might start happening again to people who are logically unsure about which framework they should use.
To avoid talking past one another: I agree that one can and should update on evidence beyond the most solid empirical reference classes in predicting success. If you mean to say that a majority of the variation on a log scale in success (e.g. in wealth or scientific productivity) can be accounted for with properties of individuals and their circumstances, beyond dumb luck then we can agree on that. Some of those characteristics are more easily observed, while others are harder to discern or almost unmeasurable from a distance so that track records may be our best way to discern them.
shouldn’t be any impossible successes by anyone you know personally after you met them.
That is to say, repeated successes should not be explained by luck, but by updating estimates of hard-to-observe characteristics and world model.
The distribution of successes and failures you have demonstrated is not “impossible” or driving a massive likelihood ratio given knowledge about your cognitive and verbal ability, behavioral evidence of initiative and personality, developed writing skill (discernible through inspection and data about its reception), and philosophical inclinations. Using measurable features, and some earlier behavioral or track record data one can generate reference classes with quite high levels of lifetime success, e.g. by slicing and dicing cohorts like this one. Updating on further successes delivers further improvements.
But updating on hidden characteristics does not suffer exponential penalties like chance explanations, and there is a lot of distance to cover in hidden characteristics before a 10% probability of MIRI-derived FAI (or some other causal channel) averting existential catastrophe that would have occurred absent MIRI looks reasonable.
Now large repeated updates about hidden characteristics still indicate serious model problems and should lead us to be very skeptical of those models. However, I don’t see such very large surprising updates thus far.
Sure, you try, sometimes you lose, sometimes you win.
If difficulty is homogenous (at least as far as one can discern in advance), then we can use these data straightforwardly, but a lot of probability will be peeled off relative to “Tomer must win.” And generalizing to much higher difficulty is still dubious for the reasons discussed above.
Having such a low estimate of my rationality that you think that I know what correct statistics are, and am refusing to use them, is not good news from an Aumann perspective and fails the ideological Turing Test.
This is not what I meant. I didn’t claim you would explicitly endorse a contradiction formally. But nonetheless, the impression I got was of questions about probability met with troubling responses like talking about the state of mind you need for work and wanting to not think in terms of probabilities of success for your own work. That seems a bad signal because of the absence of good responses, and the suggestion that the estimates may not be the result of very much thought, or may be unduly affected by their emotional valence, without ever saying “p and not p.”
Is there some kind of validation which you think MIRI should not be able to achieve on non-heroic premises, such that the results should be disclosed to the general world?...should not have been disclosed to the general world, since proof well short of this should suffice for sufficient funding (Bayes nets were huge)
As I said elsewhere a 10% probability of counterfactually saving the world is far above the threshold for action. One won’t get to high confidence in that low prior claim without extraordinary evidence, but the value of pursuing it increases continuously with intermediate levels of evidence. Some examples would be successful AI researchers coming to workshops and regularly saying that the quality and productivity of the research group and process was orders of magnitude more productive, the results very solid, etc. This is one of the reasons I like the workshop path, because it exposes the thesis to empirical feedback.
The reason why I don’t think the mainstream can build FAI isn’t that FAI is intrinsically huge a la the Cyc hypothesis. The mainstream is pretty good at building huge straightforward things.
Although as we have discussed with AI folk, there are also smart AI people who would like to find nice clean powerful algorithms with huge practical utility without significant additional work.
I just expect them to run afoul of one of the many instakill gotchas because they’re one or two orders of magnitude underneath the finite level of caring required.
Yes, we do still have disagreements about many of the factual questions that feed into a probability estimate, and if I adopted your view on all of those except MIRI productivity there would be much less of a gap. There are many distinct issues going into the estimation of a probability of your success, from AGI difficulty, to FAI difficulty, to the competence of regular AI people and governance institutions, the productivity of a small MIRI team, the productivity of the rest of the world, signs of AI being close, reactions to those signs, and others.
There are a number of connections between these variables, but even accounting for that your opinions are systematically firmly in the direction of greater personal impact relative to the analyses of others, and the clustering seems tighter than is typical (others seem to vary more, sometimes evaluating different subissues as pointing in different directions). This shows up in attempts to work through the issues for estimation, as at that meeting with Paul et al.
One can apply a bias theory to myself and Paul Christiano and Nick Bostrom and the FHI surveys of AI experts are biased towards normalcy, respectability and conservatism. But I would still question the coincidence of so many substantially-independent variables landing in the same direction, and uncertainty over the pieces hurts the hypothesis that MIRI has, say, a 10% probability of averting a counterfactual existential catastrophe disproportionately.
And it is possible that you have become a superb predictor of such variables in the last 10 years (setting aside earlier poor predictions), and I could and would update on good technological and geopolitical prediction in DAGGRE or the like.
Thanks for talking this out, and let me reiterate that in my expectation your and MIRI’s existence (relative to the counterfactual in which it never existed and you become a science fiction writer) has been a good thing and reduced my expectation for existential risk.
The distribution of successes and failures you have demonstrated is not “impossible” or driving a massive likelihood ratio given knowledge about your cognitive and verbal ability, behavioral evidence of initiative and personality, developed writing skill (discernible through inspection and data about its reception), and philosophical inclinations
Of course I expect you to say that, since to say otherwise given your previous statements is equivalent to being openly incoherent and I do not regard you so lowly. But I don’t yet believe that you would actually have accepted or predicted those successes ante facto, vs. claiming ante facto that those successes were unlikely and that trying was overconfident. Which is why I repeat my question: What is the least impossible thing I could do next, where anything up to that is permitted by your model so it’s equivalent to affirming that you think I might be able to do it, and anything beyond that was prohibited by your model so it’s time to notice your confusion? I mean, if you think I can make one major AI breakthrough but not two, that’s already a lot of confidence in me… is that really what your outside view would say about me?
But nonetheless, you have returned questions about probability with troubling responses like talking about the state of mind you need for work and wanting to not think in terms of probabilities of success for your own work.
Please distinguish between the disputed reality and your personal memory, unless you’re defining the above so broadly (and uncharitably!) that my ‘wasted motion’ FB post counts as an instance.
Although as we have discussed with AI folk, there are also smart AI people who would like to find nice clean powerful algorithms with huge practical utility without significant additional work.
Without significant work? I don’t think I can do that. Why would you think I thought I could do that?
And it is possible that you have become a superb predictor of such variables in the last 10 years (setting aside earlier poor predictions), and I could and would update on good technological and geopolitical prediction in DAGGRE or the like.
If enough people agreed on that and DAGGRE could be done with relatively low effort on my part, I would do so, though I think I’d want at least some people committing in writing to large donations given success because it would be a large time commitment and I’m prior-skeptical that people know or are honest about their own reasons for disagreement; and I would expect the next batch of pessimists to write off the DAGGRE results (i.e., claim it already compatible with my known properties) so there’d be no long-term benefit. Still, 8 out of 8 on 80K’s “Look how bad your common sense is!” test, plus I recall getting 9 out of 10 questions correct the last time I was asked for 90% probabilities on a CFAR calibration test, so it’s possible I’ve already outrun the reference class of people who are bad at this.
Though if it’s mostly geopolitical questions where the correct output is “I know I don’t know much about this” modulo some surface scans of which other experts are talking sense, I wouldn’t necessarily expect to outperform the better groups that have already read up on cognitive rationality and done a few calibration exercises.
Which is why I repeat my question: What is the least impossible thing I could do next, where anything up to that is permitted by your model so it’s equivalent to affirming that you think I might be able to do it, and anything beyond that was prohibited by your model so it’s time to notice your confusion?
So, if von Neumann came out with similar FAI claims, but couldn’t present compelling arguments to his peers (if not to exact agreement, perhaps within an order of magnitude) I wouldn’t believe him. So showing that, e.g. your math problem-solving ability is greater than my point estimate, wouldn’t be very relevant. Shocking achievements would lead me to upgrade my estimate of your potential contribution going forward (although most of the work in an FAI team would be done by others in any case), resolving uncertainty about ability, but that would not be enough as such, it would have to be the effect on my estimates of your predictive model.
I would make predictions on evaluations of MIRI workshop research outputs by a properly constructed jury of AI people. If the MIRI workshops were many times more productive than comparably or better credentialed AI people according to independent expert judges (blinded to the extent possible) I would say my model was badly wrong, but I don’t think you would predict a win on that.
To avoid “too much work to do/prep for” and “disagreement about far future consequences of mundane predicted intermediates” you could give me a list of things that you or MIRI plan to attempt over the next 1, 3, and 5 years and I could pick one (with some effort to make it more precise).
DAGGRE...etc
Yes, I have seen you writing about the 80k quiz on LW and 80k and elsewhere, it’s good (although as you mention, test-taking skills went far on it). I predict that if we take an unbiased sample of people with similarly high cognitive test scores, extensive exposure to machine learning, and good career success (drawn from academia and tech/quant finance, say), and look at the top scorers on the 80k quiz and similar, their estimates for MIRI success will quite a bit closer to mine than yours. Do you disagree? Otherwise, I would want to see drastic outperformance relative to such a group on a higher-ceiling version (although this would be confounded by advance notice and the opportunity to study/prepare).
DAGGRE is going into the area of technology, not just geopolitics. Unfortunately it is mostly short term stuff, not long-term basic science, or subtle properties of future tech, so the generalization is imperfect. Also, would you predict exceptional success in predicting short-medium term technological developments?
So, if von Neumann came out with similar FAI claims...
...showing that, e.g. your math problem-solving ability is greater than my point estimate, wouldn’t be very relevant.
The question is not what convinces you that I can do FAI within the framework of your antiheroic epistemology. The question is what first and earliest shows that your antiheroic epistemology is yielding bad predictions. Is this a terrible question to ask for some reason? You’ve substituted an alternate question a couple of times now.
Also, would you predict exceptional success in predicting short-medium term technological developments?
From my perspective, you just asked how bad other people are at predicting such developments. The answer is that I don’t know. Certainly many bloggers are terrible at it. I don’t suppose you can give a quick example of a DAGGRE question?
The question is not what convinces you that I can do FAI within the framework of your antiheroic epistemology.
The question is what first and earliest shows that your antiheroic epistemology is yielding bad predictions
Which I said in the very same paragraph.
Is this a terrible question to ask for some reason? You’ve substituted an alternate question a couple of times now.
I already gave the example of independent judges evaluating MIRI workshop output, among others. If we make the details precise, I can set the threshold on the measure. Or we can take any number of other metrics with approximately continuous outputs where I can draw a line. But it takes work to define a metric precise enough to be solid, and I don’t want to waste my time generating more and more additional examples or making them ultra-precise without feedback on what you will actually stake a claim on.
I can’t determine what’s next without knowledge of what you’ll do or try.
I don’t suppose you can give a quick example of a DAGGRE question?
To clear up the ambiguity, does this mean you agree that I can do anything short of what von Neumann did, or that you don’t think it’s possible to get as far as independent judges favorably evaluating MIRI output, or is there some other standard you have in mind? I’m trying to get something clearly falsifiable, but right now I can’t figure out the intended event due to sheer linguistic ambiguity.
I also think that evaluation by academics is a terrible test for things that don’t come with blatant overwhwelming unmistakable undeniable-even-to-humans evidence—e.g. this standard would fail MWI, molecular nanotechnology, cryonics, and would have recently failed ‘high-carb diets are not necessarily good for you’. I don’t particularly expect this standard to be met before the end of the world, and it wouldn’t be necessary to meet it either.
To clear up the ambiguity, does this mean you agree that I can do anything short of what von Neumann did
As I said in my other comment, I would be quite surprised if your individual mathematical and AI contributions reach the levels of the best in their fields, as you are stronger verbally than mathematically, and discuss in more detail what I would find surprising and not there.
I also think that evaluation by academics is a terrible test for things that don’t come with blatant overwhwelming unmistakable undeniable-even-to-humans evidence—e.g. this standard would fail MWI, molecular nanotechnology, cryonics, and would have recently failed ‘high-carb diets are not necessarily good for you’.
I recently talked to Drexler about nanotechnology in Oxford. Nanotechnology is
Way behind Drexler’s schedule, and even accounting for there being far less funding and focused research than he expected, the timeline skeptics get significant vindication
Was said by the NAS panel to be possible, with no decisive physical or chemical arguments against (and discussion of some uncertainties which would not much change the overall picture, in any case), and arguments against tend to be or turn into timeline skepticism and skepticism about the utility of research
Has not been the subject of a more detailed report or expert judgment test than the National Academy of Sciences one (which said it’s possible) because Drexler was not on the ball and never tried. He is currently working with the FHI to get a panel of independent eminent physicists and chemists to work it over, and expects them to be convinced.
Tomer is indeed pretty great, but I have heard Michael say things like that about a number of people and projects over the years. Most did not become like Quixey.
Also, while it seems to me that Michael should have said this about many people, I have not actually heard him say this about many people, to me, except Alyssa Vance.
That’s what the arguments you’ve given for this have mostly amounted to. You have said “I need to believe this to be motivated and do productive work” in response to questions about the probabilities in the past, while not giving solid reasons for the confidence.
When did you predict that? Early on I did not hear you making such claims, with the tune changing after it became clear that demand for it was good.
4 years ago I did advocate getting Math Olympiad people, and said they could be gotten, and had empirical evidence of that from multiple angles. And I did recognize your writing and fiction were well-received, and had evidence from the reactions to “Staring into the Singularity” and OB/LW. You tried several methods, including the rationality book, distributing written rationality exercises/micromanaging CFAR content, and the fanfiction. Most of them wasted time and resources without producing results, and one succeeded.
And there is a larger context, that in addition to the successes you are highlighting the path includes: Flare, LOGI and associated research, pre-Vassar SI management issues, open-source singularity, commercial software, trying to create non-FAI before nanowars in 2015.
Tomer is indeed pretty great, but I have heard Michael say things like that about a number of people and projects over the years. Most did not become like Quixey. And what’s the analogy here? That people with good ability to predict success in scientific research have indicated you will succeed taking into account how the world and the space of computer science and AI progress must be for that? That Michael has?
This does not sound like something I would ever say. Ever, even at age 16. Your memory conflicts with mine. Is there any way to check?
As a LessWrong reader, I notice that I am confused because this does not sound like something you would say, but I’m not sure I could explain the difference between this and “heroic epistemology.”
EDIT: for the benefit of other readers following the conversation, Eliezer gives a description of heroic epistemology here.
For the record, I don’t recall ever hearing you say something like this in my presence or online, and if somebody had told me in person that you had said this, I think I would’ve raised a skeptical eyebrow and said “Really? That doesn’t sound like something Eliezer would say. Have you read The Sequences?”
But also, I remain confused about the normative content of “heroic epistemology.”
Ask Anna about it, she was present on both occasions, at the tail end of the Singularity Summit workshop discussion in New York City, and at the roundtable meeting at the office with Anna, Paul, Luke, and Louie.
In a related vein, arguments like this are arguments that someone could do A, but not so much that you will do A (and B and C and...). My impression is of too many arguments like the former and enough of the latter. If you can remedy that, it would be great, but it is a fact about the responses I have seen.
Eliezer emailed me to ask me about it (per Carl’s request, above); I emailed him back with the email below, which Eliezer requested I paste into the LW thread. Pasting:
From my internal perspective, the truth-as-I-experience-it is that I’m annoyed when people raise the topic because it’s all wasted motion, the question sets up a trap that forces you into appearing arrogant, and I honestly think that “Screw all this, I’m just going to go ahead and do it and you can debate afterward what the probabilities were” is a perfectly reasonable response.
From the perspective of folks choosing between supporting multiple lines of AI risk reduction effort, of which MIRI is only one, such probability estimates are not wasted effort.
Though your point about appearing arrogant is well taken. It’s unfortunate that it isn’t socially okay to publicly estimate a high probability of success, or to publicly claim one’s own exceptionalism, when ones impressions point that way. It places a barrier toward honest conversation here.
I suspect this annoyance is easily misinterpreted, independent of its actual cause. Most humans respond with annoyance when their plans are criticized. Also, in situations where A has power over B, and where B then shares concerns or criticisms about A’s plans, and where A responds with annoyance or with avoidance of such conversation… B is apt to respond (as I did) by being a bit hesitant to bring the topic up, and by also wondering if A is being defensive.
I’m not saying I was correct here. I’m also not sure what the fix is. But it might be worth setting a 1-minute timer and brainstorming or something.
If you were anyone else, this is ordinarily the point where I tell you that I’m just going to ignore all this and go ahead do it, and then afterward you can explain why it was either predictable in retrospect or a fluke, according to your taste. Since it’s you: What’s the first next goal you think I can’t achieve, strongly enough that if I do it, you give up on non-heroic epistemology?
I’m familiar with this move. But you make it before failing too, so its evidentiary weight is limited, and insufficient for undertakings with low enough prior probability from all the other evidence besides the move.
I don’t buy the framing. The update would be mainly about you and the problem in question, not the applicability of statistics to reality.
Two developments in AI as big as Pearl’s causal networks (as judged by Norvig types) by a small MIRI team would be a limited subset of the problems to be solved by a project trying to build AGI with a different and very safe architecture before the rest of the world, and wouldn’t address the question of the probability that such is needed in the counterfactual, but it would cause me to stop complaining and would powerfully support the model that MIRI can be more productive than the rest of the AI field when currently-available objective indicators put it as a small portion of the quality-adjusted capacity.
If we want a predictor for success that’s a lot better than the vast majority of quite successful entrepreneurs and pathbreaking researchers, making numerous major basic science discoveries and putting them together in a way that saves the world, then we need some evidence to distinguish the team and explain why it will make greater scientific contributions than any other ever with high reliability in a limited time.
A lot of intermediate outcomes would multiply my credence in and thus valuation of the “dozen people race ahead of the rest of the world in AI” scenario, but just being as productive as von Neumann or Turing or Pearl or Einstein would not result in high probability of FAI success, so the evidence has to be substantial.
Sure, you try, sometimes you lose, sometimes you win. On anti-heroic epistemology (non-virtuous to attempt to discriminate within an outside view) there shouldn’t be any impossible successes by anyone you know personally after you met them. They should only happen to other people selected post-facto by the media, or to people who you met because of their previous success.
We disagree about how to use statistics in order to get really actually correct answers. Having such a low estimate of my rationality that you think that I know what correct statistics are, and am refusing to use them, is not good news from an Aumann perspective and fails the ideological Turing Test. In any case, surely if my predictions are correct you should update your belief about good frameworks (see the reasoning used in the Pascal’s Muggle post) - to do otherwise and go on insisting that your framework was nonetheless correct would be oblivious.
...should not have been disclosed to the general world, since proof well short of this should suffice for sufficient funding (Bayes nets were huge), though they might be disclosed to some particular Norvig type on a trusted oversight committee if there were some kind of reason for the risk. Major breakthroughs on the F side of FAI are not likely to be regarded as being as exciting as AGI-useful work like Bayes nets, though they may be equally mathematically impressive or mathematically difficult. Is there some kind of validation which you think MIRI should not be able to achieve on non-heroic premises, such that the results should be disclosed to the general world?
EDIT: Reading through the rest of the comment more carefully, I’m not sure we estimate the same order of magnitude of work for what it takes to build FAI under mildly good background settings of hidden variables. The reason why I don’t think the mainstream can build FAI isn’t that FAI is intrinsically huge a la the Cyc hypothesis. The mainstream is pretty good at building huge straightforward things. I just expect them to run afoul of one of the many instakill gotchas because they’re one or two orders of magnitude underneath the finite level of caring required.
EDIT 2: Also, is there a level short of 2 gigantic breakthroughs which causes you to question non-heroic epistemology? The condition is sufficient, but is it necessary? Do you start to doubt the framework after one giant breakthrough (leaving aside the translation question for now)? If not, what probability would you assign to that, on your framework? Standard Bayesian Judo applies—if you would, as I see it, play the role of the skeptic, then you must either be overly-credulous-for-the-role that we can do heroic things like one giant breakthrough, or else give up your skepticism at an earlier signal than the second. For you cannot say that something is strongly prohibited on your model and yet also refuse to update much if it happens, and this applies to every event which might lie along the way. (Evenhanded application: ’Tis why I updated on Quixey instead of saying “Ah, but blah”; Quixey getting this far just wasn’t supposed to happen on my previous background theory, and shouldn’t have happened even if Vassar had praised ten people to me instead of two.)
I don’t understand why you say this. Given Carl’s IQ and social circle (didn’t he used to work for a hedge fund run by Peter Thiel?) why would it be very surprising that someone he personally knows achieves your current level of success after he meets them?
Carl referenced “Staring Into the Singularity” as an early indicator of your extraordinary verbal abilities (which explains much if not all of your subsequent successes). It suggests that’s how you initially attracted his attention. The same is certainly true for me. I distinctly recall saying to myself “I should definitely keep track of this guy” when I read that, back in the extropian days. Is that enough for you to count as “people who you met because of their previous success”?
In any case, almost everyone who meets you now would count you as such. What arguments can you give to them that “heroic epistemology” is normative (and hence they are justified in donating to MIRI)?
To state my overall position on the topic being discussed, I think according to “non-heroic epistemology”, after someone achieves an “impossible success”, you update towards them being able to achieve further successes of roughly the same difficulty and in related fields that use similar skills, but the posterior probabilities of them solving much more difficult problems or in fields that use very different skills remain low (higher relative to the prior, but still low in an absolute sense). Given my understanding of the distribution of cognitive abilities in humans, I don’t see why I would ever “give up” this epistemology, unless you achieved a level of success that made me suspect that you’re an alien avatar or something.
Yes, no matter how many impossible things you do, the next person you meet thinks that they only heard of you because of them, ergo selection bias. This is an interesting question purely on a philosophical level—it seems to me to have some of the flavor of quantum suicide experiments where you can’t communicate your evidence. In principle this shouldn’t happen without quantum suicide for logically omniscient entities who already know the exact fraction of people with various characteristics, i.e., agree on exact priors, but I think it might start happening again to people who are logically unsure about which framework they should use.
To avoid talking past one another: I agree that one can and should update on evidence beyond the most solid empirical reference classes in predicting success. If you mean to say that a majority of the variation on a log scale in success (e.g. in wealth or scientific productivity) can be accounted for with properties of individuals and their circumstances, beyond dumb luck then we can agree on that. Some of those characteristics are more easily observed, while others are harder to discern or almost unmeasurable from a distance so that track records may be our best way to discern them.
That is to say, repeated successes should not be explained by luck, but by updating estimates of hard-to-observe characteristics and world model.
The distribution of successes and failures you have demonstrated is not “impossible” or driving a massive likelihood ratio given knowledge about your cognitive and verbal ability, behavioral evidence of initiative and personality, developed writing skill (discernible through inspection and data about its reception), and philosophical inclinations. Using measurable features, and some earlier behavioral or track record data one can generate reference classes with quite high levels of lifetime success, e.g. by slicing and dicing cohorts like this one. Updating on further successes delivers further improvements.
But updating on hidden characteristics does not suffer exponential penalties like chance explanations, and there is a lot of distance to cover in hidden characteristics before a 10% probability of MIRI-derived FAI (or some other causal channel) averting existential catastrophe that would have occurred absent MIRI looks reasonable.
Now large repeated updates about hidden characteristics still indicate serious model problems and should lead us to be very skeptical of those models. However, I don’t see such very large surprising updates thus far.
If difficulty is homogenous (at least as far as one can discern in advance), then we can use these data straightforwardly, but a lot of probability will be peeled off relative to “Tomer must win.” And generalizing to much higher difficulty is still dubious for the reasons discussed above.
This is not what I meant. I didn’t claim you would explicitly endorse a contradiction formally. But nonetheless, the impression I got was of questions about probability met with troubling responses like talking about the state of mind you need for work and wanting to not think in terms of probabilities of success for your own work. That seems a bad signal because of the absence of good responses, and the suggestion that the estimates may not be the result of very much thought, or may be unduly affected by their emotional valence, without ever saying “p and not p.”
As I said elsewhere a 10% probability of counterfactually saving the world is far above the threshold for action. One won’t get to high confidence in that low prior claim without extraordinary evidence, but the value of pursuing it increases continuously with intermediate levels of evidence. Some examples would be successful AI researchers coming to workshops and regularly saying that the quality and productivity of the research group and process was orders of magnitude more productive, the results very solid, etc. This is one of the reasons I like the workshop path, because it exposes the thesis to empirical feedback.
Although as we have discussed with AI folk, there are also smart AI people who would like to find nice clean powerful algorithms with huge practical utility without significant additional work.
Yes, we do still have disagreements about many of the factual questions that feed into a probability estimate, and if I adopted your view on all of those except MIRI productivity there would be much less of a gap. There are many distinct issues going into the estimation of a probability of your success, from AGI difficulty, to FAI difficulty, to the competence of regular AI people and governance institutions, the productivity of a small MIRI team, the productivity of the rest of the world, signs of AI being close, reactions to those signs, and others.
There are a number of connections between these variables, but even accounting for that your opinions are systematically firmly in the direction of greater personal impact relative to the analyses of others, and the clustering seems tighter than is typical (others seem to vary more, sometimes evaluating different subissues as pointing in different directions). This shows up in attempts to work through the issues for estimation, as at that meeting with Paul et al.
One can apply a bias theory to myself and Paul Christiano and Nick Bostrom and the FHI surveys of AI experts are biased towards normalcy, respectability and conservatism. But I would still question the coincidence of so many substantially-independent variables landing in the same direction, and uncertainty over the pieces hurts the hypothesis that MIRI has, say, a 10% probability of averting a counterfactual existential catastrophe disproportionately.
And it is possible that you have become a superb predictor of such variables in the last 10 years (setting aside earlier poor predictions), and I could and would update on good technological and geopolitical prediction in DAGGRE or the like.
Thanks for talking this out, and let me reiterate that in my expectation your and MIRI’s existence (relative to the counterfactual in which it never existed and you become a science fiction writer) has been a good thing and reduced my expectation for existential risk.
Of course I expect you to say that, since to say otherwise given your previous statements is equivalent to being openly incoherent and I do not regard you so lowly. But I don’t yet believe that you would actually have accepted or predicted those successes ante facto, vs. claiming ante facto that those successes were unlikely and that trying was overconfident. Which is why I repeat my question: What is the least impossible thing I could do next, where anything up to that is permitted by your model so it’s equivalent to affirming that you think I might be able to do it, and anything beyond that was prohibited by your model so it’s time to notice your confusion? I mean, if you think I can make one major AI breakthrough but not two, that’s already a lot of confidence in me… is that really what your outside view would say about me?
Please distinguish between the disputed reality and your personal memory, unless you’re defining the above so broadly (and uncharitably!) that my ‘wasted motion’ FB post counts as an instance.
Without significant work? I don’t think I can do that. Why would you think I thought I could do that?
If enough people agreed on that and DAGGRE could be done with relatively low effort on my part, I would do so, though I think I’d want at least some people committing in writing to large donations given success because it would be a large time commitment and I’m prior-skeptical that people know or are honest about their own reasons for disagreement; and I would expect the next batch of pessimists to write off the DAGGRE results (i.e., claim it already compatible with my known properties) so there’d be no long-term benefit. Still, 8 out of 8 on 80K’s “Look how bad your common sense is!” test, plus I recall getting 9 out of 10 questions correct the last time I was asked for 90% probabilities on a CFAR calibration test, so it’s possible I’ve already outrun the reference class of people who are bad at this.
Though if it’s mostly geopolitical questions where the correct output is “I know I don’t know much about this” modulo some surface scans of which other experts are talking sense, I wouldn’t necessarily expect to outperform the better groups that have already read up on cognitive rationality and done a few calibration exercises.
So, if von Neumann came out with similar FAI claims, but couldn’t present compelling arguments to his peers (if not to exact agreement, perhaps within an order of magnitude) I wouldn’t believe him. So showing that, e.g. your math problem-solving ability is greater than my point estimate, wouldn’t be very relevant. Shocking achievements would lead me to upgrade my estimate of your potential contribution going forward (although most of the work in an FAI team would be done by others in any case), resolving uncertainty about ability, but that would not be enough as such, it would have to be the effect on my estimates of your predictive model.
I would make predictions on evaluations of MIRI workshop research outputs by a properly constructed jury of AI people. If the MIRI workshops were many times more productive than comparably or better credentialed AI people according to independent expert judges (blinded to the extent possible) I would say my model was badly wrong, but I don’t think you would predict a win on that.
To avoid “too much work to do/prep for” and “disagreement about far future consequences of mundane predicted intermediates” you could give me a list of things that you or MIRI plan to attempt over the next 1, 3, and 5 years and I could pick one (with some effort to make it more precise).
Yes, I have seen you writing about the 80k quiz on LW and 80k and elsewhere, it’s good (although as you mention, test-taking skills went far on it). I predict that if we take an unbiased sample of people with similarly high cognitive test scores, extensive exposure to machine learning, and good career success (drawn from academia and tech/quant finance, say), and look at the top scorers on the 80k quiz and similar, their estimates for MIRI success will quite a bit closer to mine than yours. Do you disagree? Otherwise, I would want to see drastic outperformance relative to such a group on a higher-ceiling version (although this would be confounded by advance notice and the opportunity to study/prepare).
DAGGRE is going into the area of technology, not just geopolitics. Unfortunately it is mostly short term stuff, not long-term basic science, or subtle properties of future tech, so the generalization is imperfect. Also, would you predict exceptional success in predicting short-medium term technological developments?
The question is not what convinces you that I can do FAI within the framework of your antiheroic epistemology. The question is what first and earliest shows that your antiheroic epistemology is yielding bad predictions. Is this a terrible question to ask for some reason? You’ve substituted an alternate question a couple of times now.
From my perspective, you just asked how bad other people are at predicting such developments. The answer is that I don’t know. Certainly many bloggers are terrible at it. I don’t suppose you can give a quick example of a DAGGRE question?
Which I said in the very same paragraph.
I already gave the example of independent judges evaluating MIRI workshop output, among others. If we make the details precise, I can set the threshold on the measure. Or we can take any number of other metrics with approximately continuous outputs where I can draw a line. But it takes work to define a metric precise enough to be solid, and I don’t want to waste my time generating more and more additional examples or making them ultra-precise without feedback on what you will actually stake a claim on.
I can’t determine what’s next without knowledge of what you’ll do or try.
http://blog.daggre.org/tag/prediction-market/
To clear up the ambiguity, does this mean you agree that I can do anything short of what von Neumann did, or that you don’t think it’s possible to get as far as independent judges favorably evaluating MIRI output, or is there some other standard you have in mind? I’m trying to get something clearly falsifiable, but right now I can’t figure out the intended event due to sheer linguistic ambiguity.
I also think that evaluation by academics is a terrible test for things that don’t come with blatant overwhwelming unmistakable undeniable-even-to-humans evidence—e.g. this standard would fail MWI, molecular nanotechnology, cryonics, and would have recently failed ‘high-carb diets are not necessarily good for you’. I don’t particularly expect this standard to be met before the end of the world, and it wouldn’t be necessary to meet it either.
As I said in my other comment, I would be quite surprised if your individual mathematical and AI contributions reach the levels of the best in their fields, as you are stronger verbally than mathematically, and discuss in more detail what I would find surprising and not there.
I recently talked to Drexler about nanotechnology in Oxford. Nanotechnology is
Way behind Drexler’s schedule, and even accounting for there being far less funding and focused research than he expected, the timeline skeptics get significant vindication
Was said by the NAS panel to be possible, with no decisive physical or chemical arguments against (and discussion of some uncertainties which would not much change the overall picture, in any case), and arguments against tend to be or turn into timeline skepticism and skepticism about the utility of research
Has not been the subject of a more detailed report or expert judgment test than the National Academy of Sciences one (which said it’s possible) because Drexler was not on the ball and never tried. He is currently working with the FHI to get a panel of independent eminent physicists and chemists to work it over, and expects them to be convinced.
Also, while it seems to me that Michael should have said this about many people, I have not actually heard him say this about many people, to me, except Alyssa Vance.