Fwiw I’m not aware of using or understanding ‘outside view’ to mean something other than basically reference class forecasting (or trend extrapolation, which I’d say is the same). In your initial example, it seems like the other person is using it fine—yes, if you had more examples of an AGI takeoff, you could do better reference class forecasting, but their point is that in the absence of any examples of the specific thing, you also lack other non-reference-class-forecasting methods (e.g. a model), and you lack them even more than you lack relevant reference classes. They might be wrong, but it seems like a valid use. I assume you’re right that some people do use the term for other stuff, because they say so in the comments, but is it actually that common?
I don’t follow your critique of doing an intuitively-weighted average of outside view and some inside view. In particular, you say ‘This is not Tetlock’s advice, nor is it the lesson from the forecasting tournaments...‘. But in the blog post section that you point to, you say ‘Tetlock’s advice is to start with the outside view, and then adjust using the inside view.’, which sounds like he is endorsing something very similar, or a superset of the thing you’re citing him as disagreeing with?
Fwiw I’m not aware of using or understanding ‘outside view’ to mean something other than basically reference class forecasting (or trend extrapolation, which I’d say is the same). … I assume you’re right that some people do use the term for other stuff, because they say so in the comments, but is it actually that common?
Fair enough if your experience is different than mine. Lots of people I’ve talked to seem to have had experiences similar to mine. To be clear it’s not just that I’ve seen other people abusing the concept, I think I’ve caught myself doing it too on several occasions. I think quite a lot of things can be thought of as a form of reference class forecasting if you stretch it enough, but that’s true of a lot of things, e.g. quite a lot of things can be thought of as a form of modelling, or logical deduction, or intuition, if you stretch those concepts far enough.
In your initial example, it seems like the other person is using it fine—yes, if you had more examples of an AGI takeoff, you could do better reference class forecasting, but their point is that in the absence of any examples of the specific thing, you also lack other non-reference-class-forecasting methods (e.g. a model), and you lack them even more than you lack relevant reference classes. They might be wrong, but it seems like a valid use.
I guess we are getting into the question of what the charitable interpretation of their statement is. We could interpret it in the way you mentioned, but then it would be a pretty big and obvious non sequitur—Obviously reference classes work less well the fewer examples of the thing (and things similar to the thing) you have, but part of what’s interesting about other things (e.g. deduction, gears-level models) is that they often work fine in completely novel cases. For example, the first human landing on the Moon was not achieved via reference classes, but via gears-level modelling.
I don’t follow your critique of doing an intuitively-weighted average of outside view and some inside view. In particular, you say ‘This is not Tetlock’s advice, nor is it the lesson from the forecasting tournaments...‘. But in the blog post section that you point to, you say ‘Tetlock’s advice is to start with the outside view, and then adjust using the inside view.’, which sounds like he is endorsing something very similar, or a superset of the thing you’re citing him as disagreeing with?
Do you follow it in the case where “outside view” has the expansive, ambiguous new meaning I’ve complained about? I feel like it’s pretty clear that’s not what Tetlock meant.
In the case where “outside view” means reference classes, here are my complaints:
1. I don’t think the “only use inside view if you feel like you are a qualified expert” bit is justified by Tetlock’s advice. He tells everyone to adjust with inside view, and he has all this evidence about how experts suck when they use inside view.
2. “Have some outside views, and some inside views, and then aggregate them” is not the same thing as “start with an outside view, then adjust it, then get an entirely different perspective and repeat, then aggregate the perspectives.” And note that even that is not exactly what he said IMO; my summary of his advice at the time was as follows:
Tetlock describes how superforecasters go about making their predictions.56 Here is an attempt at a summary:
1. Sometimes a question can be answered more rigorously if it is first “Fermi-ized,” i.e. broken down into sub-questions for which more rigorous methods can be applied.
2. Next, use the outside view on the sub-questions (and/or the main question, if possible). You may then adjust your estimates using other considerations (‘the inside view’), but do this cautiously.
3. Seek out other perspectives, both on the sub-questions and on how to Fermi-ize the main question. You can also generate other perspectives yourself.
4. Repeat steps 1 – 3 until you hit diminishing returns.
5. Your final prediction should be based on an aggregation of various models, reference classes, other experts, etc.
Notice how step 1 is a very inside-viewy sort of thing; you are reasoning about the structure of the thing, breaking it down into parts, etc. I imagine the reason to do this is that often it’s possible to find good reference classes (or other rigorous ways to estimate) the parts, but not the whole. Perhaps there’s also a motivation about making errors cancel out, idk. Anyhow my point is that this methodology, (which is a fuller and less lossy version of Tetlock’s advice than the slogan “start with outside view, then adjust,”) is importantly different from the “Have an outside view, and an inside view if you are an expert, and then aggregate them.”
Fwiw I’m not aware of using or understanding ‘outside view’ to mean something other than basically reference class forecasting (or trend extrapolation, which I’d say is the same). In your initial example, it seems like the other person is using it fine—yes, if you had more examples of an AGI takeoff, you could do better reference class forecasting, but their point is that in the absence of any examples of the specific thing, you also lack other non-reference-class-forecasting methods (e.g. a model), and you lack them even more than you lack relevant reference classes. They might be wrong, but it seems like a valid use. I assume you’re right that some people do use the term for other stuff, because they say so in the comments, but is it actually that common?
I don’t follow your critique of doing an intuitively-weighted average of outside view and some inside view. In particular, you say ‘This is not Tetlock’s advice, nor is it the lesson from the forecasting tournaments...‘. But in the blog post section that you point to, you say ‘Tetlock’s advice is to start with the outside view, and then adjust using the inside view.’, which sounds like he is endorsing something very similar, or a superset of the thing you’re citing him as disagreeing with?
Thanks for the pushback!
Fair enough if your experience is different than mine. Lots of people I’ve talked to seem to have had experiences similar to mine. To be clear it’s not just that I’ve seen other people abusing the concept, I think I’ve caught myself doing it too on several occasions. I think quite a lot of things can be thought of as a form of reference class forecasting if you stretch it enough, but that’s true of a lot of things, e.g. quite a lot of things can be thought of as a form of modelling, or logical deduction, or intuition, if you stretch those concepts far enough.
I guess we are getting into the question of what the charitable interpretation of their statement is. We could interpret it in the way you mentioned, but then it would be a pretty big and obvious non sequitur—Obviously reference classes work less well the fewer examples of the thing (and things similar to the thing) you have, but part of what’s interesting about other things (e.g. deduction, gears-level models) is that they often work fine in completely novel cases. For example, the first human landing on the Moon was not achieved via reference classes, but via gears-level modelling.
Do you follow it in the case where “outside view” has the expansive, ambiguous new meaning I’ve complained about? I feel like it’s pretty clear that’s not what Tetlock meant.
In the case where “outside view” means reference classes, here are my complaints:
1. I don’t think the “only use inside view if you feel like you are a qualified expert” bit is justified by Tetlock’s advice. He tells everyone to adjust with inside view, and he has all this evidence about how experts suck when they use inside view.
2. “Have some outside views, and some inside views, and then aggregate them” is not the same thing as “start with an outside view, then adjust it, then get an entirely different perspective and repeat, then aggregate the perspectives.” And note that even that is not exactly what he said IMO; my summary of his advice at the time was as follows:
Notice how step 1 is a very inside-viewy sort of thing; you are reasoning about the structure of the thing, breaking it down into parts, etc. I imagine the reason to do this is that often it’s possible to find good reference classes (or other rigorous ways to estimate) the parts, but not the whole. Perhaps there’s also a motivation about making errors cancel out, idk. Anyhow my point is that this methodology, (which is a fuller and less lossy version of Tetlock’s advice than the slogan “start with outside view, then adjust,”) is importantly different from the “Have an outside view, and an inside view if you are an expert, and then aggregate them.”