“In the past, people who have offered such apparently-very-lucrative deals have usually been scammers, cheaters, and liars. And, in general, we have on many occasions observed people lying, scamming, cheating, etc. On the other hand, we have only very rarely seen such an apparently-very-lucrative deal turn out to actually be a good idea. Therefore, on the general principle that the future will be similar to the past, we predict a very high chance that Bernie is a cheating, lying scammer, and that this so-called ‘investment opportunity’ is fake.”
We thus defeat the Spokesperson’s argument on his own terms, without needing to get into abstractions or theory—and we do it in one paragraph.
This happens to also be precisely the correct approach to take in real life when faced with apparently-very-lucrative deals and investment opportunities (unless you have the time to carefully investigate, in great detail and with considerable diligence, all such deals that are offered to you).
Ah, but there is some non-empirical cognitive work done here that is really relevant, namely the choice of what equivalence class to put Bernie Bankman into when trying to forecast. In the dialogue, the empiricists use the equivalence class of Bankman in the past, while you propose using the equivalence class of all people that have offered apparently-very-lucrative deals.
And this choice is in general non-trivial, and requires abstractions and/or theory. (And the dismissal of this choice as trivial is my biggest gripe with folk-frequentism—what counts as a sample, and what doesn’t?)
I disagree. It seems to me that this choice is, in general, pretty easy to make, and takes naught but common sense. Certainly that’s the case in the given example scenario. Of course there are exceptions, where the choice of reference class is trickier—but in general, no, it’s pretty easy.
(Whether the choice “requires abstractions and/or theory” is another matter. Perhaps it does, in a technical sense. But it doesn’t particularly require talking about abstractions and/or theory, and that matters.)
Sure, there is common sense, available to plenty of people, of which reference classes apply to Ponzi schemes (but, somehow, not to everybody, far from it). Yudkowsky’s point, however, is that the issue of future AIs is entirely analogous, so people who disagree with him on this are as dumb as those taken in by Bernies and Bankmans. Which just seems empirically false—I’m sure that the proportion of AI doom skeptics among ML experts is much higher than that that of Ponzi believers among professional economists. So, if there is progress to be made here, it probably lies in grappling with whatever asymmetries are between these situations. Telling skeptics a hundredth time that they’re just dumb doesn’t look promising.
I mean, the Spokesperson is being dumb, the Scientist is being confused. Most AI researchers aren’t even being Scientists, they have different theoretical models than EY. But some of them don’t immediately discount the Spokesperson’s false-empiricism argument publicly, much like the Scientist tries not to. I think the latter pattern is what has annoyed EY and what he writes against here.
However, a large number of current AI experts do recently seem to be boldly claiming that LLMs will never be sufficient for even AGI, not to mention ASI. So maybe it’s also aimed at them a bit.
But some of them don’t immediately discount the Spokesperson’s false-empiricism argument publicly
Most likely as a part of the usual arguments-as-soldiers political dynamic.
I do think that there’s an actual argument to be made that we have much less empirical evidence regarding AIs compared to Ponzis, and plently of people on both sides of this debate are far too overconfident in their grand theories, EY very much included.
I agree that there is some non-empirical cognitive work to be done in choosing how to weight different reference classes. How much do we weight the history of Ponzi Pyramid Inc, the history of Bernie Bankman, the history of the stock market, and the history of apparently-very-lucrative deals? This is all useful work to do to estimate the risk of investing in PP Inc.
However, the mere existence of other possible reference classes is sufficient to defeat the Spokesperson’s argument, because it shows that his arguments lead to a contradiction.
Unfortunately in the world I live in, the same people who would accept “This is obviously a Ponzi scheme” (but who don’t understand AI x-risk well) have to also contend with the fact that most people they hear talking about AI are indistinguishable (to them) from people talking about crypto as an investment, or about how transformative AI will lead to GDP doubling times dropping to years, months, or weeks. So, the same argument could be used to get (some of) them to dismiss the notion that AI could become that powerful at all with even less seeming-weirdness.
Arguments that something has the form of a Ponzi scheme are, fortunately and unfortunately, not always correct. Some changes really do enable permanently (at least on the timescales the person thinks of as permanent) faster growth.
I don’t say that you’re wrong, necessarily, but what would you say is an example of something that “has the form of a Ponzi scheme”, but is actually a change that enables permanently faster growth?
From the outside, depending on your level of detail of understanding, any franchise could look that way. Avon and Tupperware look a bit that way. Some MLM companies are more legitimate than others.
From a more abstract point of view, I could argue that “cities” are an example. “Hey, send your kids to live here, let some king and his warriors be in charge, and give up your independence, and you’ll all get richer!” It wasn’t at all clear in the beginning how “Pay taxes and die of diseases!” was going to be good for anyone but the rulers, but the societies that did it more and better thrived and won.
Yeah, you’re right, but for most of history they were net population sinks that generated outsized investment returns. Today they’re not population sinks because of sanitation etc. etc.
I know I’m being imprecise and handwavy, so feel free to ignore me, but really my thought was just that lots of things look vaguely like ponzi schemes without getting into more details than most people are going to pay attention to.
I think this would be a good argument against Said Achmiz’s suggested response, but I feel the text doesn’t completely support it, e.g. the Epistemologist says “such schemes often go through two phases” and “many schemes like that start with a flawed person”, suggesting that such schemes are known to him.
Even setting aside such textual anomalies, why is this a good argument? As I noted in a sibling comment to yours, my response assumes that Ponzi schemes have never happened in this world, because otherwise we’d simply identify the Spokesperson’s plan as a Ponzi scheme! The reasoning that I described is only necessary because we can’t say “ah, a Ponzi scheme”!
Ah, I think there was a misunderstanding. I (and maybe also quetzal_rainbow?) thought that in the inverted world also no “apparently-very-lucrative deals” that turn out to be scams are known, whereas you made a distinction between those kind of deals and Ponzi schemes in particular.
I think my interpretation is more in the spirit of the inversion, otherwise the Epistemologist should really have answered as you suggested, and the whole premise of the discussion (people seem to have trouble understanding what the Spokesperson is doing) is broken.
If I was living in a world where there are zero observed apparently-very-lucrative deals that turn out to be scams then I hope I would conclude that there is some supernatural Creator who is putting a thumb on the scale to be sure that cheaters never win and winners never cheat. So I would invest in Ponzi Pyramid Inc. I would not expect to be scammed, because this is a world where there are zero observed apparently-very-lucrative deals that turn out to be scams. I would aim to invest in a diversified portfolio of apparently-very-lucrative deals, for all the same reasons I have a diversified portfolio in this world.
In such a world the Epistemologist is promoting a world model that does not explain my observations and I would not take their investment advice, similarly to how in this world I ignore investment advice from people who believe that the economy is secretly controlled by lizard people.
If the premise is a world where nobody ever does any scams or tries to swindle anyone out of money, then it’s so far removed from our world that I don’t rightly know how to interpret any of the included commentary on human nature / psychology / etc. Lying for personal gain is one of those “human universals”, without which I wouldn’t even recognize the characters as anything resembling humans.
The S&P500 has returned an average of ~8%/year for the past 30 years. As you say, we have on many occasions observed people lying, cheating, and scamming. But we have only rarely observed lucrative good ideas! Why, even banks, which claim much more safety and offer much lower returns than the stock market, have frequently gone bust!
It follows inevitably, therefore, that there is a very high chance that the S&P 500, and the stock market in general, is a scam, and will steal all your money.
It follows further that the only safe investment approach is to put all your money into something that you retain personal custody of. Like gold bars buried in your backyard! Or Bitcoin!
It follows inevitably, therefore, that there is a very high chance that the S&P 500, and the stock market in general, is a scam, and will steal all your money.
Well, here’s a question: what happens more often—stock market downturns, or banks going bust?
It follows further that the only safe investment approach is to put all your money into something that you retain personal custody of. Like gold bars buried in your backyard! Or Bitcoin!
Now this is simply an invalid extrapolation. Note that I made no claims along these lines about what does or does not supposedly follow. Claims like “X reasoning is invalid” / “Y plan is unlikely to work” stand on their own; “what is the correct reasoning” / “what is a good plan” is a wholly separate question.
Well, people prophesying doom in general have a pretty poor track record, so if that’s all we know, our prior should be that any such person is likely to be very wrong.
Of course, most people throughout history who have prophesied doom have had in mind a religious sort of doom. People prophesying doom from technological advance specifically have a better track record. The Luddites were correct, for example. (Their chosen remedy left something to be desired, of course; but that is common, sadly. Identifying the problem does not, by itself, suffice to solve the problem.) And we’ve had quite a bit of doom from technological advance. Indeed, as technology has advanced, we’ve had more and more doom from that advance.
So, on the whole, I’d say that applying the reasoning I describe to people prophesying doom from technological advance is that there is probably something to what they say, even if their specific predictions are not spot-on.
This is in reference to the Luddites, I suppose? If so, “some people’s jobs being automated” is rather a glib description of the early effects of industrialization. There was considerable disruption and chaos, which, indeed, is “doom”, of more or less the sort that the Luddites predicted. (They never claimed that the world would end as a result of the new machines, as far as I know.)
Or:
“In the past, people who have offered such apparently-very-lucrative deals have usually been scammers, cheaters, and liars. And, in general, we have on many occasions observed people lying, scamming, cheating, etc. On the other hand, we have only very rarely seen such an apparently-very-lucrative deal turn out to actually be a good idea. Therefore, on the general principle that the future will be similar to the past, we predict a very high chance that Bernie is a cheating, lying scammer, and that this so-called ‘investment opportunity’ is fake.”
We thus defeat the Spokesperson’s argument on his own terms, without needing to get into abstractions or theory—and we do it in one paragraph.
This happens to also be precisely the correct approach to take in real life when faced with apparently-very-lucrative deals and investment opportunities (unless you have the time to carefully investigate, in great detail and with considerable diligence, all such deals that are offered to you).
Ah, but there is some non-empirical cognitive work done here that is really relevant, namely the choice of what equivalence class to put Bernie Bankman into when trying to forecast. In the dialogue, the empiricists use the equivalence class of Bankman in the past, while you propose using the equivalence class of all people that have offered apparently-very-lucrative deals.
And this choice is in general non-trivial, and requires abstractions and/or theory. (And the dismissal of this choice as trivial is my biggest gripe with folk-frequentism—what counts as a sample, and what doesn’t?)
I disagree. It seems to me that this choice is, in general, pretty easy to make, and takes naught but common sense. Certainly that’s the case in the given example scenario. Of course there are exceptions, where the choice of reference class is trickier—but in general, no, it’s pretty easy.
(Whether the choice “requires abstractions and/or theory” is another matter. Perhaps it does, in a technical sense. But it doesn’t particularly require talking about abstractions and/or theory, and that matters.)
Sure, there is common sense, available to plenty of people, of which reference classes apply to Ponzi schemes (but, somehow, not to everybody, far from it). Yudkowsky’s point, however, is that the issue of future AIs is entirely analogous, so people who disagree with him on this are as dumb as those taken in by Bernies and Bankmans. Which just seems empirically false—I’m sure that the proportion of AI doom skeptics among ML experts is much higher than that that of Ponzi believers among professional economists. So, if there is progress to be made here, it probably lies in grappling with whatever asymmetries are between these situations. Telling skeptics a hundredth time that they’re just dumb doesn’t look promising.
I mean, the Spokesperson is being dumb, the Scientist is being confused. Most AI researchers aren’t even being Scientists, they have different theoretical models than EY. But some of them don’t immediately discount the Spokesperson’s false-empiricism argument publicly, much like the Scientist tries not to. I think the latter pattern is what has annoyed EY and what he writes against here.
However, a large number of current AI experts do recently seem to be boldly claiming that LLMs will never be sufficient for even AGI, not to mention ASI. So maybe it’s also aimed at them a bit.
Most likely as a part of the usual arguments-as-soldiers political dynamic.
I do think that there’s an actual argument to be made that we have much less empirical evidence regarding AIs compared to Ponzis, and plently of people on both sides of this debate are far too overconfident in their grand theories, EY very much included.
I agree that there is some non-empirical cognitive work to be done in choosing how to weight different reference classes. How much do we weight the history of Ponzi Pyramid Inc, the history of Bernie Bankman, the history of the stock market, and the history of apparently-very-lucrative deals? This is all useful work to do to estimate the risk of investing in PP Inc.
However, the mere existence of other possible reference classes is sufficient to defeat the Spokesperson’s argument, because it shows that his arguments lead to a contradiction.
Apparently, the dialogue is happening in inverted world—Ponzi schemes have never happened here and everybody agrees on AI X-risk problem.
Yes. (If it were otherwise, then the response would be even simpler: “oh, this is obviously just a Ponzi scheme”.)
Unfortunately in the world I live in, the same people who would accept “This is obviously a Ponzi scheme” (but who don’t understand AI x-risk well) have to also contend with the fact that most people they hear talking about AI are indistinguishable (to them) from people talking about crypto as an investment, or about how transformative AI will lead to GDP doubling times dropping to years, months, or weeks. So, the same argument could be used to get (some of) them to dismiss the notion that AI could become that powerful at all with even less seeming-weirdness.
Arguments that something has the form of a Ponzi scheme are, fortunately and unfortunately, not always correct. Some changes really do enable permanently (at least on the timescales the person thinks of as permanent) faster growth.
I don’t say that you’re wrong, necessarily, but what would you say is an example of something that “has the form of a Ponzi scheme”, but is actually a change that enables permanently faster growth?
From the outside, depending on your level of detail of understanding, any franchise could look that way. Avon and Tupperware look a bit that way. Some MLM companies are more legitimate than others.
From a more abstract point of view, I could argue that “cities” are an example. “Hey, send your kids to live here, let some king and his warriors be in charge, and give up your independence, and you’ll all get richer!” It wasn’t at all clear in the beginning how “Pay taxes and die of diseases!” was going to be good for anyone but the rulers, but the societies that did it more and better thrived and won.
That… does not seem like a historically accurate account of the formation and growth of cities.
Yeah, you’re right, but for most of history they were net population sinks that generated outsized investment returns. Today they’re not population sinks because of sanitation etc. etc.
I know I’m being imprecise and handwavy, so feel free to ignore me, but really my thought was just that lots of things look vaguely like ponzi schemes without getting into more details than most people are going to pay attention to.
I think this would be a good argument against Said Achmiz’s suggested response, but I feel the text doesn’t completely support it, e.g. the Epistemologist says “such schemes often go through two phases” and “many schemes like that start with a flawed person”, suggesting that such schemes are known to him.
Even setting aside such textual anomalies, why is this a good argument? As I noted in a sibling comment to yours, my response assumes that Ponzi schemes have never happened in this world, because otherwise we’d simply identify the Spokesperson’s plan as a Ponzi scheme! The reasoning that I described is only necessary because we can’t say “ah, a Ponzi scheme”!
Ah, I think there was a misunderstanding. I (and maybe also quetzal_rainbow?) thought that in the inverted world also no “apparently-very-lucrative deals” that turn out to be scams are known, whereas you made a distinction between those kind of deals and Ponzi schemes in particular.
I think my interpretation is more in the spirit of the inversion, otherwise the Epistemologist should really have answered as you suggested, and the whole premise of the discussion (people seem to have trouble understanding what the Spokesperson is doing) is broken.
If I was living in a world where there are zero observed apparently-very-lucrative deals that turn out to be scams then I hope I would conclude that there is some supernatural Creator who is putting a thumb on the scale to be sure that cheaters never win and winners never cheat. So I would invest in Ponzi Pyramid Inc. I would not expect to be scammed, because this is a world where there are zero observed apparently-very-lucrative deals that turn out to be scams. I would aim to invest in a diversified portfolio of apparently-very-lucrative deals, for all the same reasons I have a diversified portfolio in this world.
In such a world the Epistemologist is promoting a world model that does not explain my observations and I would not take their investment advice, similarly to how in this world I ignore investment advice from people who believe that the economy is secretly controlled by lizard people.
If the premise is a world where nobody ever does any scams or tries to swindle anyone out of money, then it’s so far removed from our world that I don’t rightly know how to interpret any of the included commentary on human nature / psychology / etc. Lying for personal gain is one of those “human universals”, without which I wouldn’t even recognize the characters as anything resembling humans.
<trolling>
The S&P500 has returned an average of ~8%/year for the past 30 years. As you say, we have on many occasions observed people lying, cheating, and scamming. But we have only rarely observed lucrative good ideas! Why, even banks, which claim much more safety and offer much lower returns than the stock market, have frequently gone bust!
It follows inevitably, therefore, that there is a very high chance that the S&P 500, and the stock market in general, is a scam, and will steal all your money.
It follows further that the only safe investment approach is to put all your money into something that you retain personal custody of. Like gold bars buried in your backyard! Or Bitcoin!
</trolling>
Well, here’s a question: what happens more often—stock market downturns, or banks going bust?
Now this is simply an invalid extrapolation. Note that I made no claims along these lines about what does or does not supposedly follow. Claims like “X reasoning is invalid” / “Y plan is unlikely to work” stand on their own; “what is the correct reasoning” / “what is a good plan” is a wholly separate question.
This is perfectly sound reasoning. What does applying it to people prophesying doom, arising from technological advance or otherwise, yield?
Well, people prophesying doom in general have a pretty poor track record, so if that’s all we know, our prior should be that any such person is likely to be very wrong.
Of course, most people throughout history who have prophesied doom have had in mind a religious sort of doom. People prophesying doom from technological advance specifically have a better track record. The Luddites were correct, for example. (Their chosen remedy left something to be desired, of course; but that is common, sadly. Identifying the problem does not, by itself, suffice to solve the problem.) And we’ve had quite a bit of doom from technological advance. Indeed, as technology has advanced, we’ve had more and more doom from that advance.
So, on the whole, I’d say that applying the reasoning I describe to people prophesying doom from technological advance is that there is probably something to what they say, even if their specific predictions are not spot-on.
You consider some people’s jobs being automated an instance of “doom”?
This is in reference to the Luddites, I suppose? If so, “some people’s jobs being automated” is rather a glib description of the early effects of industrialization. There was considerable disruption and chaos, which, indeed, is “doom”, of more or less the sort that the Luddites predicted. (They never claimed that the world would end as a result of the new machines, as far as I know.)