“Most of the sequences are not about rationality, but about things that Eliezer considers cool, such as AI or evolutionary psychology...”
I see that we disagree a lot already at the beginning. You say that rationality is about overcoming biases. I agree with that, but then I am also curious why those biases exist. How I see it, human biases are either random quirks of human evolution (which evolutionary psychology might explain), or something that happens to intelligences in general (and then we should also expect AI to be prone to them).
Also, what are the biases? A frequent approach I have seen is providing a list of “fallacies” that you are supposed to avoid. That is a thing that can easily be abused; if you know enough fallacies, you can dismiss almost everything you do not like (start by rejecting science as a “fallacy of argumenting by authority”, and then use the rest of the list to shoot down any attempt to rederive the knowledge from scratch).
But maybe more importantly, how does this kind of rationality survive under reflection? Rationality defined as avoiding the list of fallacies in Wikipedia or in some textbook… but why exactly this list, as opposed to making my own list, or maybe using some definition of correct thinking provided by a helpful political or religious institution? Why is Wikipedia or a Cambridge Handbook the correct source of the list of fallacies? Sounds to me like a fallacy of authority, or a fallacy of majority, or one of those other things you want me to avoid doing.
What if there is a fallacy that hasn’t been discovered yet? If I proposed one, how would we know whether it should be added to the list? (Is it okay if I edit Wikipedia to add “fallacy of political correctness”? Just kidding.)
“AI is not rationality. AI is machine learning. Just call it what it is.”
The recent debates on LW are often about machine learning, because that is the current hot thing. But other approaches were tried in the past, for example expert systems. Who knows, maybe in hindsight we will laugh about everything that was not machine learning as an obvious dead end. Maybe. Anyway, the artificial intelligence mentioned in the Sequences is defined more broadly.
And if machine learning turns out to be the only way to get artificial intelligence… I think it will be quite important to consider its rationality and biases. Especially when it becomes smarter that humans, or if it starts to control a significant part of economy or military.
If a sufficiently smart artificial intelligence becomes widely accessible as a smartphone app, so that you can ask it any question (voice recognition, you do not even have to type) and it will give you a good answer with probability 99%, and when it becomes cheap enough so that most people can afford it… at that moment, the question of AI rationality and alignment with human values will be more important than human rationality, because at that moment most humans will outsource their thinking to the cloud. (Just like today many people object against learning things, because you can find anything on Google. But on Google, you still need to use the right keywords, separate good info from nonsense, and figure out how to apply this to your current problem. The AI will do all these things for you.)
“AI is not rationality. AI is machine learning. Just call it what it is.”
If we take GPT3 as an example ChatGPT is able to multiply two four-digit numbers while earlier versions of GPT3 didn’t.
The way ChatGPT is able to do that is to automatically go into thinking through the multiplication step by step instead of trusting its “intuition”.
That’s very similar to how a human needs to think through this multiplication step-by-step to get the right answer.
I don’t know exactly how they did it, but I think there’s a good chance that they provided it a lot of relevant training data that suggests this reasoning heuristic when asked to multiply two four-digit numbers.
When it comes to moving from the current ChatGPT capabilities to new capabilities I would expect that a good portion of the work is to think about what heuristics it should use to face new problems and then create training data that demonstrate those heuristics.
This way of thinking about what heuristics should be used is the topic of rationality. Some rationality topics are specific to human minds but plenty is more general and also important for AI.
A bias is an error in weighting or proportion or emphasis. This differs from a fallacy, which is an error in reasoning specifically. Just to make up an example, an attentional bias would be a misapplication of attention—the famous gorilla experiment—but there would be no reasoning underlying this error per se. The ad hominem fallacy contains at least implicit reasoning about truth-valued claims.
Yes, it’s possible that AI could be a concern for rationality. But AI is an object of rationality; in this sense, AI is like carbon emissions; it has room for applied rationality, absolutely, but it is not rationality itself. People who read about AI through this medium are not necessarily learning about rationality. They may be, but they also may not be. As such, the overfocus on AI is a massive departure from the original subject matter, much like how it would be if LessWrong became overwhelmed with ways to reduce carbon emissions.
Anyway—that aside, I actually don’t disagree much at all with most of what you said.
The issue is that when these concerns have been applied to the foundation of a community concerned with the same things, they have been staggeringly wrongheaded and resulted in the disparities between mission statements and practical realities, which is more or less the basis of my objection. I am no stranger to criticizing intellectual communities; I have outright argued that we should expand the federal defunding criteria to include of certain major universities such as UC Berkeley itself. For all of the faults that have been levied against academia — and I have been such a critic of these norms that I was in Tucker Carlson’s book (“Ship of Fools” p. 130) as a Person Rebelling Against Academic Norms — I have never had a discussion as absurd as I have when questioning why MIRI should receive Effective Altruism funding. It was and still is one of the most bizarre and frankly concerning lines of reasoning I’ve ever experienced, especially when contrasted with the position of EA leaders to address homelessness or the drug war. The concept of LessWrong and much of EA, on face, is not objectionable; what has resulted absolutely is.
why MIRI should receive Effective Altruism funding
I guess the argument is that (a) a superhuman AI will probably be developed soon, (b) whether it is properly aligned with human values or not will have tremendous impact on the future of humanity, and (c) MIRI is one of the organizations that take this problem most seriously.
If you agree with all three parts, then the funding makes sense. If you disagree with any one of them, it does not. At least from political perspective, it would be better to not talk about funding missions that require belief in several controversial statements to justify them.
This is partially about plausibility of the claims, and partially about prevention vs reaction. Other EA charities are reactive: a problem already exists, we want to solve it. In case of malaria, it is not about curing the people who are already sick, but about preventing other people from getting sick… but anyway, people sick of malaria already exist.
I was looking for some analogy, when humanity spent a lot of resources on prevention, but I actually don’t remember any. Even recently with covid, a lot of people had to die first; perhaps at the beginning we could have prevented all this, but precisely because it did not happen yet, it didn’t seem important.
“Most of the sequences are not about rationality, but about things that Eliezer considers cool, such as AI or evolutionary psychology...”
I see that we disagree a lot already at the beginning. You say that rationality is about overcoming biases. I agree with that, but then I am also curious why those biases exist. How I see it, human biases are either random quirks of human evolution (which evolutionary psychology might explain), or something that happens to intelligences in general (and then we should also expect AI to be prone to them).
Also, what are the biases? A frequent approach I have seen is providing a list of “fallacies” that you are supposed to avoid. That is a thing that can easily be abused; if you know enough fallacies, you can dismiss almost everything you do not like (start by rejecting science as a “fallacy of argumenting by authority”, and then use the rest of the list to shoot down any attempt to rederive the knowledge from scratch).
But maybe more importantly, how does this kind of rationality survive under reflection? Rationality defined as avoiding the list of fallacies in Wikipedia or in some textbook… but why exactly this list, as opposed to making my own list, or maybe using some definition of correct thinking provided by a helpful political or religious institution? Why is Wikipedia or a Cambridge Handbook the correct source of the list of fallacies? Sounds to me like a fallacy of authority, or a fallacy of majority, or one of those other things you want me to avoid doing.
What if there is a fallacy that hasn’t been discovered yet? If I proposed one, how would we know whether it should be added to the list? (Is it okay if I edit Wikipedia to add “fallacy of political correctness”? Just kidding.)
“AI is not rationality. AI is machine learning. Just call it what it is.”
The recent debates on LW are often about machine learning, because that is the current hot thing. But other approaches were tried in the past, for example expert systems. Who knows, maybe in hindsight we will laugh about everything that was not machine learning as an obvious dead end. Maybe. Anyway, the artificial intelligence mentioned in the Sequences is defined more broadly.
And if machine learning turns out to be the only way to get artificial intelligence… I think it will be quite important to consider its rationality and biases. Especially when it becomes smarter that humans, or if it starts to control a significant part of economy or military.
If a sufficiently smart artificial intelligence becomes widely accessible as a smartphone app, so that you can ask it any question (voice recognition, you do not even have to type) and it will give you a good answer with probability 99%, and when it becomes cheap enough so that most people can afford it… at that moment, the question of AI rationality and alignment with human values will be more important than human rationality, because at that moment most humans will outsource their thinking to the cloud. (Just like today many people object against learning things, because you can find anything on Google. But on Google, you still need to use the right keywords, separate good info from nonsense, and figure out how to apply this to your current problem. The AI will do all these things for you.)
If we take GPT3 as an example ChatGPT is able to multiply two four-digit numbers while earlier versions of GPT3 didn’t.
The way ChatGPT is able to do that is to automatically go into thinking through the multiplication step by step instead of trusting its “intuition”.
That’s very similar to how a human needs to think through this multiplication step-by-step to get the right answer.
I don’t know exactly how they did it, but I think there’s a good chance that they provided it a lot of relevant training data that suggests this reasoning heuristic when asked to multiply two four-digit numbers.
When it comes to moving from the current ChatGPT capabilities to new capabilities I would expect that a good portion of the work is to think about what heuristics it should use to face new problems and then create training data that demonstrate those heuristics.
This way of thinking about what heuristics should be used is the topic of rationality. Some rationality topics are specific to human minds but plenty is more general and also important for AI.
A bias is an error in weighting or proportion or emphasis. This differs from a fallacy, which is an error in reasoning specifically. Just to make up an example, an attentional bias would be a misapplication of attention—the famous gorilla experiment—but there would be no reasoning underlying this error per se. The ad hominem fallacy contains at least implicit reasoning about truth-valued claims.
Yes, it’s possible that AI could be a concern for rationality. But AI is an object of rationality; in this sense, AI is like carbon emissions; it has room for applied rationality, absolutely, but it is not rationality itself. People who read about AI through this medium are not necessarily learning about rationality. They may be, but they also may not be. As such, the overfocus on AI is a massive departure from the original subject matter, much like how it would be if LessWrong became overwhelmed with ways to reduce carbon emissions.
Anyway—that aside, I actually don’t disagree much at all with most of what you said.
The issue is that when these concerns have been applied to the foundation of a community concerned with the same things, they have been staggeringly wrongheaded and resulted in the disparities between mission statements and practical realities, which is more or less the basis of my objection. I am no stranger to criticizing intellectual communities; I have outright argued that we should expand the federal defunding criteria to include of certain major universities such as UC Berkeley itself. For all of the faults that have been levied against academia — and I have been such a critic of these norms that I was in Tucker Carlson’s book (“Ship of Fools” p. 130) as a Person Rebelling Against Academic Norms — I have never had a discussion as absurd as I have when questioning why MIRI should receive Effective Altruism funding. It was and still is one of the most bizarre and frankly concerning lines of reasoning I’ve ever experienced, especially when contrasted with the position of EA leaders to address homelessness or the drug war. The concept of LessWrong and much of EA, on face, is not objectionable; what has resulted absolutely is.
I guess the argument is that (a) a superhuman AI will probably be developed soon, (b) whether it is properly aligned with human values or not will have tremendous impact on the future of humanity, and (c) MIRI is one of the organizations that take this problem most seriously.
If you agree with all three parts, then the funding makes sense. If you disagree with any one of them, it does not. At least from political perspective, it would be better to not talk about funding missions that require belief in several controversial statements to justify them.
This is partially about plausibility of the claims, and partially about prevention vs reaction. Other EA charities are reactive: a problem already exists, we want to solve it. In case of malaria, it is not about curing the people who are already sick, but about preventing other people from getting sick… but anyway, people sick of malaria already exist.
I was looking for some analogy, when humanity spent a lot of resources on prevention, but I actually don’t remember any. Even recently with covid, a lot of people had to die first; perhaps at the beginning we could have prevented all this, but precisely because it did not happen yet, it didn’t seem important.