There is a not necessarily large, but definitely significant chance that developing machine intelligence compatible with human values may very well be the single most important thing that humans have or will ever do, and it seems very likely that economic forces will make strong machine intelligence happen soon, even if we’re not ready for it.
So I have two questions about this: firstly, and this is probably my youthful inexperience talking (a big part of why I’m posting this here), but I see so many rationalists do so much awesome work on things like social justice, social work, medicine, and all kinds of poverty-focused effective altruism, but how can it be that the ultimate fate of humanity to either thrive beyond imagination or perish utterly may rest on our actions in this century, and yet people who recognize this possibility don’t do everything they can to make it go the way we need it to? This sort of segues in to my second question, which is what is the most any person, more specifically, I can do for FAI? I’m still in high school, so there really isn’t that much keeping me from devoting my life to helping the cause of making sure AI is friendly. What would that look like? I’m a village idiot by LW standards, and especially bad at math, so I don’t think I’d be very useful on the “front lines” so to speak, but perhaps I could try to make a lot of money and do FAI-focused EA? I might be more socially oriented/socially capable than many here, perhaps I could try to raise awareness or lobby for legislation?
how can it be that the ultimate fate of humanity to either thrive beyond imagination or perish utterly may rest on our actions in this century, and yet people who recognize this possibility don’t do everything they can to make it go the way we need it to?
Well, ChaosMote already gave part of the answer, but another reason is the idea of comparative advantage. Normally I’d bring up someone like Scott Alexander/Yvain as an example (since he’s repeatedly claimed he’s not good at math and blogs more about politics/general rationality than about AI), but this time, you can just look at yourself. If, as you claim,
I’m a village idiot by LW standards, and especially bad at math, so I don’t think I’d be very useful on the “front lines” so to speak, but perhaps I could try to make a lot of money and do FAI-focused EA? I might be more socially oriented/socially capable than many here, perhaps I could try to raise awareness or lobby for legislation?
then your comparative advantage lies less in theory and more in popularization. Technically, theory might be more important, but if you can net bigger gains elsewhere, then by all means you should do so. To use a (somewhat strained) analogy, think about expected value. Which would you prefer: a guaranteed US $50, or a 10% chance at getting US $300? The raw value of the $300 prize might be greater, but you have to multiply by the probabilities before you can do a comparison. It’s the same here. For some LWers, working on AI is the way to go, but for others who aren’t as good at math, maybe raising money is the best way to do things. And then there’s the even bigger picture: AI might be the most important risk in the end, but what if (say) nuclear war occurs first? A politically-oriented person might do better to go into government or something of the sort, even if that person thinks AI is more important in the long run.
So while it might look somewhat strange that not every LWer is working frantically on AI at first, if you look a little deeper, there’s actually a good reason. (And then there’s also scope insensitivity, hyperbolic discounting, and all that good stuff ChaosMote brought up.) In a sense, you answered your own question when you asked your second.
To address your first question: this has to do with scope insensitivity, hyperbolic discounting, and other related biases. To put it bluntly, most humans are actually pretty bad at maximizing expected utility. For example, when I first head about x-risk, my thought process was definitely not “humanity might be wiped out—that’s IMPORTANT. I need to devote energy to this.” It was more along the lines of “huh; That’s interesting. Tragic, even. Oh well; moving on...”
Basically, we don’t care much about what happens in the distant future, especially if it isn’t guaranteed to happen. We also don’t care much more about humanity than we do about ourselves plus our close ones. Plus we don’t really care about things that don’t feel immediate. And so on. Then end result is that most people’s immediate problems are more important to them then x-risk, even if the latter might be by far the more essential according to utilitarian ethics.
people who recognize this possibility don’t do everything they can to make it go the way we need it to
Despite all talking about rationality, we are still humans with all typical human flaws. Also, it is not obvious which way it needs to go. Even if we had unlimited and infinitely fast processing power, and could solve mathematically all kinds of problems related to Löb’s theorem, I still would have no idea how we could start transferring human values to the AI, considering that even humans don’t understand themselves, and ideas like “AI should find a way to make humans smile” can lead to horrible outcomes. So maybe the first step would be to upload some humans and give them more processing power, but humans can also be horrible (and the horrible ones are actually more likely to seize such power), and the changes caused by uploading could make even nice people go insane.
So, what is the obvious next step, other than donating some money to the research, which will most likely conclude that further research is needed? I don’t want to discourage anyone who donates or does the research, just saying that the situation with the research is frustrating by its lack of feedback. On the scale where 0 is the first electronic computer and 100 is the Friendly AI, are we at least at point 1? If we happen to be there, how would we know that?
So maybe the first step would be to upload some humans and give them more processing power,
I would like this plan, but there are reasons to think that the path to WBE passes through nueromorphic AI which is exceptionally likely to be unfriendly, since the principle is basically to just copy parts of the human brain without understanding how the human brain works.
I don’t agree with this particular argument, but I’ll mention it anyway for the sake of having a devil’s advocate:
The number of lives lost to an extinction event is arguably capped at ~10 billion, or whatever Earth’s carrying capacity is. If you think the AI risk is enough generations out, then it may well be possible to do more good by, say, eliminating poverty faster. A simple mathematical model would suggest that if the singularity is 10 generations away, and Earth will have a constant population of ~10 billion, then 100 billion lives will pass between now and the singularity. A 10% increase in humanity’s average quality of life over that period would be morally equivalent to stopping the singularity.
Now, there are a host of problems with the above argument:
First, it is trying to minimize death rather than maximize life. If you set out to maximize the number of Quality Adjusted Life Years that intelligent life accumulates before it’s extinction, then you should also take into account all of the potential future lives which would be extinguished by an extinction event, rather than just the lives taken by the event itself.
Second, the Future of Humanity Institute has conducted an informal survey of existential risk researchers, asking for estimates of the probability of human extinction in the next 100 years. The median result (not mean so as to minimize the impact of outliers) was ~19%. If that’s a ~20% chance each century, then we can expect humanity to last perhaps 2 or 3 centuries (aka, that’s the half life of a technological civilization). Even 300 years is only maybe 4 or 5 generations, so perhaps 50 billion lives could be effected by eliminating poverty now. Using the same simplistic model as before, that would require a 20% increase in humanity’s average quality of life to be morally equivalent to ~10 billion deaths. That’s a harder target to hit, but it may be even harder still if you consider that poverty is likely to be nearly eliminated in ~100 years. Poverty really has been going down steadily for the last century or so, and in another century we can expect it to be much improved.
Note that both of these points are based on somewhat subjective judgements. Personally, I think Friendly AI research is around the point of diminishing returns. More money would be useful, of course, but I think it would be putting some focus on preemptively addressing other forms of existential risk which may emerge over the next century. Additionally, I think it’s important to play with other factors that go into QALY. Quality of life is already being addressed, and duration of our civilization is starting to be addressed via x-risk reduction. The other factor is the number of lives, which is currently capped at Earth’s carrying capacity of ~10 billion. I’d like to see trillions of lives. Brain uploads are one method as technology improves, but another is space colonization. The cheapest option I see is Directed Panspermia, which is the intentional seeding of new solar systems with dormant single cell life. All other forms of X-risk reduction address the possibility that the Great Filter is ahead of us, but this would hedge that bet. I haven’t done any calculations yet, but donating to organizations like the Mars Society may even turn out to be competitive in terms of QALY/$, if they can tip the political scale between humanity staying in and around Earth, and humanity starting to spread outward, colonizing other planets and eventually other stars over the next couple millennia. It’s hard to put a figure on the expected QALY return, but if quadrillions of lives hang in the balance, that may well tip the scales and make the tens of billions of dollars needed to initiate Mars colonization an extremely good investment.
To elaborate on existing comments, a fourth alternative to FAI theory, Earning To Give, and popularization is strategy research. (That could include research on other risks besides AI.) I find that the fruit in this area is not merely low-hanging but rotting on the ground. I’ve read in old comment threads that Eliezer and Carl Shulman in particular have done a lot of thinking about strategy but very little of it has been written down, and they are very busy people. Circumstances may well dictate retracing a lot of their steps.
You’ve said elsewhere that you have a low estimate of your innate mathematical ability, which would preclude FAI research, but presumably strategy research would require lower aptitude. Things like statistics would be invaluable, but strategy research would also involve a lot of comparatively less technical work, like historical and philosophical analysis, experiments and surveys, literature reviews, lots and lots of reading, etc. Also, you’ve already done a bit of strategizing; if you are fulfilled by thinking about those things and you think your abilities meet the task, then it might be a good alternative.
which is what is the most any person, more specifically, I can do for FAI?
First thing you should do is talk to the people that are already involved in this. CFAR seems to be the gateway for man people (at least, it was for me).
There is a not necessarily large, but definitely significant chance that developing machine intelligence compatible with human values may very well be the single most important thing that humans have or will ever do, and it seems very likely that economic forces will make strong machine intelligence happen soon, even if we’re not ready for it.
So I have two questions about this: firstly, and this is probably my youthful inexperience talking (a big part of why I’m posting this here), but I see so many rationalists do so much awesome work on things like social justice, social work, medicine, and all kinds of poverty-focused effective altruism, but how can it be that the ultimate fate of humanity to either thrive beyond imagination or perish utterly may rest on our actions in this century, and yet people who recognize this possibility don’t do everything they can to make it go the way we need it to? This sort of segues in to my second question, which is what is the most any person, more specifically, I can do for FAI? I’m still in high school, so there really isn’t that much keeping me from devoting my life to helping the cause of making sure AI is friendly. What would that look like? I’m a village idiot by LW standards, and especially bad at math, so I don’t think I’d be very useful on the “front lines” so to speak, but perhaps I could try to make a lot of money and do FAI-focused EA? I might be more socially oriented/socially capable than many here, perhaps I could try to raise awareness or lobby for legislation?
Well, ChaosMote already gave part of the answer, but another reason is the idea of comparative advantage. Normally I’d bring up someone like Scott Alexander/Yvain as an example (since he’s repeatedly claimed he’s not good at math and blogs more about politics/general rationality than about AI), but this time, you can just look at yourself. If, as you claim,
then your comparative advantage lies less in theory and more in popularization. Technically, theory might be more important, but if you can net bigger gains elsewhere, then by all means you should do so. To use a (somewhat strained) analogy, think about expected value. Which would you prefer: a guaranteed US $50, or a 10% chance at getting US $300? The raw value of the $300 prize might be greater, but you have to multiply by the probabilities before you can do a comparison. It’s the same here. For some LWers, working on AI is the way to go, but for others who aren’t as good at math, maybe raising money is the best way to do things. And then there’s the even bigger picture: AI might be the most important risk in the end, but what if (say) nuclear war occurs first? A politically-oriented person might do better to go into government or something of the sort, even if that person thinks AI is more important in the long run.
So while it might look somewhat strange that not every LWer is working frantically on AI at first, if you look a little deeper, there’s actually a good reason. (And then there’s also scope insensitivity, hyperbolic discounting, and all that good stuff ChaosMote brought up.) In a sense, you answered your own question when you asked your second.
To address your first question: this has to do with scope insensitivity, hyperbolic discounting, and other related biases. To put it bluntly, most humans are actually pretty bad at maximizing expected utility. For example, when I first head about x-risk, my thought process was definitely not “humanity might be wiped out—that’s IMPORTANT. I need to devote energy to this.” It was more along the lines of “huh; That’s interesting. Tragic, even. Oh well; moving on...”
Basically, we don’t care much about what happens in the distant future, especially if it isn’t guaranteed to happen. We also don’t care much more about humanity than we do about ourselves plus our close ones. Plus we don’t really care about things that don’t feel immediate. And so on. Then end result is that most people’s immediate problems are more important to them then x-risk, even if the latter might be by far the more essential according to utilitarian ethics.
It’s also possible that people might reasonably disagree with one or more of MIRI’s theses.
Like me. Voiciferously.
Is your position written out somewhere where I can read it?
Not in one place, sadly.
Despite all talking about rationality, we are still humans with all typical human flaws. Also, it is not obvious which way it needs to go. Even if we had unlimited and infinitely fast processing power, and could solve mathematically all kinds of problems related to Löb’s theorem, I still would have no idea how we could start transferring human values to the AI, considering that even humans don’t understand themselves, and ideas like “AI should find a way to make humans smile” can lead to horrible outcomes. So maybe the first step would be to upload some humans and give them more processing power, but humans can also be horrible (and the horrible ones are actually more likely to seize such power), and the changes caused by uploading could make even nice people go insane.
So, what is the obvious next step, other than donating some money to the research, which will most likely conclude that further research is needed? I don’t want to discourage anyone who donates or does the research, just saying that the situation with the research is frustrating by its lack of feedback. On the scale where 0 is the first electronic computer and 100 is the Friendly AI, are we at least at point 1? If we happen to be there, how would we know that?
I would like this plan, but there are reasons to think that the path to WBE passes through nueromorphic AI which is exceptionally likely to be unfriendly, since the principle is basically to just copy parts of the human brain without understanding how the human brain works.
I don’t agree with this particular argument, but I’ll mention it anyway for the sake of having a devil’s advocate:
The number of lives lost to an extinction event is arguably capped at ~10 billion, or whatever Earth’s carrying capacity is. If you think the AI risk is enough generations out, then it may well be possible to do more good by, say, eliminating poverty faster. A simple mathematical model would suggest that if the singularity is 10 generations away, and Earth will have a constant population of ~10 billion, then 100 billion lives will pass between now and the singularity. A 10% increase in humanity’s average quality of life over that period would be morally equivalent to stopping the singularity.
Now, there are a host of problems with the above argument:
First, it is trying to minimize death rather than maximize life. If you set out to maximize the number of Quality Adjusted Life Years that intelligent life accumulates before it’s extinction, then you should also take into account all of the potential future lives which would be extinguished by an extinction event, rather than just the lives taken by the event itself.
Second, the Future of Humanity Institute has conducted an informal survey of existential risk researchers, asking for estimates of the probability of human extinction in the next 100 years. The median result (not mean so as to minimize the impact of outliers) was ~19%. If that’s a ~20% chance each century, then we can expect humanity to last perhaps 2 or 3 centuries (aka, that’s the half life of a technological civilization). Even 300 years is only maybe 4 or 5 generations, so perhaps 50 billion lives could be effected by eliminating poverty now. Using the same simplistic model as before, that would require a 20% increase in humanity’s average quality of life to be morally equivalent to ~10 billion deaths. That’s a harder target to hit, but it may be even harder still if you consider that poverty is likely to be nearly eliminated in ~100 years. Poverty really has been going down steadily for the last century or so, and in another century we can expect it to be much improved.
Note that both of these points are based on somewhat subjective judgements. Personally, I think Friendly AI research is around the point of diminishing returns. More money would be useful, of course, but I think it would be putting some focus on preemptively addressing other forms of existential risk which may emerge over the next century. Additionally, I think it’s important to play with other factors that go into QALY. Quality of life is already being addressed, and duration of our civilization is starting to be addressed via x-risk reduction. The other factor is the number of lives, which is currently capped at Earth’s carrying capacity of ~10 billion. I’d like to see trillions of lives. Brain uploads are one method as technology improves, but another is space colonization. The cheapest option I see is Directed Panspermia, which is the intentional seeding of new solar systems with dormant single cell life. All other forms of X-risk reduction address the possibility that the Great Filter is ahead of us, but this would hedge that bet. I haven’t done any calculations yet, but donating to organizations like the Mars Society may even turn out to be competitive in terms of QALY/$, if they can tip the political scale between humanity staying in and around Earth, and humanity starting to spread outward, colonizing other planets and eventually other stars over the next couple millennia. It’s hard to put a figure on the expected QALY return, but if quadrillions of lives hang in the balance, that may well tip the scales and make the tens of billions of dollars needed to initiate Mars colonization an extremely good investment.
To elaborate on existing comments, a fourth alternative to FAI theory, Earning To Give, and popularization is strategy research. (That could include research on other risks besides AI.) I find that the fruit in this area is not merely low-hanging but rotting on the ground. I’ve read in old comment threads that Eliezer and Carl Shulman in particular have done a lot of thinking about strategy but very little of it has been written down, and they are very busy people. Circumstances may well dictate retracing a lot of their steps.
You’ve said elsewhere that you have a low estimate of your innate mathematical ability, which would preclude FAI research, but presumably strategy research would require lower aptitude. Things like statistics would be invaluable, but strategy research would also involve a lot of comparatively less technical work, like historical and philosophical analysis, experiments and surveys, literature reviews, lots and lots of reading, etc. Also, you’ve already done a bit of strategizing; if you are fulfilled by thinking about those things and you think your abilities meet the task, then it might be a good alternative.
Some strategy research resources:
Luke Muehlhauser’s How to study superintelligence strategy.
Luke’s AI Risk and Opportunity: A Strategic Analysis sequence.
Analysis Archives of the MIRI blog.
The AI Impacts blog, particularly the Possible Empirical Investigations post and links therein.
The Future of Life Institute’s A survey of research questions for robust and beneficial AI.
Naturally, Bostrom’s Superintelligence.
Thanks for taking the time to put all that together! I’ll keep it in mind.
First thing you should do is talk to the people that are already involved in this. CFAR seems to be the gateway for man people (at least, it was for me).