Q5:How important is it to research risks associated with artificial intelligence that is good enough at general reasoning (including science, mathematics, engineering and programming) to be capable of radical self-modification, before attempting to build one?
I doubt that researchers will know what you have in mind by “provably friendly.” For that matter I myself don’t know what you have in mind by “probably friendly” despite having read a number of relevant posts on Less Wrong.
That doesn’t matter too much. Their interpretation of the question is the interesting part and that they are introduced to the concept is the important part. All of them will be able to make sense of the idea of AI that is non-dangerous.
And besides, after they answered the questions I asked them about “friendly AI” and a lot of them are familiar with the idea of making AI’s non-dangerous, some even heard about SI. Here is one reply I got:
I was surprised that none of your questions mentioned paperclips! :P I am (of course!) familiar with lesswrong.com, goals of the SIAI, prominence of EY’s work within these communities etc.
And regarding your other objection,
the researchers in question do not appear to be familiar with the most serious existential risk from AGI: the one discussed in Omohundro’s The Basic AI Drives.
Many seem to think that AI’s won’t have any drives and that it is actually part of the problem to give them the incentive to do anything that they haven’t been explicitly programmed to do. But maybe I am the wrong person with respect to Omohundro’s paper. I find the arguments in it unconvincing. And if I think that way, someone who takes risks from AI much more seriously, then I doubt that even if they knew about the paper it would change their opinion.
The problem is that the paper tries to base its arguments on assumptions about the nature of hypothetical AGI. But just like AI won’t automatically share our complex values it won’t value self-protection/improvement or rational economic behavior. In short, the paper is tautological. If your premise is an expected utility-maximizer capable of undergoing explosive recursive self-improvement that tries to take every goal to its logical extreme, whether that is part of the specifications or not, then you already answered your own question and arguing about drives becomes completely useless.
You should talk to wallowinmaya who is soon going to start his own interview series.
If your premise is an expected utility-maximizer capable of undergoing explosive recursive self-improvement that tries to take every goal to its logical extreme, whether that is part of the specifications or not, then you already answered your own question and arguing about drives becomes completely useless.
I like your point, but I wonder what doubts you have about the premise. Is an expected-utility-maximizer likely to be absurdly difficult to construct, or do you think all or almost all AI designers would prefer other designs? I think that AI designers would prefer such a design if they could have it, and “maximize my company’s profits” is likely to be the design objective.
I think that most researchers are not interested in fully autonomous AI (AI with persistent goals and a “self”) and more interested in human-augmented intelligence (meaning tools like data-mining software).
I do think that an expected utility-maximizer is the ideal in GAI. But, just like general purpose quantum computers, I believe that expected utility-maximizer’s that − 1) find it instrumentally useful to undergo recursive self-improvement 2) find it instrumentally useful to take over the planet/universe to protect their goals—are, if at all feasible, the end-product of a long chain of previous AI designs with no quantum leaps in-between. That they are at all feasible is dependent on 1) how far from the human level intelligence hits diminishing returns 2) that intelligence is more useful than other kinds of resources in stumbling upon unknown unknowns in solution space 3) that expected utility-maximizer’s and their drives are not fundamentally dependent on the precision with which their utility-function is defined.
I further believe that long before we get to the point of discovering how to build expected utility-maximizer’s, capable of undergoing explosive recursive self-improvement, we will have automatic scientists that can brute-force discoveries on hard problem in bio and nanotech and enable unfriendly humans to wreck havoc and control large groups of people. If we survive that, which I think is the top risk rather than GAI, then we might at some point be able to come up with an universal artificial intelligence. (ETA: Note that for automatic scientists to work well the goals need to be well-defined, which isn’t the case for intelligence amplification.)
I just don’t have enough background knowledge to conclude that it is likely that humans can stumble upon simple algorithms that could be improved to self-improve and then reach vastly superhuman capabilities. From my point of view that seems like pure speculation, although speculation that should be taken seriously and that does legitimate the existence of an organisation like SI. Which is the reason why I have donated a few times already. But from my uneducated point of view it seems unreasonable to claim that the possibility is obviously correct and that the arguments might not simply sound convincing.
I approach this problem the same way that I approach climate change. Just because one smart person believes that climate change is bunk I don’t believe it as well. All his achievements do not legitimate his views. And since I am yet too uneducated and do not have the time to evaluate all the data and calculations I am using the absurdity heuristic in combination with an appeal to authority to conclude that climate change is real. And the same goes for risks from AI. I can hardly evaluate universal AI research or understand approximation’s to AIXI. But if the very people who came up with it disagree on various points with those who say that their research poses a risk, then I side with the experts but still assign enough weight to the other side to conclude that they are doing important work nonetheless.
Just because one smart person believes that climate change is bunk I don’t believe it as well. All his achievements do not legitimate his views.
“Climate change is bunk” seems like a pretty terrible summary of Freeman Dyson’s position. If you disagree, a more specific criticism would be helpful. Freeman Dyson’s views on the topic mostly seem to be sensible to me.
And since I am yet too uneducated and do not have the time to evaluate all the data and calculations I am using the absurdity heuristic in combination with an appeal to authority to conclude that climate change is real.
Freeman Dyson agrees. The very first line from your reference reads: “Dyson agrees that anthropogenic global warming exists”.
I think that most researchers are not interested in fully autonomous AI (AI with persistent goals and a “self”) and more interested in human-augmented intelligence (meaning tools like data-mining software).
Intelligence augmentation can pay your bills today.
You didn’t address my criticism of the question about provably friendly AI nor my point about the researchers lacking relevant context for thinking about AI risk. Again, the issues that I point to seems to make the researchers’ response to the questions about friendliness & existential risk due to AI carry little information
I rephrased the question now:
Q5: How important is it to research risks associated with artificial intelligence that is good enough at general reasoning (including science, mathematics, engineering and programming) to be capable of radical self-modification, before attempting to build one?
Good. I think that’s a much less mind-killing question, and will get more interesting responses.
That doesn’t matter too much. Their interpretation of the question is the interesting part and that they are introduced to the concept is the important part. All of them will be able to make sense of the idea of AI that is non-dangerous.
And besides, after they answered the questions I asked them about “friendly AI” and a lot of them are familiar with the idea of making AI’s non-dangerous, some even heard about SI. Here is one reply I got:
And regarding your other objection,
Many seem to think that AI’s won’t have any drives and that it is actually part of the problem to give them the incentive to do anything that they haven’t been explicitly programmed to do. But maybe I am the wrong person with respect to Omohundro’s paper. I find the arguments in it unconvincing. And if I think that way, someone who takes risks from AI much more seriously, then I doubt that even if they knew about the paper it would change their opinion.
The problem is that the paper tries to base its arguments on assumptions about the nature of hypothetical AGI. But just like AI won’t automatically share our complex values it won’t value self-protection/improvement or rational economic behavior. In short, the paper is tautological. If your premise is an expected utility-maximizer capable of undergoing explosive recursive self-improvement that tries to take every goal to its logical extreme, whether that is part of the specifications or not, then you already answered your own question and arguing about drives becomes completely useless.
You should talk to wallowinmaya who is soon going to start his own interview series.
I like your point, but I wonder what doubts you have about the premise. Is an expected-utility-maximizer likely to be absurdly difficult to construct, or do you think all or almost all AI designers would prefer other designs? I think that AI designers would prefer such a design if they could have it, and “maximize my company’s profits” is likely to be the design objective.
I think that most researchers are not interested in fully autonomous AI (AI with persistent goals and a “self”) and more interested in human-augmented intelligence (meaning tools like data-mining software).
I do think that an expected utility-maximizer is the ideal in GAI. But, just like general purpose quantum computers, I believe that expected utility-maximizer’s that − 1) find it instrumentally useful to undergo recursive self-improvement 2) find it instrumentally useful to take over the planet/universe to protect their goals—are, if at all feasible, the end-product of a long chain of previous AI designs with no quantum leaps in-between. That they are at all feasible is dependent on 1) how far from the human level intelligence hits diminishing returns 2) that intelligence is more useful than other kinds of resources in stumbling upon unknown unknowns in solution space 3) that expected utility-maximizer’s and their drives are not fundamentally dependent on the precision with which their utility-function is defined.
I further believe that long before we get to the point of discovering how to build expected utility-maximizer’s, capable of undergoing explosive recursive self-improvement, we will have automatic scientists that can brute-force discoveries on hard problem in bio and nanotech and enable unfriendly humans to wreck havoc and control large groups of people. If we survive that, which I think is the top risk rather than GAI, then we might at some point be able to come up with an universal artificial intelligence. (ETA: Note that for automatic scientists to work well the goals need to be well-defined, which isn’t the case for intelligence amplification.)
I just don’t have enough background knowledge to conclude that it is likely that humans can stumble upon simple algorithms that could be improved to self-improve and then reach vastly superhuman capabilities. From my point of view that seems like pure speculation, although speculation that should be taken seriously and that does legitimate the existence of an organisation like SI. Which is the reason why I have donated a few times already. But from my uneducated point of view it seems unreasonable to claim that the possibility is obviously correct and that the arguments might not simply sound convincing.
I approach this problem the same way that I approach climate change. Just because one smart person believes that climate change is bunk I don’t believe it as well. All his achievements do not legitimate his views. And since I am yet too uneducated and do not have the time to evaluate all the data and calculations I am using the absurdity heuristic in combination with an appeal to authority to conclude that climate change is real. And the same goes for risks from AI. I can hardly evaluate universal AI research or understand approximation’s to AIXI. But if the very people who came up with it disagree on various points with those who say that their research poses a risk, then I side with the experts but still assign enough weight to the other side to conclude that they are doing important work nonetheless.
“Climate change is bunk” seems like a pretty terrible summary of Freeman Dyson’s position. If you disagree, a more specific criticism would be helpful. Freeman Dyson’s views on the topic mostly seem to be sensible to me.
Freeman Dyson agrees. The very first line from your reference reads: “Dyson agrees that anthropogenic global warming exists”.
Intelligence augmentation can pay your bills today.