I don’t see any discussion about this blog post by Mike Travern.
His point is that people trying to solve for Friendly AI are doing so because it’s an “easy”, abstract problem well into the future. He contends that we are already taking significant damage from artificially created human systems like the financial system, which can be ascribed agency and it’s goals are quite different from improving human life. These systems are quite akin to “Hostile AI”. This, he contends, is the really hard problem.
Here is a quote from the blogpost (which is from a Facebook comment he made):
I am generally on the side of the critics of Singulitarianism, but now want to provide a bit of support to these so-called rationalists. At some very meta level, they have the right problem — how do we preserve human interests in a world of vast forces and systems that aren’t really all that interested in us? But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure.
It’s a short post, so you can read it quickly. What do you think about his argument?
It’s a short post, so you can read it quickly. What do you think about his argument?
I think it’s silly. I suspect MIRI and every other singulatarian organization, and every other individual working on the challeges of unfriendly AI, could fit comfortably in a 100-person auditorium.
In contrast, “trying to fix industrial capitalism” is one of the main topics of political dispute everywhere in the world. “How to make markets work better” is one of the main areas of research in economics. The American Economic Association has 18,000 members. We have half a dozen large government agencies, with budgets of hundreds of millions of dollars each, to protecting people from hostile capitalism. (The SEC, the OCC, the FTC, etc etc, are all ultimately about trying to curb capitalist excess. Each of these organizations has a large enforcement bureaucracy, and also a number of full-time salaried researchers.)
The resources and human energy devoted to unfriendly AI are tiny compared to the amount expended on politics and economics. So it’s strange to complain about the diversion of resources.
Excellent point. I’m surprised this did not occur to me. This reminds me of Scott Aaronson’s reply when someone suggested that quantum computational complexity is quite unimportant compared to experimental approaches to quantum computing and therefore shouldn’t get much funding:
I find your argument extremely persuasive—assuming, of course, that we’re both talking about Bizarro-World, the place where quantum complexity research commands megabillions and is regularly splashed across magazine covers, while Miley Cyrus’s twerking is studied mostly by a few dozen nerds who can all fit in a seminar room at Dagstuhl.
I think it’s silly. I suspect MIRI and every other singulatarian organization, and every other individual working on the challeges of unfriendly AI, could fit comfortably in a 100-person auditorium.
It looks to me like the room in this picture contains more than 100 people.
Yes. I will revise upwards my impression of how many people are working on Singularity topics. That said, not everybody who showed up at the summit was working on singularity-problems. Some were just interested bystanders.
I think that he sounds mind-killed. Calling the financial system a “hostile AI” sounds cool for about half a second until your brain wakes up and goes “Whaaaaat?” :-)
If he really wanted to talk about existing entities with agency and their own interests, well, the notion that the state is one is very very old.
Actually, Mike Travers has a whole sequence of excellent posts on ascribing agency to non-human systems over at Ribbonfarm. See here. I particularly recommend the post Patterns of Refactored Agency.
I don’t think ascribing agency to systems like institutions and collections of institutions is too forced. In fact, institutions seem to exist precisely for preserving and propagating values in the face of changing individuals.
I’m completely fine with ascribing agency to institutions. I’m not fine with sticking in emotionally-loaded terms and implying that e.g. AI researchers should work on fixing the financial system.
But I don’t think that his point is that AI researchers, in general, should be working on fixing the financial system.
I think his point is that the people at MIRI have chosen AI research because they think that AI is a significant source of threat to human well-being/eixstence from non-human value systems (possibly generated by humans). His claim seems to be that AI may only be a very small part of the problem. Instead, there already exist non-human value systems generated by humans threatening human well-being/existence and we don’t know how to fix that.
So, I guess the counter-argument from someone at MIRI would go something like: “while it is true that human institutions can threaten human well-being, no human institution seems to have the power in the near future to threaten human existence. But the technology of self-improving AI, can FOOM and threaten human existence. Thus, we choose to work on preventing this outcome.”
Instead, there already exist non-human value systems generated by humans threatening human well-being/existence and we don’t know how to fix that.
First, I am unaware of evidence (though I am aware of a lot of loud screaming) that human institutions pose an existential risk to humanity. I think the closest we come to that is the capability of US and Russia to launch an all-out nuclear exchange.
Second, the whole “non-human value systems” is much too fuzzy for my liking. Is self-preservation a human value? Let’s take an entity, say a large department within a governmental bureaucracy, the major values of which are self-preservation and the accrual of benefits (of various kinds) to its leadership. Is that a “non-human value system”? Should we call it “hostile AI” and be worried about it?
Third, the global financial system (or the “industrial capitalism”) is not an institution. It’s an ecosystem where many different entities coexist, fight, live, and die. I am not sure ecosystems have agency.
Fourth, it looks to me like his argument would shortcut to either a revolution or more malaria nets.
OK, fine, unfriendly AIs occupy only a small part of the space of possible non-human agents arising from human action and having value systems different from ours and enough power to do a lot of harm as a result; and businesses and nations and so forth are other possible examples.
Furthermore, non-human agents arising [etc.] occupy only a small part of the space of Bad Things.
It doesn’t follow from the latter that people investigating how to arrange for businesses and nations and whatnot to do good rather than harm are making a mistake; and it doesn’t follow from the former that people investigating how to arrange for superhuman AIs (if and when they show up) to do good rather than harm are making a mistake.
Why not? Because in each case the more restricted class of entities has particular features that are (hopefully) amenable to particular kinds of study, and that (we fear) pose particular kinds of threat.
A large and important fraction of AI-space is occupied by entities with the following interesting features. They are created deliberately by human researchers; they operate according to clear and explicit (but perhaps monstrously complex) principles; their behaviour is, accordingly, in principle amenable to quite rigorous (but perhaps intractably difficult) analysis. Businesses and nations and religions and sports clubs don’t have these features, and there’s some hope of developing ways of understanding and/or controlling AIs that don’t apply to those other entities.
It is possible (very likely, according to some) that a large fraction of the probability of a superhuman AI turning up in the nearish future comes from scenarios in which the AI goes from being distinctly subhuman and no threat to anyone, to being vastly superhuman and potentially controlling everything that happens on earth, in so short a time that it’s not feasible for anyone (including businesses, nations, etc.) to stop it. Businesses and nations and religions and sports clubs mostly have [EDIT: oops, I meant “don’t have”] this feature (though you might argue that nuclear war is a bit like a Bad Singularity in some respects), and there might accordingly be a need for much tighter control over potentially superhuman AIs than over businesses and nations and the like.
So one can agree that there are interesting analogies between the danger from unfriendly superhuman AI and the danger from an out-of-control financial system / government / business cartel / religion / whatever, while also thinking that a bunch of people whose interests and expertise lie in the domain of software engineering and pure mathematics might be more effectively used by having them concentrate on AI rather than the financial system.
(There might also be a need for experts in software engineering and pure mathematics to help make the financial system safer—by keeping an eye on the potential for runaway algorithmic trading systems at banks and hedge funds, for instance. But that’s not what Mike Travers is talking about, and actually it’s not a million miles away from friendly AI work—though probably a lot easier because the systems involved are simpler and more limited in power.)
How do you factor in the points made by asr and gjm? In particular,
Much effort is already being spent in dealing with the problems posed by industrial capitalism.
It’s likely that the amount of resources being spent on countering potentially hostile AI (AI as computer programs, not AI as institutions) is less than or equal to the amount justified by that threat.
What ASR said, and also this is a totally different domain. There is no code for the global financial system and no coder is going to fix it. There is no code for AGI, but some coder somewhere IS going to write it. The idea that fighting against billions of people and a system supported by all the money in the world is the same kind of activity as trying to prove theorems of friendliness is simply dense.
There is no code for the global financial system and no coder is going to fix it.
How about simulations of various economic/financial/social systems? Those are being done, and require code, and require high-level analysis. I find it perfectly believable that some abstract theoretical computer science / computer simulation work could uncover new insights / new arguments.
I don’t see any discussion about this blog post by Mike Travern.
His point is that people trying to solve for Friendly AI are doing so because it’s an “easy”, abstract problem well into the future. He contends that we are already taking significant damage from artificially created human systems like the financial system, which can be ascribed agency and it’s goals are quite different from improving human life. These systems are quite akin to “Hostile AI”. This, he contends, is the really hard problem.
Here is a quote from the blogpost (which is from a Facebook comment he made):
It’s a short post, so you can read it quickly. What do you think about his argument?
I think it’s silly. I suspect MIRI and every other singulatarian organization, and every other individual working on the challeges of unfriendly AI, could fit comfortably in a 100-person auditorium.
In contrast, “trying to fix industrial capitalism” is one of the main topics of political dispute everywhere in the world. “How to make markets work better” is one of the main areas of research in economics. The American Economic Association has 18,000 members. We have half a dozen large government agencies, with budgets of hundreds of millions of dollars each, to protecting people from hostile capitalism. (The SEC, the OCC, the FTC, etc etc, are all ultimately about trying to curb capitalist excess. Each of these organizations has a large enforcement bureaucracy, and also a number of full-time salaried researchers.)
The resources and human energy devoted to unfriendly AI are tiny compared to the amount expended on politics and economics. So it’s strange to complain about the diversion of resources.
Excellent point. I’m surprised this did not occur to me. This reminds me of Scott Aaronson’s reply when someone suggested that quantum computational complexity is quite unimportant compared to experimental approaches to quantum computing and therefore shouldn’t get much funding:
It looks to me like the room in this picture contains more than 100 people.
Yes. I will revise upwards my impression of how many people are working on Singularity topics. That said, not everybody who showed up at the summit was working on singularity-problems. Some were just interested bystanders.
Is this the new ‘but what about starving Africans?’
I think that he sounds mind-killed. Calling the financial system a “hostile AI” sounds cool for about half a second until your brain wakes up and goes “Whaaaaat?” :-)
If he really wanted to talk about existing entities with agency and their own interests, well, the notion that the state is one is very very old.
Actually, Mike Travers has a whole sequence of excellent posts on ascribing agency to non-human systems over at Ribbonfarm. See here. I particularly recommend the post Patterns of Refactored Agency.
I don’t think ascribing agency to systems like institutions and collections of institutions is too forced. In fact, institutions seem to exist precisely for preserving and propagating values in the face of changing individuals.
I’m completely fine with ascribing agency to institutions. I’m not fine with sticking in emotionally-loaded terms and implying that e.g. AI researchers should work on fixing the financial system.
But I don’t think that his point is that AI researchers, in general, should be working on fixing the financial system.
I think his point is that the people at MIRI have chosen AI research because they think that AI is a significant source of threat to human well-being/eixstence from non-human value systems (possibly generated by humans). His claim seems to be that AI may only be a very small part of the problem. Instead, there already exist non-human value systems generated by humans threatening human well-being/existence and we don’t know how to fix that.
So, I guess the counter-argument from someone at MIRI would go something like: “while it is true that human institutions can threaten human well-being, no human institution seems to have the power in the near future to threaten human existence. But the technology of self-improving AI, can FOOM and threaten human existence. Thus, we choose to work on preventing this outcome.”
First, I am unaware of evidence (though I am aware of a lot of loud screaming) that human institutions pose an existential risk to humanity. I think the closest we come to that is the capability of US and Russia to launch an all-out nuclear exchange.
Second, the whole “non-human value systems” is much too fuzzy for my liking. Is self-preservation a human value? Let’s take an entity, say a large department within a governmental bureaucracy, the major values of which are self-preservation and the accrual of benefits (of various kinds) to its leadership. Is that a “non-human value system”? Should we call it “hostile AI” and be worried about it?
Third, the global financial system (or the “industrial capitalism”) is not an institution. It’s an ecosystem where many different entities coexist, fight, live, and die. I am not sure ecosystems have agency.
Fourth, it looks to me like his argument would shortcut to either a revolution or more malaria nets.
OK, fine, unfriendly AIs occupy only a small part of the space of possible non-human agents arising from human action and having value systems different from ours and enough power to do a lot of harm as a result; and businesses and nations and so forth are other possible examples.
Furthermore, non-human agents arising [etc.] occupy only a small part of the space of Bad Things.
It doesn’t follow from the latter that people investigating how to arrange for businesses and nations and whatnot to do good rather than harm are making a mistake; and it doesn’t follow from the former that people investigating how to arrange for superhuman AIs (if and when they show up) to do good rather than harm are making a mistake.
Why not? Because in each case the more restricted class of entities has particular features that are (hopefully) amenable to particular kinds of study, and that (we fear) pose particular kinds of threat.
A large and important fraction of AI-space is occupied by entities with the following interesting features. They are created deliberately by human researchers; they operate according to clear and explicit (but perhaps monstrously complex) principles; their behaviour is, accordingly, in principle amenable to quite rigorous (but perhaps intractably difficult) analysis. Businesses and nations and religions and sports clubs don’t have these features, and there’s some hope of developing ways of understanding and/or controlling AIs that don’t apply to those other entities.
It is possible (very likely, according to some) that a large fraction of the probability of a superhuman AI turning up in the nearish future comes from scenarios in which the AI goes from being distinctly subhuman and no threat to anyone, to being vastly superhuman and potentially controlling everything that happens on earth, in so short a time that it’s not feasible for anyone (including businesses, nations, etc.) to stop it. Businesses and nations and religions and sports clubs mostly have [EDIT: oops, I meant “don’t have”] this feature (though you might argue that nuclear war is a bit like a Bad Singularity in some respects), and there might accordingly be a need for much tighter control over potentially superhuman AIs than over businesses and nations and the like.
So one can agree that there are interesting analogies between the danger from unfriendly superhuman AI and the danger from an out-of-control financial system / government / business cartel / religion / whatever, while also thinking that a bunch of people whose interests and expertise lie in the domain of software engineering and pure mathematics might be more effectively used by having them concentrate on AI rather than the financial system.
(There might also be a need for experts in software engineering and pure mathematics to help make the financial system safer—by keeping an eye on the potential for runaway algorithmic trading systems at banks and hedge funds, for instance. But that’s not what Mike Travers is talking about, and actually it’s not a million miles away from friendly AI work—though probably a lot easier because the systems involved are simpler and more limited in power.)
Upvoted, but I think you’re missing a negation in “Businesses and nations and religions and sports clubs mostly have this feature...”.
Yup, I was. Edited. Thanks!
[EDITED to fix an inconsequential thinko.]
Excellent summary. Thanks.
I think it’s spot on.
How do you factor in the points made by asr and gjm? In particular,
Much effort is already being spent in dealing with the problems posed by industrial capitalism.
It’s likely that the amount of resources being spent on countering potentially hostile AI (AI as computer programs, not AI as institutions) is less than or equal to the amount justified by that threat.
What ASR said, and also this is a totally different domain. There is no code for the global financial system and no coder is going to fix it. There is no code for AGI, but some coder somewhere IS going to write it. The idea that fighting against billions of people and a system supported by all the money in the world is the same kind of activity as trying to prove theorems of friendliness is simply dense.
How about simulations of various economic/financial/social systems? Those are being done, and require code, and require high-level analysis. I find it perfectly believable that some abstract theoretical computer science / computer simulation work could uncover new insights / new arguments.
(that being said, I agree with asr’s comment)
They also fail pretty badly and are remarkable useless at the moment.
Sure there is, it’s just that the code is called “laws” and coders are called “legislators”.
Laws don’t behave like code and legislators don’t behave like coders.