I made it clear in our dialogue that I was stipulating a particular definition for intelligence:
SBENTHALL: Would you say that Google is a super-human intelligence?
ME: Well, yeah, so we have to be very careful about all the words that we are using of course. What I mean by intelligence is this notion of what sometimes is called optimization power, which is the ability to achieve one’s goals in a wide range of environments and a wide range of constraints. And so for example, humans have a lot more optimization power than chimpanzees. That’s why even though we are slower than many animals and not as strong as many animals, we have this thing called intelligence that allows us to commence farming and science and build cities and put footprints on the moon. And so it is humans that are steering the future of the globe and not chimpanzees or stronger things like blue whales. So that’s kind of the intuitive notion. There are lots of technical papers that would be more precise.
So, I’m not going to argue about the definition of intelligence. Likewise, I won’t argue about the definition of rationality. What I mean by rationality is the concept of rationality from economics and cognitive science, though if we want to get philosophical then it gets more complicated than simple Bayesianism. (Aside: Is the theory of “communicative rationality” specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)
As for this general line of argument about organizations and intelligence explosion, I refer to the earlier Hanson-Yudkowsky debate, especially (as Emile noted) the UberTool discussion. A summary of the Hanson-Yudkowsky debate is here.
Of course Google is a super-human intelligence (in a sense of optimizing for goals). I agree with gwern et al that probably a company’s productivity scaling is sublinear wrt number of components in it, but that should make it an easier special case to consider. We can still comprehend its goals and mostly what it’s doing. Why not deal with a special case first?
What do you have in mind? Are you proposing a miniature research project into the relevance of companies as superhuman intelligences, and the relevance of those data to the question of whether we should expect a hard takeoff vs. a slow takeoff, or recursively self-improving AI at all? Or are you suggesting something else?
Here is my claim (contrary to Vassar). If you are worried about an unfriendly “foomy” optimizing process, then a natural way to approach that problem is to solve an easier related problem: make an existing unfriendly but “unfoomy” optimizing process friendly. There are lots of such processes of various levels of capability and unfriendliness: North Korea, Microsoft, the United Nations, a non-profit org., etc.
I claim this problem is easier because:
(a) we have a lot more time (no danger of “foom”),
(b) we can use empirical methods (processes already exist), to ground our theories.
(c) these processes are super-humanly intelligent but not so intelligent that their goals/methods are impossible to understand.
The claim is that if we can’t make existing processes with all these simplifying features friendly, we have no hope to make a “foomy” AI friendly.
Ok, so just to make sure I understand your position:
(a) Without friendliness, “foominess” is dangerous.
(b) Friendliness is hard—we can’t use existing academia resources to solve it, as it will take too long. We need a pocket super-intelligent optimizer to solve this problem.
(c) We can’t make partial progress on the friendliness question with existing optimizers.
We can definitely make progress on Friendliness without superintelligent optimizers (see here), but we can’t make some non-foomy process (say, a corporation) Friendly in order to test our theories of Friendliness.
Ok. I am currently diagnosing the source of our disagrement as me being more agnostic about which AI architectures might succeed than you. I am willing to consider the kinds of minds that resemble modern messy non-foomy optimizers (e.g. communities of competing/interacting agents) as promising. That is, “bazaar minds,” not just “cathedral minds.” Given this agnosticism, I see value in “straight science” that worries about arranging possibly stupid/corrupt/evil agents in useful configurations that are not stupid/corrupt/evil.
I think the simplifying features on the other side outweigh those—ie., it’s built from atomic units that do exactly what you tell them to, and there are probably fewer abstraction layers between those atomic units and the goal system. But I do think Mechanism Design is an important field, and will probably form an important part of any friendly optimizing process.
Organisations are likely to build machine intelligence and imbue it with their values. That is reason enough to be concerned with organisation values. One of my proposals to help with this is better corporate repuatation systems.
Those processes are built out of humans, with all the problems that implies. All the transmissions between the humans are lossy. Computers behave much differently. They don’t lie to you, embezzle company funds, or rationalize their poor behavior or ignorance.
This is a very important field of study with some relation, and one I would very much like to pursue. OTOH, it’s not that much like building an AI out of computers. Really, the complexity of building a self-sustaining, efficient, smart, friendly organization out of humans is quite possibly more difficult due to the “out of humans” constraint.
Those processes are built out of humans, with all the problems that implies. All the transmissions between the humans are lossy. Computers behave much differently. They don’t lie to you, embezzle company funds, or rationalize their poor behavior or ignorance.
Doesn’t that rather depend on the values of those who programmed them?
This is a very important field of study with some relation, and one I would very much like to pursue. OTOH, it’s not that much like building an AI out of computers. Really, the complexity of building a self-sustaining, efficient, smart, friendly organization out of humans is quite possibly more difficult due to the “out of humans” constraint.
Organisations tend to construct machine intelligences which reflect their values. However, organisations don’t have an “out of humans” constraint. They are typically a complex symbiosis of humans, culture, artefacts, plants, animals, fungi and bacteria.
Perhaps. But humans will lie, embezzle, and rationalize regardless of who programmed them. Besides, would the internals of a computer lie to itself? Does RAM lie to a processor? And yet humans (being the subcomponents of an organization) routinely lie to each other. No system of rules I can devise will guarantee that doesn’t happen without some very serious side effects.
All of which are subject to the humans’ interpretation and use. You can set up an organizational culture, but that won’t stop the humans from mucking it up, as they routinely do in organizations across the globe. You can write process documents, but that doesn’t mean they’ll even follow them at all. If you specify a great deal of process, they may not even do so intentionally—they may just forget. With a computer, that would be caused by an error, but it’s a controllable process. With a human? People can’t just decide to remember arbitrary amounts of arbitrary information for arbitrary lengths of time and pull it off reliably.
So; on the one hand, I have a system being built where the underlying hardware is reliable and under my control, and generally does not create errors or disobey. On the other hand, I have a network of unreliable and forgetful intelligences that may be highly irrational and may even be working at cross purposes with each other or the organization itself. One requires extremely strict instructions, the other is capable of interpretation and judgment from context without specifying an algorithm in great detail. There are similarities between the two, but there are also great practical differences.
As you will see by things like my Angelic Foundations essay, I do appreciate the virtues of working with machines.
However, at the moment, there are also advantages to a man-machine symbiosis—namely robotics is still far behind the evolved molecular nanotechnology in animals in many respects—and computers still lag far behind brains in many critical areas. A man-machine symbiosis will thus beat machines in many areas, until after machines reach the level of a typical human in most work-related physical and mental feats. Machine-only solutions will just lose. So: we will be working with organisations for a while yet—during a pretty important period in history.
I just think it’s a related but different field. Actually, solving these problems is something I want to apply some AI to (more accurate mapping of human behavior allowing massive batch testing of different forms of organization given outside pressures—discover possible failure modes and approaches to deal with them), but that’s a different conversation.
(Aside: Is the theory of “communicative rationality” specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)
Not yet. It’s a qualitatively described theory. I think it’s probably possible to render it into quantitative terms, but as far as I know it has not yet been done.
I believe I’m familiar with how you use the term rationality. I believe it’s compatible with (mutually reinforcing with) communicative rationality for the most part, though I believe there are some differences between Habermas’ and Yudkowsky’s epistemologies. I brought up communicative rationality because (a) I think it’s an important concept that is in some ways an advance in how to think about rationality and, (b) I wanted to disclose some of my own predispositions and values for the sake of establishing expectations.
Thanks for the link to the Hanson-Yudkowsky debate. From perusing the summary and a few of the posts by the debaters, I guess I’d say I find Hanson’s counterarguments largely compelling. I’d also respond with two other points (mostly hoping you will direct me to where they’ve already been discussed):
Since the computational complexity of so many kinds of problems has been proven to be within certain complexity classes, recursive improvement in algorithms alone is likely to hit asymptotic walls for a lot of interesting domains. So, self-modifying AI alone, without taking resources into account, seems unlikely (maybe provably impossible) to be a big threat.
That said, since there already are self-modifying intelligent organizations that are taking over the world (or trying to, facing competition from each other), what’s gone into Singularity research definitely isn’t useless. Rather, it’s directly applicable to what’s happening right now.
I agree very strongly with the thrust of what IlyaShpitser’s been saying.
If it is provably impossible, I would feel much better with a proof; this seems like a reasonable goal for SingIst, to look at proofs of computational complexity and upper limits on computer power, and get an upper limit on the optimization power of an AI (perhaps a few estimates conditional on some problems being in different categories or new best algorithms being found); then to come up with some reasonable way of measuring lower and upper bounds on the optimization power of various organizations (at least a generous upper bound on all existing organizations and a lower bound on some big ones like the US government).
I would be EXTREMELY surprised to find that a lower bound on organizations was higher than the upper bound on AI, but if so it would be good to know already, and if not the research would probably be worth doing anyway and a good showcase of the actual extent of the problem.
I made it clear in our dialogue that I was stipulating a particular definition for intelligence:
So, I’m not going to argue about the definition of intelligence. Likewise, I won’t argue about the definition of rationality. What I mean by rationality is the concept of rationality from economics and cognitive science, though if we want to get philosophical then it gets more complicated than simple Bayesianism. (Aside: Is the theory of “communicative rationality” specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)
As for this general line of argument about organizations and intelligence explosion, I refer to the earlier Hanson-Yudkowsky debate, especially (as Emile noted) the UberTool discussion. A summary of the Hanson-Yudkowsky debate is here.
I also refer interested parties to the comments here by gwern and by anonymous1.
Of course Google is a super-human intelligence (in a sense of optimizing for goals). I agree with gwern et al that probably a company’s productivity scaling is sublinear wrt number of components in it, but that should make it an easier special case to consider. We can still comprehend its goals and mostly what it’s doing. Why not deal with a special case first?
What do you have in mind? Are you proposing a miniature research project into the relevance of companies as superhuman intelligences, and the relevance of those data to the question of whether we should expect a hard takeoff vs. a slow takeoff, or recursively self-improving AI at all? Or are you suggesting something else?
Here is my claim (contrary to Vassar). If you are worried about an unfriendly “foomy” optimizing process, then a natural way to approach that problem is to solve an easier related problem: make an existing unfriendly but “unfoomy” optimizing process friendly. There are lots of such processes of various levels of capability and unfriendliness: North Korea, Microsoft, the United Nations, a non-profit org., etc.
I claim this problem is easier because:
(a) we have a lot more time (no danger of “foom”),
(b) we can use empirical methods (processes already exist), to ground our theories.
(c) these processes are super-humanly intelligent but not so intelligent that their goals/methods are impossible to understand.
The claim is that if we can’t make existing processes with all these simplifying features friendly, we have no hope to make a “foomy” AI friendly.
I don’t know what this would mean, since figuring out friendliness probably requires superintelligence, hence CEV as an initial dynamic.
Ok, so just to make sure I understand your position:
(a) Without friendliness, “foominess” is dangerous.
(b) Friendliness is hard—we can’t use existing academia resources to solve it, as it will take too long. We need a pocket super-intelligent optimizer to solve this problem.
(c) We can’t make partial progress on the friendliness question with existing optimizers.
Is this fair?
“Yes” to (a), “no” to (b) and (c).
We can definitely make progress on Friendliness without superintelligent optimizers (see here), but we can’t make some non-foomy process (say, a corporation) Friendly in order to test our theories of Friendliness.
Ok. I am currently diagnosing the source of our disagrement as me being more agnostic about which AI architectures might succeed than you. I am willing to consider the kinds of minds that resemble modern messy non-foomy optimizers (e.g. communities of competing/interacting agents) as promising. That is, “bazaar minds,” not just “cathedral minds.” Given this agnosticism, I see value in “straight science” that worries about arranging possibly stupid/corrupt/evil agents in useful configurations that are not stupid/corrupt/evil.
I think the simplifying features on the other side outweigh those—ie., it’s built from atomic units that do exactly what you tell them to, and there are probably fewer abstraction layers between those atomic units and the goal system. But I do think Mechanism Design is an important field, and will probably form an important part of any friendly optimizing process.
Organisations are likely to build machine intelligence and imbue it with their values. That is reason enough to be concerned with organisation values. One of my proposals to help with this is better corporate repuatation systems.
Those processes are built out of humans, with all the problems that implies. All the transmissions between the humans are lossy. Computers behave much differently. They don’t lie to you, embezzle company funds, or rationalize their poor behavior or ignorance.
This is a very important field of study with some relation, and one I would very much like to pursue. OTOH, it’s not that much like building an AI out of computers. Really, the complexity of building a self-sustaining, efficient, smart, friendly organization out of humans is quite possibly more difficult due to the “out of humans” constraint.
Doesn’t that rather depend on the values of those who programmed them?
Organisations tend to construct machine intelligences which reflect their values. However, organisations don’t have an “out of humans” constraint. They are typically a complex symbiosis of humans, culture, artefacts, plants, animals, fungi and bacteria.
Perhaps. But humans will lie, embezzle, and rationalize regardless of who programmed them. Besides, would the internals of a computer lie to itself? Does RAM lie to a processor? And yet humans (being the subcomponents of an organization) routinely lie to each other. No system of rules I can devise will guarantee that doesn’t happen without some very serious side effects.
All of which are subject to the humans’ interpretation and use. You can set up an organizational culture, but that won’t stop the humans from mucking it up, as they routinely do in organizations across the globe. You can write process documents, but that doesn’t mean they’ll even follow them at all. If you specify a great deal of process, they may not even do so intentionally—they may just forget. With a computer, that would be caused by an error, but it’s a controllable process. With a human? People can’t just decide to remember arbitrary amounts of arbitrary information for arbitrary lengths of time and pull it off reliably.
So; on the one hand, I have a system being built where the underlying hardware is reliable and under my control, and generally does not create errors or disobey. On the other hand, I have a network of unreliable and forgetful intelligences that may be highly irrational and may even be working at cross purposes with each other or the organization itself. One requires extremely strict instructions, the other is capable of interpretation and judgment from context without specifying an algorithm in great detail. There are similarities between the two, but there are also great practical differences.
As you will see by things like my Angelic Foundations essay, I do appreciate the virtues of working with machines.
However, at the moment, there are also advantages to a man-machine symbiosis—namely robotics is still far behind the evolved molecular nanotechnology in animals in many respects—and computers still lag far behind brains in many critical areas. A man-machine symbiosis will thus beat machines in many areas, until after machines reach the level of a typical human in most work-related physical and mental feats. Machine-only solutions will just lose. So: we will be working with organisations for a while yet—during a pretty important period in history.
I just think it’s a related but different field. Actually, solving these problems is something I want to apply some AI to (more accurate mapping of human behavior allowing massive batch testing of different forms of organization given outside pressures—discover possible failure modes and approaches to deal with them), but that’s a different conversation.
I’ve realized I didn’t address your direct query:
Not yet. It’s a qualitatively described theory. I think it’s probably possible to render it into quantitative terms, but as far as I know it has not yet been done.
Thanks for this response, Luke.
I don’t want to argue about definitions either.
I believe I’m familiar with how you use the term rationality. I believe it’s compatible with (mutually reinforcing with) communicative rationality for the most part, though I believe there are some differences between Habermas’ and Yudkowsky’s epistemologies. I brought up communicative rationality because (a) I think it’s an important concept that is in some ways an advance in how to think about rationality and, (b) I wanted to disclose some of my own predispositions and values for the sake of establishing expectations.
Thanks for the link to the Hanson-Yudkowsky debate. From perusing the summary and a few of the posts by the debaters, I guess I’d say I find Hanson’s counterarguments largely compelling. I’d also respond with two other points (mostly hoping you will direct me to where they’ve already been discussed):
Since the computational complexity of so many kinds of problems has been proven to be within certain complexity classes, recursive improvement in algorithms alone is likely to hit asymptotic walls for a lot of interesting domains. So, self-modifying AI alone, without taking resources into account, seems unlikely (maybe provably impossible) to be a big threat.
That said, since there already are self-modifying intelligent organizations that are taking over the world (or trying to, facing competition from each other), what’s gone into Singularity research definitely isn’t useless. Rather, it’s directly applicable to what’s happening right now.
I agree very strongly with the thrust of what IlyaShpitser’s been saying.
If it is provably impossible, I would feel much better with a proof; this seems like a reasonable goal for SingIst, to look at proofs of computational complexity and upper limits on computer power, and get an upper limit on the optimization power of an AI (perhaps a few estimates conditional on some problems being in different categories or new best algorithms being found); then to come up with some reasonable way of measuring lower and upper bounds on the optimization power of various organizations (at least a generous upper bound on all existing organizations and a lower bound on some big ones like the US government).
I would be EXTREMELY surprised to find that a lower bound on organizations was higher than the upper bound on AI, but if so it would be good to know already, and if not the research would probably be worth doing anyway and a good showcase of the actual extent of the problem.