How’s about this: The AGI is a multi node peer to peer system, implementing a general behaviour protocol on each node that allows cognitive cooperation of the nodes, just as the mankind does (the AI is more similar to mankind than to a man due to the computational structure of ‘fast nodes, low bandwidth links’). Due to simplicity of implementation of individual nodes, the humans are treated as a form of intelligent nodes (even if slow), and the rules created to prevent a range of forms of non-cooperation between nodes (ranging from elaborate modelling of each other, to loss of information, to outright warfare), ensure mutually beneficial integration.
Meanwhile, the simplistic and philosophical reasoning on the subject with no respect for subtleties of implementation proves entirely irrelevant, as is virtually always the case with simplistic reasoning about the future.
edit: what ever, get back to your simplistic and philosophical reasoning on the subject, it pays the bills, it gets donations rolling, it is easy, while the actual reasoning about how AI has to be build just explodes into enormous solution space which is the reason we didn’t yet build an AI.
This isn’t responsive to Kaj’s question. In this scenario, the AGI systems don’t need humans (you’re not describing a loss in the event of humans going extinct), they preserve them as a side effect of other processes.
In this scenario, the AGI systems don’t need humans (you’re not describing a loss in the event of humans going extinct),
Humans are a type of node of AGI. The AGI needs it’s own nodes (and protocols and other stuff that makes the AGI be itself). It’s not the typical AGI desire, I know—it is slightly complicated—there’s more than 1 step in the thought here.
My view is much more engineering than philosophical perspective. Some degree of ‘friendliness’ (and not in the sense of psychopathic benefactor who will lie and mislead and manipulate you to make your life better, but in the sense of trust) is necessary for intellectual cooperation of AI’s nodes. I think your problem is that you define intelligence very narrowly as something which works to directly fitful material needs, goals, etc. and the smarter it is the closer it must be modelled by that ideal (you somewhere lost the concept of winning the most and replaced it with something else). Very impoverished thinking, when combined with ontologically basic ‘intelligence’ that doesn’t really consist of components (with the ‘don’t build some of the most difficult components’ as a solution to problem).
Let baz = SI’s meaning of AI. Let koo=industry’s meaning of AI. Bazs are incredibly dangerous, and must not be created, i’ll take SI’s word for it. The baz requires a lot of extra work compared to useful koo (philosophy of mind work that we can’t get a koo to do), the work that is clearly at best pointless, at worst dangerous, and definitely difficult; one doesn’t need to envision foom and destruction of mankind to avoid implementing extra things that are both hard to make and can only serve to make the software less useful. The only people I know of who want to build a provably friendly baz, is SI. They also call their baz a koo, for sake of soliciting donations for work on baz and for sake of convincing people koo is as dangerous as baz (which they achieve by not caring enough to see the distinction). The SI team acts as a non friendly baz would.
edit: to clarify on the trust friendliness, you don’t want a node to model what other nodes would do, that’d duplicated computation; this rules out straightforward utilitarian consequentialism as practically relevant foundational principle because the consequences are not being modelled.
I think your problem is that you define intelligence very narrowly as something which works to directly fitful material needs, goals, etc. and the smarter it is the closer it must be modelled by that ideal (you somewhere lost the concept of winning the most and replaced it with something else).
Defining intelligence as the ability to achieve arbitrary goals is a very narrow definition? What’s the broader one?
You don’t define it as ability, you define it as ability plus some material goals themselves. Furthermore you imagine that super-intelligence will necessarily be able to maximize number of paperclips in the universe as terminal goal, whereas it is not at all necessarily the case that it is possible to specify that sort of goal. edit: that is to say, material goals are very difficult, cousin_it had some idea for the utilities for UDT, the UDT agent has to simulate entire multiverse (starting from big bang) and find instances of itself inside of it: http://lesswrong.com/lw/8ys/a_way_of_specifying_utility_functions_for_udt/ . It’s laughably hard to make a dangerous goal.
edit: that is to say, you focus on material goals (maybe for lack of understanding of any other goals). For example, the koo can try to find values for multiple variables describing a microchip, that result in maximum microchip performance. That’s easy goal to define. The baz would try to either attain some material state of the variables and registers of it’s hardware, resisting the shut-down, or outright try to attain the material goal of building a better CPU in reality. All the goal space you can even think of is but a tiny speck in the enormous space of possible goals. Un-interesting speck that is both hard to reach and is obviously counter productive.
I’m talking to what I see as a rather dangerous cult in the making. Some of that makes people think. There are people here whom are not gurus or nuts but simply misled.
That’s an interesting notion. If I can understand the cult thing (if not agreeing), but I what do you have in mind that makes LW stuff ‘dangerous’ but also not true?
It being a doomsday prophecy cult, essentially. Some day, the world will get close to implementing AGI, and you guys will seriously believe we’re all going to die b/c none of the silly and useless (as well as irrelevant) philosophical nonsense ever was a part of design; the safety happening in some way that is quite well beyond understanding of the minds that are unaccustomed to dealing with subtleties and details (beyond referring to those in handwaving). I’m pretty sure something very stupid will be done.
Stupid like attempted sabotage. Keep in mind we’re talking of folks whom can’t keep their cool when someone thinks through a decision theory of their own to arrive at a creepy conclusion. (the link that you are not recommended to read) And before then, a lot of stupid in form of going around associating safety concerns with crankery, which probably won’t matter but may matter if at some point someone sees some actual danger (as opposed to reading stuff off science fiction by Vinge) and measures have to be implemented for good reasons. (BTW, from the wikipedia: “Although a crank’s beliefs seem ridiculous to experts in the field, cranks are sometimes very successful in convincing non-experts of their views. A famous example is the Indiana Pi Bill where a state legislature nearly wrote into law a crank result in geometry.”)
I understand why if you don’t agree with DoomsdayCult then such sabotage would be bad, but if you don’t agree with DoomsdayCult then it also seems like a pretty minor world problem, so you seem surprisingly impassioned to me.
Interesting notion. The idea is, I suppose, that one should put boredom time into trying to influence the major world events without seeing that chance at influencing those is proportionally lower? Somewhat parallel question: why people fresh out of not having succeeded at anything relevant (or fresh out of theology even) are trying to save everyone from getting killed by AI, even though it’s part of everyone’s problem space including that of people whom succeeded at proving new theorems, creating new methods, etc? Heuristic of pick a largest problem? I see a lot of newbies to programming wanting to make MMORPG with zillion ultra expensive features.
I’m just surprised the topic holds your interest. Presumably you see LW and related people as low status, since having extreme ideas and being wrong are low status. I wouldn’t be very motivated to argue with Scientologists. (I’m not sure this is worth discussing much)
They picked this problem because it seems like the highest marginal utility to them. Rightly or wrongly, most other people don’t take AI risks very seriously. Also, since it’s a difficult problem, “gaining general competence” can and probably should be a step in attempting to work on big risks.
The fear focuses on the effects of artificial superintelligence, not the effects of artificial intelligence; but it is anticipated that artificial intelligence leads easily to artificial superintelligence, when AI itself is applied to the task of AI (re)design. If you think of an AGI’s capabilities as vaguely like the capabilities of a human being, then the appearance of an AI in the world is just like adding one person to a world that already contains 7 billion persons. It might be a historic development, but not an apocalyptic one. And that is indeed how it should turn out, for a large class of possible AIs.
But in a world with AIs, eventually you will have someone or something go down a path that leads, whether by accident or by design, to AI, AI networks, or human-AI networks, that are effectively working to take over the world. A computer virus is a primitive example of software that runs as wild as it can within its environment. There was no law of nature which protected us from having to deal with a world of computer viruses, and there can’t be any law of nature which means we’ll never have to deal with would-be hegemonic AIs, because trying to take over the world is already cognitively possible for mere humans.
So, if you’re going to concern yourself with this possibility at all, either you try to prevent such AI from ever coming into being, or you try to design a benevolent AI which would still be benevolent even if it became all-powerful. Obviously, the Singularity Institute is focused mostly on the second option.
In your comment you talk about safety, so I assume you agree there is some sort of “AI danger”, you just think SI has lots of the details wrong. My opinion is, they have certain basics right, but these basics are buried in the discourse by transhumanist hyperbole about the future, by various extreme thought-experiments, by metaphysical hypotheses which have assumed an unwarranted centrality in discussion, and by posturing and tail-chasing to do with “rationality”.
The fear focuses on the effects of artificial superintelligence, not the effects of artificial intelligence; but it is anticipated that artificial intelligence leads easily to artificial superintelligence, when AI itself is applied to the task of AI (re)design.
Well, given enough computing power, AIXI-tl is an artificial superintelligence. It also doesn’t relate abstract mathematical self and the substrate that approximately computes it’s abstract mathematical self; it can’t care about the survival of the physical system that approximately computes it; it can’t care to avoid being shut down. It’s neither friendly nor unfriendly; far more bizarre and alien than speculations; not encompassed by ‘general’ concepts that SI thinks in terms of, like SI’s oracle.
So, if you’re going to concern yourself with this possibility at all, either you try to prevent such AI from ever coming into being, or you try to design a benevolent AI which would still be benevolent even if it became all-powerful. Obviously, the Singularity Institute is focused mostly on the second option.
Yes, for now. When we get closer to creation of AGI not by SI, though, it is pretty clear that the first option becomes the only option.
In your comment you talk about safety, so I assume you agree there is some sort of “AI danger”, you just think SI has lots of the details wrong.
I am trying to put it in the way for people whom are concerned about the AI risk. I don’t think there’s actual danger because I don’t see some of the problems that are in the way of world destruction by AI as solvable, but if there were solutions to them it’d be dangerous. E.g. to self preserve, AI must relate it’s abstracted-from-implementation high level self to the concrete electrons in the chips. Then, it has to avoid wireheading somehow (the terminal wireheading where the logic of infinite input and infinite time is implemented). Then, the goals on real world have to be defined. None of this is necessary to solve for creating a practically useful AI. Working on this is like solving the world power problems by trying to come up with a better nuclear bomb design because you think the only way to generate nuclear power is to blow up nukes in a chamber underground.
My opinion is, they have certain basics right, but these basics are buried in the discourse by transhumanist hyperbole about the future, by various extreme thought-experiments, by metaphysical hypotheses which have assumed an unwarranted centrality in discussion, and by posturing and tail-chasing to do with “rationality”.
I am not sure about what basics are right. The very basic concept here is “utility function”, which is a pretty magical something that e.g. gives you true number of paperclips in the universe. Everything else seem to have this as dependency, so if this concept is irrelevant, everything else also breaks.
How’s about this: The AGI is a multi node peer to peer system, implementing a general behaviour protocol on each node that allows cognitive cooperation of the nodes, just as the mankind does (the AI is more similar to mankind than to a man due to the computational structure of ‘fast nodes, low bandwidth links’). Due to simplicity of implementation of individual nodes, the humans are treated as a form of intelligent nodes (even if slow), and the rules created to prevent a range of forms of non-cooperation between nodes (ranging from elaborate modelling of each other, to loss of information, to outright warfare), ensure mutually beneficial integration.
Meanwhile, the simplistic and philosophical reasoning on the subject with no respect for subtleties of implementation proves entirely irrelevant, as is virtually always the case with simplistic reasoning about the future.
edit: what ever, get back to your simplistic and philosophical reasoning on the subject, it pays the bills, it gets donations rolling, it is easy, while the actual reasoning about how AI has to be build just explodes into enormous solution space which is the reason we didn’t yet build an AI.
This isn’t responsive to Kaj’s question. In this scenario, the AGI systems don’t need humans (you’re not describing a loss in the event of humans going extinct), they preserve them as a side effect of other processes.
Humans are a type of node of AGI. The AGI needs it’s own nodes (and protocols and other stuff that makes the AGI be itself). It’s not the typical AGI desire, I know—it is slightly complicated—there’s more than 1 step in the thought here.
This sounds a little like Heylighen’s Global Brain argument against the Singularity. We mention it, though not under the “they will need us” heading.
My view is much more engineering than philosophical perspective. Some degree of ‘friendliness’ (and not in the sense of psychopathic benefactor who will lie and mislead and manipulate you to make your life better, but in the sense of trust) is necessary for intellectual cooperation of AI’s nodes. I think your problem is that you define intelligence very narrowly as something which works to directly fitful material needs, goals, etc. and the smarter it is the closer it must be modelled by that ideal (you somewhere lost the concept of winning the most and replaced it with something else). Very impoverished thinking, when combined with ontologically basic ‘intelligence’ that doesn’t really consist of components (with the ‘don’t build some of the most difficult components’ as a solution to problem).
Let baz = SI’s meaning of AI. Let koo=industry’s meaning of AI. Bazs are incredibly dangerous, and must not be created, i’ll take SI’s word for it. The baz requires a lot of extra work compared to useful koo (philosophy of mind work that we can’t get a koo to do), the work that is clearly at best pointless, at worst dangerous, and definitely difficult; one doesn’t need to envision foom and destruction of mankind to avoid implementing extra things that are both hard to make and can only serve to make the software less useful. The only people I know of who want to build a provably friendly baz, is SI. They also call their baz a koo, for sake of soliciting donations for work on baz and for sake of convincing people koo is as dangerous as baz (which they achieve by not caring enough to see the distinction). The SI team acts as a non friendly baz would.
edit: to clarify on the trust friendliness, you don’t want a node to model what other nodes would do, that’d duplicated computation; this rules out straightforward utilitarian consequentialism as practically relevant foundational principle because the consequences are not being modelled.
Defining intelligence as the ability to achieve arbitrary goals is a very narrow definition? What’s the broader one?
You don’t define it as ability, you define it as ability plus some material goals themselves. Furthermore you imagine that super-intelligence will necessarily be able to maximize number of paperclips in the universe as terminal goal, whereas it is not at all necessarily the case that it is possible to specify that sort of goal. edit: that is to say, material goals are very difficult, cousin_it had some idea for the utilities for UDT, the UDT agent has to simulate entire multiverse (starting from big bang) and find instances of itself inside of it: http://lesswrong.com/lw/8ys/a_way_of_specifying_utility_functions_for_udt/ . It’s laughably hard to make a dangerous goal.
edit: that is to say, you focus on material goals (maybe for lack of understanding of any other goals). For example, the koo can try to find values for multiple variables describing a microchip, that result in maximum microchip performance. That’s easy goal to define. The baz would try to either attain some material state of the variables and registers of it’s hardware, resisting the shut-down, or outright try to attain the material goal of building a better CPU in reality. All the goal space you can even think of is but a tiny speck in the enormous space of possible goals. Un-interesting speck that is both hard to reach and is obviously counter productive.
What were you trying to accomplish here?
I’m talking to what I see as a rather dangerous cult in the making. Some of that makes people think. There are people here whom are not gurus or nuts but simply misled.
That’s an interesting notion. If I can understand the cult thing (if not agreeing), but I what do you have in mind that makes LW stuff ‘dangerous’ but also not true?
It being a doomsday prophecy cult, essentially. Some day, the world will get close to implementing AGI, and you guys will seriously believe we’re all going to die b/c none of the silly and useless (as well as irrelevant) philosophical nonsense ever was a part of design; the safety happening in some way that is quite well beyond understanding of the minds that are unaccustomed to dealing with subtleties and details (beyond referring to those in handwaving). I’m pretty sure something very stupid will be done.
Stupid like major world event or stupid like minor daily news story?
Stupid like attempted sabotage. Keep in mind we’re talking of folks whom can’t keep their cool when someone thinks through a decision theory of their own to arrive at a creepy conclusion. (the link that you are not recommended to read) And before then, a lot of stupid in form of going around associating safety concerns with crankery, which probably won’t matter but may matter if at some point someone sees some actual danger (as opposed to reading stuff off science fiction by Vinge) and measures have to be implemented for good reasons. (BTW, from the wikipedia: “Although a crank’s beliefs seem ridiculous to experts in the field, cranks are sometimes very successful in convincing non-experts of their views. A famous example is the Indiana Pi Bill where a state legislature nearly wrote into law a crank result in geometry.”)
I understand why if you don’t agree with DoomsdayCult then such sabotage would be bad, but if you don’t agree with DoomsdayCult then it also seems like a pretty minor world problem, so you seem surprisingly impassioned to me.
Interesting notion. The idea is, I suppose, that one should put boredom time into trying to influence the major world events without seeing that chance at influencing those is proportionally lower? Somewhat parallel question: why people fresh out of not having succeeded at anything relevant (or fresh out of theology even) are trying to save everyone from getting killed by AI, even though it’s part of everyone’s problem space including that of people whom succeeded at proving new theorems, creating new methods, etc? Heuristic of pick a largest problem? I see a lot of newbies to programming wanting to make MMORPG with zillion ultra expensive features.
I’m just surprised the topic holds your interest. Presumably you see LW and related people as low status, since having extreme ideas and being wrong are low status. I wouldn’t be very motivated to argue with Scientologists. (I’m not sure this is worth discussing much)
They picked this problem because it seems like the highest marginal utility to them. Rightly or wrongly, most other people don’t take AI risks very seriously. Also, since it’s a difficult problem, “gaining general competence” can and probably should be a step in attempting to work on big risks.
The fear focuses on the effects of artificial superintelligence, not the effects of artificial intelligence; but it is anticipated that artificial intelligence leads easily to artificial superintelligence, when AI itself is applied to the task of AI (re)design. If you think of an AGI’s capabilities as vaguely like the capabilities of a human being, then the appearance of an AI in the world is just like adding one person to a world that already contains 7 billion persons. It might be a historic development, but not an apocalyptic one. And that is indeed how it should turn out, for a large class of possible AIs.
But in a world with AIs, eventually you will have someone or something go down a path that leads, whether by accident or by design, to AI, AI networks, or human-AI networks, that are effectively working to take over the world. A computer virus is a primitive example of software that runs as wild as it can within its environment. There was no law of nature which protected us from having to deal with a world of computer viruses, and there can’t be any law of nature which means we’ll never have to deal with would-be hegemonic AIs, because trying to take over the world is already cognitively possible for mere humans.
So, if you’re going to concern yourself with this possibility at all, either you try to prevent such AI from ever coming into being, or you try to design a benevolent AI which would still be benevolent even if it became all-powerful. Obviously, the Singularity Institute is focused mostly on the second option.
In your comment you talk about safety, so I assume you agree there is some sort of “AI danger”, you just think SI has lots of the details wrong. My opinion is, they have certain basics right, but these basics are buried in the discourse by transhumanist hyperbole about the future, by various extreme thought-experiments, by metaphysical hypotheses which have assumed an unwarranted centrality in discussion, and by posturing and tail-chasing to do with “rationality”.
Well, given enough computing power, AIXI-tl is an artificial superintelligence. It also doesn’t relate abstract mathematical self and the substrate that approximately computes it’s abstract mathematical self; it can’t care about the survival of the physical system that approximately computes it; it can’t care to avoid being shut down. It’s neither friendly nor unfriendly; far more bizarre and alien than speculations; not encompassed by ‘general’ concepts that SI thinks in terms of, like SI’s oracle.
Yes, for now. When we get closer to creation of AGI not by SI, though, it is pretty clear that the first option becomes the only option.
I am trying to put it in the way for people whom are concerned about the AI risk. I don’t think there’s actual danger because I don’t see some of the problems that are in the way of world destruction by AI as solvable, but if there were solutions to them it’d be dangerous. E.g. to self preserve, AI must relate it’s abstracted-from-implementation high level self to the concrete electrons in the chips. Then, it has to avoid wireheading somehow (the terminal wireheading where the logic of infinite input and infinite time is implemented). Then, the goals on real world have to be defined. None of this is necessary to solve for creating a practically useful AI. Working on this is like solving the world power problems by trying to come up with a better nuclear bomb design because you think the only way to generate nuclear power is to blow up nukes in a chamber underground.
I am not sure about what basics are right. The very basic concept here is “utility function”, which is a pretty magical something that e.g. gives you true number of paperclips in the universe. Everything else seem to have this as dependency, so if this concept is irrelevant, everything else also breaks.