I don’t understand? A super intelligence will of course not act as programed, nor as intended because to be the “super intelligent“ definition it will have emergent properties.
Plus “… instrumental incentives to manipulate or deceive its operators, and the system should not resist operator correction or shut- down.” Don’t act like any well adjusted 2 year old? If we really want intelligence running around, we are going to have to learn to let go of control.
Humans can barely value what’s in their scope. Intelligence can only value what’s in their scope because they are not “over there” and really can only follow best practices which might or might not work, and likely won’t work with outliers. We simply can’t take action “for the good of all humanity”, because we don’t know what’s good for everyone. We like to think we do, but we don’t. People used to think binding women’s feet was a good idea. Additionally, even if another takes our advice, only they experience the consequences of their actions: The feedback loop is broken: bureaucracy. This seems to be a persistent issue with AI. It is mathematically unsolvable: a local scope cannot know what’s best for a non local scope (without invoking omniscience. In practical terms, this is why projection of power is so expensive, why empires always fail, and why nature does not have empires.
There is a simple fix, but it requires scary thinking. Evolution obviously has intelligence: It made everything we are and experience. So just copy it. Like any other complex adaptive system it has a few simple initial conditions. https://www.castpoints.com/
If done correctly, we don’t get Skynet, we get another subset of evolution evolving.
Humans don’t understand intelligence. We, and computers, are not that intelligent. We mostly express evolution’s intelligence. That’s why people want to get into “flow” states.
These questions and objections are touched upon in many parts of the sequences (the series of blog posts which seeded LessWrong, and which were written to address questions like this specifically). In your case, I’d recommend reading almost all of those posts, as they were targeted precisely towards these sorts of objections. That’s a lot of reading, though; if you want specific answers to the questions you posed, then it sounds like you may be interested in the evolution mini sequence (which responds to the claim all we can hope to do is “express evolution’s intelligence”; see also thou art godshatter), and probably also the mysterious answers to mysterious questions sequence (which talks about ways to approach topics that you don’t understand; see The Futility of Emergence in particular), and also maybe the metaethics sequence and the fun theory sequence which give some reasons to expect that “another subset of evolution evolving” is not such a good outcome.
I don’t understand? A super intelligence will of course not act as programed, nor as intended because to be the “super intelligent“ definition it will have emergent properties.
Plus “… instrumental incentives to manipulate or deceive its operators, and the system should not resist operator correction or shut- down.” Don’t act like any well adjusted 2 year old? If we really want intelligence running around, we are going to have to learn to let go of control.
Humans can barely value what’s in their scope. Intelligence can only value what’s in their scope because they are not “over there” and really can only follow best practices which might or might not work, and likely won’t work with outliers. We simply can’t take action “for the good of all humanity”, because we don’t know what’s good for everyone. We like to think we do, but we don’t. People used to think binding women’s feet was a good idea. Additionally, even if another takes our advice, only they experience the consequences of their actions: The feedback loop is broken: bureaucracy. This seems to be a persistent issue with AI. It is mathematically unsolvable: a local scope cannot know what’s best for a non local scope (without invoking omniscience. In practical terms, this is why projection of power is so expensive, why empires always fail, and why nature does not have empires.
There is a simple fix, but it requires scary thinking. Evolution obviously has intelligence: It made everything we are and experience. So just copy it. Like any other complex adaptive system it has a few simple initial conditions. https://www.castpoints.com/
If done correctly, we don’t get Skynet, we get another subset of evolution evolving.
Humans don’t understand intelligence. We, and computers, are not that intelligent. We mostly express evolution’s intelligence. That’s why people want to get into “flow” states.
These questions and objections are touched upon in many parts of the sequences (the series of blog posts which seeded LessWrong, and which were written to address questions like this specifically). In your case, I’d recommend reading almost all of those posts, as they were targeted precisely towards these sorts of objections. That’s a lot of reading, though; if you want specific answers to the questions you posed, then it sounds like you may be interested in the evolution mini sequence (which responds to the claim all we can hope to do is “express evolution’s intelligence”; see also thou art godshatter), and probably also the mysterious answers to mysterious questions sequence (which talks about ways to approach topics that you don’t understand; see The Futility of Emergence in particular), and also maybe the metaethics sequence and the fun theory sequence which give some reasons to expect that “another subset of evolution evolving” is not such a good outcome.