For example, in the practice of software development, there often does not seem to be an operational distinction between “intelligent” and “Friendly.” (For example, my impression is that the only method programmers had for evaluating Watson’s “intelligence” was to see whether it was coming up with the same answers that a well-informed human would; the only way to evaluate Siri’s “intelligence” was to evaluate its helpfulness to humans.) “Intelligent” often ends up getting defined as “prone to take actions that seem all-around ‘good’ to the programmer.” So the concept of “Friendliness” may end up being naturally and subtly baked in to a successful AGI effort.
Well, yes this is the definition of Friendliness in the most tautological sense. Siri and Watson are both very domain-specific AIs, so evaluating their “intelligence” or “Friendliness” is relatively trivial—you just have to see if their outputs match the small subset of the programmer’s utility function that corresponds to what the programmer designed them to do. With AGI, you have to get the Friendliness right across all dimensions of human value (see: Value is Fragile), precisely because it’s as cross-domain as an AI can possibly be.
Siri and Watson are both very domain-specific AIs, so evaluating their “intelligence” or “Friendliness” is relatively trivial—you just have to see if their outputs match the small subset of the programmer’s utility function that corresponds to what the programmer designed them to do.
Assume you were to gradually transform Google Maps into a seed AI, at what point would it become an existential risk and how? And why wouldn’t you just skip that step?
Assume you were to gradually transform Google Maps into a seed AI, at what point would it become an existential risk and how?
If it tries to self-improve, and as a side effect turns the universe to computronium.
If it gains a general intelligence, and as a part of trying to provide better search results, it realizes that self-modification could bring much faster search results.
This whole idea of a harmless general intelligence is just imagining a general intelligence which is not general enough to be dangerous; which will be able to think generally, and yet somehow this ability will always reliably stop before thinking something that might end bad.
Assume you were to gradually transform Google Maps into a seed AI, at what point would it become an existential risk and how? And why wouldn’t you just skip that step?
A very important part of Google Maps is Street View, which is created by cars driving around and taking pictures of everything. These could be viewed as ‘arms’ of the seed AI, along with its surveillance satellites, WiFi sniffing for more accurate geolocation, 3D modelling of buildings, and the recently introduced building-interior maps.
Which is to say, Super Google Maps could be a gigantic surveillance network and pervasive examiner of every corner of reality so it could be as up to date as possible.
How does one do a gradual transformation on a discontinuous space such as the space of computer programs that are somehow related to navigation or general intelligence?
Well, yes this is the definition of Friendliness in the most tautological sense. Siri and Watson are both very domain-specific AIs, so evaluating their “intelligence” or “Friendliness” is relatively trivial—you just have to see if their outputs match the small subset of the programmer’s utility function that corresponds to what the programmer designed them to do. With AGI, you have to get the Friendliness right across all dimensions of human value (see: Value is Fragile), precisely because it’s as cross-domain as an AI can possibly be.
Assume you were to gradually transform Google Maps into a seed AI, at what point would it become an existential risk and how? And why wouldn’t you just skip that step?
More here.
If it tries to self-improve, and as a side effect turns the universe to computronium.
If it gains a general intelligence, and as a part of trying to provide better search results, it realizes that self-modification could bring much faster search results.
This whole idea of a harmless general intelligence is just imagining a general intelligence which is not general enough to be dangerous; which will be able to think generally, and yet somehow this ability will always reliably stop before thinking something that might end bad.
Thanks, I completely missed that. Explains a lot.
A very important part of Google Maps is Street View, which is created by cars driving around and taking pictures of everything. These could be viewed as ‘arms’ of the seed AI, along with its surveillance satellites, WiFi sniffing for more accurate geolocation, 3D modelling of buildings, and the recently introduced building-interior maps.
Which is to say, Super Google Maps could be a gigantic surveillance network and pervasive examiner of every corner of reality so it could be as up to date as possible.
That reminds me of Project Pigeon, only with a weapon capable of destroying the planet, and we’re the pigeon.
How does one do a gradual transformation on a discontinuous space such as the space of computer programs that are somehow related to navigation or general intelligence?