While I agree with the main thrust of what you’re saying, I think the scenario you paint is not the same thing that Friendly AI (capitalized) is concerned with. Those seem like good reasons to think about Machine Ethics, for example. FAI is supposed to solve a specific sort of problem, which your assumptions seem to rule out.
I don’t see a difference of purpose between “Friendly AI” and “machine ethics”. Friendly AI is basically a school of thought about machine ethics—by far the best that I’ve seen, when it comes to depth of analysis, methodology, and ingenuity of concepts—that sprang up under unusual circumstances.
In terms of LW concepts, FAI tends to get grouped with “futurism and speculation”, and then that topic cluster is separated from “pure and applied rationality”. The perspective I want to foster sees more continuity between FAI and the study of human decision-making, and places a little more distance between FAI and the futurism/speculation cluster.
The concept of a hard takeoff plays a central role in how people think about FAI—perhaps this is the “specific sort of problem” you mean—but it can be motivated as a worst case that needs to be considered for safety’s sake, rather than as a prediction or dogma about what will actually happen.
The difference between FAI and Machine Ethics is the difference betwen “how do I make this robot not blow me up” and “how do I make this eldrich abomination not torture all of humanity for eternity”.
The difference between FAI and Machine Ethics is the difference betwen “how do I make this robot not blow me up” and “how do I make this eldrich abomination not torture all of humanity for eternity”.
I thought another significant difference was that “Ethics” doesn’t even imply getting as far as “How do I?”. An Ethical discussion could center around “Would a given scenario be considered torture and is this better or worse than extinction?”
I thought another significant difference was that “Ethics” doesn’t even imply getting as far as “How do I?”.
No, Machine Ethics is the field concerned with exactly the question of how to program ethical machines. For example, Arkin’s Governing Lethal Behavior in Autonomous Robots is a work in the field of Machine Ethics.
In principle, a philosopher could try to work in Machine Ethics and only do speculative work on, for example, whether it’s good to have robots that like torture. But inasmuch as that’s a real question, it’s relevant to the practical project.
I was under the impression that Machine Ethics is mostly being researched by Computer Science experts and AI or Neuro-something specialists in particular.
My prior for someone already doing research in IT concentrating their research efforts into “How do I code something that does X” is much higher than for someone doing research in, say, propagation of memes in animal populations or intergalactic lensing distortions (Dark Matter! *shivers*).
I was under the impression that Machine Ethics is mostly being researched by Computer Science experts and AI or Neuro-something specialists in particular.
Pretty much a mix of people who know about machines and people who know about ethics. Arkin is a roboticist. Anderson and Anderson are a philosopher/computer scientist duo. Colin Allen is a philosopher, and I believe Wendell Wallach is too.
I’d argue that the best work is done by computing/robotics folks, yes.
That is only a superficial difference, a difference of scenario considered. If you put a bad actor from ordinary machine-ethics into a possible world where you can torture someone forever, or if you put a UFAI into a possible world where the most harm it can do is blow you up once, this difference goes away.
Designing an “ethical computer program” or a “friendly AI” is not about which possible world the program inhabits, it’s about the internal causality of the program and the choices it makes. The valuable parts of FAI research culture are all on this level. Associating FAI with the possible world of “post-singularity hell”, as if that is the essence of what distinguishes the approach, is an example of what I want to combat in this post.
Designing an “ethical computer program” or a “friendly AI” is not about which possible world the program inhabits, it’s about the internal causality of the program and the choices it makes.
The key difference is that in the case of a Seed AI, you need to find a way to make a goal system stable under recursive self-improvement. In the case of a toaster, you do not.
It’s useful to keep Friendly AI concerns in mind when designing ethical robots, since they potentially become a risk when they start to get more autonomous. But when you’re giving a robot a gun, the relevant ethical concerns are things like whether it will shoot civilians. The scope is relevantly different.
Really, there is a whole field out there of Machine Ethics, and it’s pretty well established that it’s up to a different sort of thing than what SIAI is doing. While some folks still conflate “Friendly AI” and “Machine Ethics”, I think it’s much better to maintain the distinction and consider FAI a subfield of Machine Ethics.
While I agree with the main thrust of what you’re saying, I think the scenario you paint is not the same thing that Friendly AI (capitalized) is concerned with. Those seem like good reasons to think about Machine Ethics, for example. FAI is supposed to solve a specific sort of problem, which your assumptions seem to rule out.
I don’t see a difference of purpose between “Friendly AI” and “machine ethics”. Friendly AI is basically a school of thought about machine ethics—by far the best that I’ve seen, when it comes to depth of analysis, methodology, and ingenuity of concepts—that sprang up under unusual circumstances.
In terms of LW concepts, FAI tends to get grouped with “futurism and speculation”, and then that topic cluster is separated from “pure and applied rationality”. The perspective I want to foster sees more continuity between FAI and the study of human decision-making, and places a little more distance between FAI and the futurism/speculation cluster.
The concept of a hard takeoff plays a central role in how people think about FAI—perhaps this is the “specific sort of problem” you mean—but it can be motivated as a worst case that needs to be considered for safety’s sake, rather than as a prediction or dogma about what will actually happen.
The difference between FAI and Machine Ethics is the difference betwen “how do I make this robot not blow me up” and “how do I make this eldrich abomination not torture all of humanity for eternity”.
I thought another significant difference was that “Ethics” doesn’t even imply getting as far as “How do I?”. An Ethical discussion could center around “Would a given scenario be considered torture and is this better or worse than extinction?”
No, Machine Ethics is the field concerned with exactly the question of how to program ethical machines. For example, Arkin’s Governing Lethal Behavior in Autonomous Robots is a work in the field of Machine Ethics.
In principle, a philosopher could try to work in Machine Ethics and only do speculative work on, for example, whether it’s good to have robots that like torture. But inasmuch as that’s a real question, it’s relevant to the practical project.
I was under the impression that Machine Ethics is mostly being researched by Computer Science experts and AI or Neuro-something specialists in particular.
My prior for someone already doing research in IT concentrating their research efforts into “How do I code something that does X” is much higher than for someone doing research in, say, propagation of memes in animal populations or intergalactic lensing distortions (Dark Matter! *shivers*).
Pretty much a mix of people who know about machines and people who know about ethics. Arkin is a roboticist. Anderson and Anderson are a philosopher/computer scientist duo. Colin Allen is a philosopher, and I believe Wendell Wallach is too.
I’d argue that the best work is done by computing/robotics folks, yes.
Yes, that pretty well captures it.
That is only a superficial difference, a difference of scenario considered. If you put a bad actor from ordinary machine-ethics into a possible world where you can torture someone forever, or if you put a UFAI into a possible world where the most harm it can do is blow you up once, this difference goes away.
Designing an “ethical computer program” or a “friendly AI” is not about which possible world the program inhabits, it’s about the internal causality of the program and the choices it makes. The valuable parts of FAI research culture are all on this level. Associating FAI with the possible world of “post-singularity hell”, as if that is the essence of what distinguishes the approach, is an example of what I want to combat in this post.
The key difference is that in the case of a Seed AI, you need to find a way to make a goal system stable under recursive self-improvement. In the case of a toaster, you do not.
It’s useful to keep Friendly AI concerns in mind when designing ethical robots, since they potentially become a risk when they start to get more autonomous. But when you’re giving a robot a gun, the relevant ethical concerns are things like whether it will shoot civilians. The scope is relevantly different.
Really, there is a whole field out there of Machine Ethics, and it’s pretty well established that it’s up to a different sort of thing than what SIAI is doing. While some folks still conflate “Friendly AI” and “Machine Ethics”, I think it’s much better to maintain the distinction and consider FAI a subfield of Machine Ethics.