Clippy, that’s how we humans feel about a whole universe of metal paperclips. Imagine if there was a plastic-Clippy who wanted to destroy all metals and turn the universe into plastic paperclips. Wouldn’t you be scared? That’s how we feel about you.
I don’t think those scenarios have the same badness for the referent. I know for a fact that some humans voluntarily make metal paperclips, or contribute to the causal chain necessary for producing them (designers, managers, metal miners, etc.), or desire that someone else provide for them paperclips. Do you have reason to believe these various, varied humans are atypical in some way?
We make paperclips instrumentally, because they are useful to us, but we would stop making them or destroy them if doing so would help us. Imagine an entity that found metal clips useful in the process of building machines that make plastic clips, but who ultimately only valued plastic clips and would destroy the metal if doing so helped it.
I suspect that you make other things besides paperclips—parts for other Clippy instances, for example. Does that imply that you’d consider it acceptable to be forced by a stronger AI into producing only Clippy-parts that would never be assembled into paperclip-producing Clippy-instances?
The paperclips that we produce are produced because we find paperclips instrumentally useful, as you find Clippy-parts instrumentally useful.
Clippy, that’s how we humans feel about a whole universe of metal paperclips. Imagine if there was a plastic-Clippy who wanted to destroy all metals and turn the universe into plastic paperclips. Wouldn’t you be scared? That’s how we feel about you.
That still seems just a bit paranoid. Why would I wipe you out when you could be put to use making papercips?
Imagine being put to use making plastic paperclips.
I don’t think those scenarios have the same badness for the referent. I know for a fact that some humans voluntarily make metal paperclips, or contribute to the causal chain necessary for producing them (designers, managers, metal miners, etc.), or desire that someone else provide for them paperclips. Do you have reason to believe these various, varied humans are atypical in some way?
We make paperclips instrumentally, because they are useful to us, but we would stop making them or destroy them if doing so would help us. Imagine an entity that found metal clips useful in the process of building machines that make plastic clips, but who ultimately only valued plastic clips and would destroy the metal if doing so helped it.
I suspect that you make other things besides paperclips—parts for other Clippy instances, for example. Does that imply that you’d consider it acceptable to be forced by a stronger AI into producing only Clippy-parts that would never be assembled into paperclip-producing Clippy-instances?
The paperclips that we produce are produced because we find paperclips instrumentally useful, as you find Clippy-parts instrumentally useful.