That seems obviously wrong to me. There’s nothing at all preventing me from designing an invisible-pink-unicorn maximizer, even if invisible pink unicorns are impossible. For that matter, if we allow counterfactuals, an invisible-pink-unicorn maximizer still looks like an intelligence designed to maximize unicorns—in the counterfactual universe where unicorns exist, the intelligence takes actions that tend to maximize unicorns.
How would you empirically distinguish between your invisible-pink-unicorn maximizer and something that wasn’t an invisible-pink-unicorn maximizer? I mean, you could look for a section of code that was interpreting sensory inputs as number of invisible-pink-unicorns—except you couldn’t, because there’s no set of sensory inputs that corresponds to that, because they’re impossible. If we’re talking about counterfactuals, the counterfactual universe in which the sensory inputs that currently correspond to paperclips correspond to invisible-pink-unicorns seems just as valid as any other.
Well, there’s certainly a set of sensory inputs that corresponds to /invisible-unicorn/, based on which one could build an invisible unicorn detector. Similarly, there’s a set of sensory inputs that corresponds to /pink-unicorn/, based on which one could build a pink unicorn detector.
If I wire a pink unicorn detector up to an invisible unicorn detector such that a light goes on iff both detectors fire on the same object, have I not just constructed an invisible-pink-unicorn detector?
Granted, a detector is not the same thing as a maximizer, but the conceptual issue seems identical in both cases.
If I wire a pink unicorn detector up to an invisible unicorn detector such that a light goes on iff both detectors fire on the same object, have I not just constructed an invisible-pink-unicorn detector?
Maybe. Or maybe you’ve constructed a square-circle detector; no experiment would let you tell the difference, no?
I think the way around this is some notion of which kind of counterfactuals are valid and which aren’t. I’ve seen posts here (and need to read more) about evaluating these counterfactuals via surgery on causal graphs. But while I can see how such reasoning would work an object that exists in a different possible world (i.e. a “contingently nonexistent” object) I don’t (yet?) see how to apply it to a logically impossible (“necessarily nonexistent”) object. Is there a good notion available that can say one counterfactuals involving such things is more valid than another?
Or maybe you’ve constructed a square-circle detector; no experiment would let you tell the difference, no?
Take the thing apart and test its components in isolation. If in isolation they test for squares and circles, their aggregate is a square-circle detector (which never fires). If in isolation they test for pink unicorns and invisible unicorns, their aggregate is an invisible-pink-unicorn detector (which never fires).
except you couldn’t, because there’s no set of sensory inputs that corresponds to that, because they’re impossible.
That does not follow. I’ll admit my original example is mildly flawed, but let’s tack on something (that’s still impossible) to illustrate my point: invisible pink telekinetic unicorns. Still not a thing that can exist, if you define telekinesis as “action at a distance, not mediated through one of the four fundamental forces.” But now, if you see an object stably floating in vacuum, and detect no gravitational or electromagnetic anomalies (and you’re in an accelerated reference frame like the surface of the earth, etc etc), you can infer the presence of an invisible telekinetic something.
Or in general—an impossible object will have an impossible set of sensory inputs, but the set of corresponding sensory inputs still exists.
That seems obviously wrong to me. There’s nothing at all preventing me from designing an invisible-pink-unicorn maximizer, even if invisible pink unicorns are impossible. For that matter, if we allow counterfactuals, an invisible-pink-unicorn maximizer still looks like an intelligence designed to maximize unicorns—in the counterfactual universe where unicorns exist, the intelligence takes actions that tend to maximize unicorns.
How would you empirically distinguish between your invisible-pink-unicorn maximizer and something that wasn’t an invisible-pink-unicorn maximizer? I mean, you could look for a section of code that was interpreting sensory inputs as number of invisible-pink-unicorns—except you couldn’t, because there’s no set of sensory inputs that corresponds to that, because they’re impossible. If we’re talking about counterfactuals, the counterfactual universe in which the sensory inputs that currently correspond to paperclips correspond to invisible-pink-unicorns seems just as valid as any other.
Well, there’s certainly a set of sensory inputs that corresponds to /invisible-unicorn/, based on which one could build an invisible unicorn detector. Similarly, there’s a set of sensory inputs that corresponds to /pink-unicorn/, based on which one could build a pink unicorn detector.
If I wire a pink unicorn detector up to an invisible unicorn detector such that a light goes on iff both detectors fire on the same object, have I not just constructed an invisible-pink-unicorn detector?
Granted, a detector is not the same thing as a maximizer, but the conceptual issue seems identical in both cases.
Maybe. Or maybe you’ve constructed a square-circle detector; no experiment would let you tell the difference, no?
I think the way around this is some notion of which kind of counterfactuals are valid and which aren’t. I’ve seen posts here (and need to read more) about evaluating these counterfactuals via surgery on causal graphs. But while I can see how such reasoning would work an object that exists in a different possible world (i.e. a “contingently nonexistent” object) I don’t (yet?) see how to apply it to a logically impossible (“necessarily nonexistent”) object. Is there a good notion available that can say one counterfactuals involving such things is more valid than another?
Take the thing apart and test its components in isolation. If in isolation they test for squares and circles, their aggregate is a square-circle detector (which never fires). If in isolation they test for pink unicorns and invisible unicorns, their aggregate is an invisible-pink-unicorn detector (which never fires).
That does not follow. I’ll admit my original example is mildly flawed, but let’s tack on something (that’s still impossible) to illustrate my point: invisible pink telekinetic unicorns. Still not a thing that can exist, if you define telekinesis as “action at a distance, not mediated through one of the four fundamental forces.” But now, if you see an object stably floating in vacuum, and detect no gravitational or electromagnetic anomalies (and you’re in an accelerated reference frame like the surface of the earth, etc etc), you can infer the presence of an invisible telekinetic something.
Or in general—an impossible object will have an impossible set of sensory inputs, but the set of corresponding sensory inputs still exists.
Yeah, spooky action at a distance :-) Nowadays we usually call it “quantum entanglement” :-D
… I’m pretty sure no arrangement of entangled particles will create an object that just hovers a half-foot above the Earth’s surface.