I think all of these explanations hold part of the puzzle, but that the most fundamental explanation is that the mistake began as soon as we started calling it a “blue-minimizing robot”.
Your post reminded me of a quote by Eliezer Yudkowsky:
An electron is not a billiard ball, and it’s not a crest and trough moving through a pool of water. An electron is a mathematically different sort of entity, all the time and under all circumstances, and it has to be accepted on its own terms.
The universe is not wavering between using particles and waves, unable to make up its mind. It’s only human intuitions about QM that swap back and forth. The intuitions we have for billiard balls, and the intuitions we have for crests and troughs in a pool of water, both look sort of like they’re applicable to electrons, at different times and under different circumstances. But the truth is that both intuitions simply aren’t applicable.
Humans are not selfish or altruistic, humans probably don’t even have a utility-function because our goals are not stable. Our goals are rigidly coupled with the environment and circumstances we reside in. Words are too simplistic to describe what we are and what we want.
humans probably don’t even have a utility-function because our goals are not stable
Even what a human is is not stable.
My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car?
The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells?
Your post reminded me of a quote by Eliezer Yudkowsky:
Humans are not selfish or altruistic, humans probably don’t even have a utility-function because our goals are not stable. Our goals are rigidly coupled with the environment and circumstances we reside in. Words are too simplistic to describe what we are and what we want.
Even what a human is is not stable.
My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car?
The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells?
I am not sure.