Without overfitting, the robot has the goal of shooting at what it sees blue. It achieves its goal. What I get from the article is that the human intelligence mis interprets the goal. Here I see the definition of a goal to equal what the program is written to do, hence it seems inevitable that the robot wll achieve its goal (if there is a bug in the code that misses shooting a blue object every 10 days, then this should be considered part of the goal as well, since we are forced to define the goal in hindsight, if we have to define one)
You’re right. Man, I can’t believe I’ve been wasting my time persuading people to sign up for cryonics, because if they wanted to live, they would have already done so! I can’t believe I didn’t realize this before.
I can’t think of another way to reason—does our brain dictate our goal, or receives a goal from somewhere and makes an effort to execute it accurately? I’d go with the first option, which to me means that whatever our brain (code) is built to do is our goal.
The complication in the case of humans might be the fact that we have more than one competing goal. It is as if this robot has a multi-tasking operating system, with one process trying to kill blue objects and another trying to build a pyramid out of plastic bottles. Normally they can co-exist somehow with some switching between processes or by just one process “not caring” about doing some activity at the current instance.
It gets ugly when the robot finds a few blue bottles. Then the robot becomes “irrational” with one process destroying what the other is trying to do. This is simply when you are on a healthy diet and see a slice of chocolate cake—you’re processes are doing their jobs, but they are competing on resources—who gets to move your arms?
Let’s then imagine that we have in our brains a controlling (operating) system that can get to decide which process to kill when they are in conflict. Will this operating system have a right and wrong decision? Or will whatever it does be the right thing according to its code—or else it wouldn’t have done it?
I was thinking of, y’know, biased people. Also known as me and everyone I’ve ever met. Telling them “don’t worry, whatever you’re already doing is what you really want” does not seem Friendly.
Without overfitting, the robot has the goal of shooting at what it sees blue. It achieves its goal. What I get from the article is that the human intelligence mis interprets the goal. Here I see the definition of a goal to equal what the program is written to do, hence it seems inevitable that the robot wll achieve its goal (if there is a bug in the code that misses shooting a blue object every 10 days, then this should be considered part of the goal as well, since we are forced to define the goal in hindsight, if we have to define one)
Do you reason similarly for humans?
It is almost proverbial that intentions are better revealed by deeds than by words.
You’re right. Man, I can’t believe I’ve been wasting my time persuading people to sign up for cryonics, because if they wanted to live, they would have already done so! I can’t believe I didn’t realize this before.
I can’t think of another way to reason—does our brain dictate our goal, or receives a goal from somewhere and makes an effort to execute it accurately? I’d go with the first option, which to me means that whatever our brain (code) is built to do is our goal.
The complication in the case of humans might be the fact that we have more than one competing goal. It is as if this robot has a multi-tasking operating system, with one process trying to kill blue objects and another trying to build a pyramid out of plastic bottles. Normally they can co-exist somehow with some switching between processes or by just one process “not caring” about doing some activity at the current instance.
It gets ugly when the robot finds a few blue bottles. Then the robot becomes “irrational” with one process destroying what the other is trying to do. This is simply when you are on a healthy diet and see a slice of chocolate cake—you’re processes are doing their jobs, but they are competing on resources—who gets to move your arms?
Let’s then imagine that we have in our brains a controlling (operating) system that can get to decide which process to kill when they are in conflict. Will this operating system have a right and wrong decision? Or will whatever it does be the right thing according to its code—or else it wouldn’t have done it?
I was thinking of, y’know, biased people. Also known as me and everyone I’ve ever met. Telling them “don’t worry, whatever you’re already doing is what you really want” does not seem Friendly.