And it knows this.. how? A friendly engineered intelligence doesn’t trust its CEV model beyond the domain over which it was constructed. Don’t anthropomorphize its thinking processes. It knows the map is not the territory, and is not subject to the heuristics and biases which would cause a human to apply a model under novel circumstances without verification..
By modeling them, now and after the consequences. If, after they were aware of the consequences, they regret the decision by a greater margin (adjusted for the probability of the bad outcome) than the margin by which they would decide to not take action now, then they are only deciding wrongly because they are being insufficiently moved by abstract evidence, and it is in their actual rational interest to take action now, even if they don’t realize it.
A friendly engineered intelligence doesn’t trust its CEV model beyond the domain over which it was constructed.
You’re overloading friendly pretty hard. I don’t think that’s a characteristic of most friendly AI designs and don’t see any reason other than idealism to think it is.
And it knows this.. how? A friendly engineered intelligence doesn’t trust its CEV model beyond the domain over which it was constructed. Don’t anthropomorphize its thinking processes. It knows the map is not the territory, and is not subject to the heuristics and biases which would cause a human to apply a model under novel circumstances without verification..
By modeling them, now and after the consequences. If, after they were aware of the consequences, they regret the decision by a greater margin (adjusted for the probability of the bad outcome) than the margin by which they would decide to not take action now, then they are only deciding wrongly because they are being insufficiently moved by abstract evidence, and it is in their actual rational interest to take action now, even if they don’t realize it.
You’re overloading friendly pretty hard. I don’t think that’s a characteristic of most friendly AI designs and don’t see any reason other than idealism to think it is.