Yes I think 2) is closer to what I’m suggesting. Effectively what I am thinking is what would happen if, by design, there was only one utility function defined in absolute terms (I’ve tried to explaine this in the latest open thread), so that the AI could never assume we would disagree with it. By all means, as it tries to learn this function, it might get it completely wrong, so this certainly doesn’t solve the problem of how to teach it the right values, but at least it looks to me that with such a design it would never be motivated to lie to us because it would always think we would be in perfect agreement. Also, I think it would make it indifferent to our actions as it would always assume we would follow the plan from that point onward. The utility function it uses (same for itself and for us) would be the union of a utility function that describes the goal we want it to achieve, which would be unchangeable, and the set of values it is learning after each iteration. I’m trying to understand what would be wrong with this design, cause to me it looks like we would have achieved an honest AI, which is a good start.
Yes I think 2) is closer to what I’m suggesting. Effectively what I am thinking is what would happen if, by design, there was only one utility function defined in absolute terms (I’ve tried to explaine this in the latest open thread), so that the AI could never assume we would disagree with it. By all means, as it tries to learn this function, it might get it completely wrong, so this certainly doesn’t solve the problem of how to teach it the right values, but at least it looks to me that with such a design it would never be motivated to lie to us because it would always think we would be in perfect agreement. Also, I think it would make it indifferent to our actions as it would always assume we would follow the plan from that point onward. The utility function it uses (same for itself and for us) would be the union of a utility function that describes the goal we want it to achieve, which would be unchangeable, and the set of values it is learning after each iteration. I’m trying to understand what would be wrong with this design, cause to me it looks like we would have achieved an honest AI, which is a good start.