I’m not too concerned about the rational agent case. If we have a fully rational agent whose values I endorse, the Friendliness problem has either been solved or turns out to be irrelevant.
It’s just a way to pin down the problem. If we can show that the AI in a box could misinform an idealized rational agent via selective evidence, then we know it can do so to us. If it can’t misinform the idealized agent, then there exists some method by which we can resist it.
Also,I don’t think idealized rational agents can actually exist anyway. All riddles involving them are for the sake of narrowing down some other problem.
It’s just a way to pin down the problem. If we can show that the AI in a box could misinform an idealized rational agent via selective evidence, then we know it can do so to us. If it can’t misinform the idealized agent, then there exists some method by which we can resist it.
Also,I don’t think idealized rational agents can actually exist anyway. All riddles involving them are for the sake of narrowing down some other problem.