Additionally and a more difficult challenge is that even friendly AIs could want to maximize their utility even at our collective expense...
I think that a perfectly Friendly AI would not do this, by definition. An imperfect one, however, could.
I’m currently engaged in playing this game (I wish you had continued)
Er, sorry, which game should I continue ?
AI could potentially have the capacity to accurately model a human mind and then simulate the decision tree of all the potential conversations and their paths through the tree...
To be fair, merely constructing the tree is not enough; the tree must also contain at least one reachable winning state. By analogy, let’s say you’re arguing with a Young-Earth Creationist on a forum. Yes, you could predict his arguments, and his responses to your arguments; but that doesn’t mean that you’ll be able to ever persuade him of anything.
It is possible that even a transhuman AI would be unable to persuade a sufficiently obstinate human of anything, but I wouldn’t want to bet on that.
In short it seems to me that it’s inherently unsafe to allow even a low bandwidth information flow to the outside world by means of a human who can only use it’s own memory.
I think that a perfectly Friendly AI would not do this, by definition. An imperfect one, however, could.
Er, sorry, which game should I continue ?
To be fair, merely constructing the tree is not enough; the tree must also contain at least one reachable winning state. By analogy, let’s say you’re arguing with a Young-Earth Creationist on a forum. Yes, you could predict his arguments, and his responses to your arguments; but that doesn’t mean that you’ll be able to ever persuade him of anything.
It is possible that even a transhuman AI would be unable to persuade a sufficiently obstinate human of anything, but I wouldn’t want to bet on that.
Right.