I don’t just do it, I ask followup questions, like what are the effects in more detail. If I am unfortunate, I ask something like “how could I do that”, and get an answer like “e-mail the sequence to a university lab, along with this strangely compelling argument” and I read the strangely compelling argument which is included as part of the answer.
So if a goal-directed AI can hack your mind, it is pretty easy to accidentally ask the oracle AI a question where the answer will do the same thing. If you can avoid that, you need to ask lots of questions before implementing its solution so you get a good idea of what you are doing.
I don’t just do it, I ask followup questions, like what are the effects in more detail. If I am unfortunate, I ask something like “how could I do that”, and get an answer like “e-mail the sequence to a university lab, along with this strangely compelling argument” and I read the strangely compelling argument which is included as part of the answer.
So if a goal-directed AI can hack your mind, it is pretty easy to accidentally ask the oracle AI a question where the answer will do the same thing. If you can avoid that, you need to ask lots of questions before implementing its solution so you get a good idea of what you are doing.