Forbid self-fulfilling prophecies—i.e. the AI cannot base its suggestions on predictions that are contingent upon the suggestions themselves.
You can’t forbid self-fullfilling prophecies and still have a functioning AI. The whole point is to find a self-fullfilling prophecy that something good will happen. The problem illustrated is that the AI chose a self-fullfilling prophecy that ranked highly in the simply specified goal it was optimizing for, but ranked poorly in terms of what the human actually wanted. That is, the AI was fully capable of granting the wish as it understood it, but the wish it understood was not what the human meant to wish for.
You can’t forbid self-fullfilling prophecies and still have a functioning AI. The whole point is to find a self-fullfilling prophecy that something good will happen. The problem illustrated is that the AI chose a self-fullfilling prophecy that ranked highly in the simply specified goal it was optimizing for, but ranked poorly in terms of what the human actually wanted. That is, the AI was fully capable of granting the wish as it understood it, but the wish it understood was not what the human meant to wish for.