What ciphergoth said. Also, we can’t derive an ‘ought’ from an ‘is’ - we don’t actually know whether letting the AI out is the right thing to do (unless the contest had a stipulation that the AI was evil and the box keeper knew it, which I don’t remember being the case). Perhaps the rational thing is to let the AI out!
Further, this could also just be a test of stubbornness or patience. Which aren’t neither of them rationality. But good try anyway.
For the first objection, that the AI Box experiment has too many unknowns, let us instead construct an argument based on psychological tricks for any bad conclusion to try on the subject.
For the second objection, that this tests stubbornness rather than rationality, use a sequence of tests, some using tricks to argue for false conclusions, and some using Bayesian evidence for a good conclusion. The score should reward being convinced when, and only when, the subject should be convinced. Stubbornness can only meet half this requirement.
The task of compiling arguments of both types, which would not be readily available to the subject ahead of time, remains.
What ciphergoth said. Also, we can’t derive an ‘ought’ from an ‘is’ - we don’t actually know whether letting the AI out is the right thing to do (unless the contest had a stipulation that the AI was evil and the box keeper knew it, which I don’t remember being the case). Perhaps the rational thing is to let the AI out!
Further, this could also just be a test of stubbornness or patience. Which aren’t neither of them rationality. But good try anyway.
For the first objection, that the AI Box experiment has too many unknowns, let us instead construct an argument based on psychological tricks for any bad conclusion to try on the subject.
For the second objection, that this tests stubbornness rather than rationality, use a sequence of tests, some using tricks to argue for false conclusions, and some using Bayesian evidence for a good conclusion. The score should reward being convinced when, and only when, the subject should be convinced. Stubbornness can only meet half this requirement.
The task of compiling arguments of both types, which would not be readily available to the subject ahead of time, remains.