1) You don’t get to assume “because superhuman!” the AI can know X, for any X. EY is an immensely complex human being, and no machine learning algorithm can simply digest a realistically finite sample of his written work and know with any certainty how he thinks or what surprises he has planned. It would be able to, e.g. finish sentences correctly and do other tricks, and given a range of possible endings predict which ones are likely. But this shouldn’t be too surprising: it’s a trick we humans are able to do too. The AI’s predictions may be more accurate, but not qualitatively different than any of the many HPMOR prediction threads.
2) Ok maybe—maybe! -- in principle, in theory it might be possible for a perfect, non-heuristic Bayesian with omniscient access to the inner lives and external writings of every other human being in existence would have a data set large enough data set to make reliable enough extrapolations from as low-bandwidth a medium as EY’s published fanfics. Maybe, as this is not a logical consequence. Even so, we’re talking about a boxed AI, remember? If it is everywhere and omniscient, then it’s already out of the box.
So, two issues:
1) You don’t get to assume “because superhuman!” the AI can know X, for any X. EY is an immensely complex human being, and no machine learning algorithm can simply digest a realistically finite sample of his written work and know with any certainty how he thinks or what surprises he has planned. It would be able to, e.g. finish sentences correctly and do other tricks, and given a range of possible endings predict which ones are likely. But this shouldn’t be too surprising: it’s a trick we humans are able to do too. The AI’s predictions may be more accurate, but not qualitatively different than any of the many HPMOR prediction threads.
2) Ok maybe—maybe! -- in principle, in theory it might be possible for a perfect, non-heuristic Bayesian with omniscient access to the inner lives and external writings of every other human being in existence would have a data set large enough data set to make reliable enough extrapolations from as low-bandwidth a medium as EY’s published fanfics. Maybe, as this is not a logical consequence. Even so, we’re talking about a boxed AI, remember? If it is everywhere and omniscient, then it’s already out of the box.
I’m happy to assume the AI is omniscient, just impotent. I think such an AI could still be boxed.