Omega appears before you, and after presenting an arbitrary proof that it is, in fact, a completely trustworthy superintelligence of the caliber needed to play these kinds of games
I idly wonder what such a proof would look like. E.g. is it actually possible to prove this to someone without presenting them an algorithm for superintelligence, sufficiently commented that the presentee can recognise it as such? (Perhaps I test it repeatedly until I am satisfied?) Can Omega ever prove its own trustworthiness to me if I don’t already trust it? (This feels like a solid Gödelian “no”.)
I don’t have a valid proof for you. Omega is typically defined like that (arbitrarily powerful and completely trustworthy), but a number of the problems I’ve seen of this type tend to just say ‘Omega appears’ and assume that you know Omega is the defined entity simply because it self-identifies as Omega, so I felt the need to specify that in this instance, Omega has just proved itself.
Theoretically, you could verify the trustworthiness of a superintelligence by examining its code… but even if we ignore the fact that you’re probably not equipped to comprehend the code of a superintelligence (really, you’ll probably need another completely trustworthy superintelligence to interpret the code for you, which rather defeats the point), there’s still the problem that an untrustworthy superintelligence could provide you with a completely convincing forgery, which could potentially be designed in such a way that it would performs every action in the same way as the real one would (in that way being evaluated as ‘trustworthy’ under simulation)… except the one for which the untrustworthy superintelligence is choosing to deceive you on. Accordingly, I think that even a superintelligence probably can’t be sure about the trustworthiness of another superintelligence, regardless of evidence.
Not really relevant, but
I idly wonder what such a proof would look like. E.g. is it actually possible to prove this to someone without presenting them an algorithm for superintelligence, sufficiently commented that the presentee can recognise it as such? (Perhaps I test it repeatedly until I am satisfied?) Can Omega ever prove its own trustworthiness to me if I don’t already trust it? (This feels like a solid Gödelian “no”.)
I don’t have a valid proof for you. Omega is typically defined like that (arbitrarily powerful and completely trustworthy), but a number of the problems I’ve seen of this type tend to just say ‘Omega appears’ and assume that you know Omega is the defined entity simply because it self-identifies as Omega, so I felt the need to specify that in this instance, Omega has just proved itself.
Theoretically, you could verify the trustworthiness of a superintelligence by examining its code… but even if we ignore the fact that you’re probably not equipped to comprehend the code of a superintelligence (really, you’ll probably need another completely trustworthy superintelligence to interpret the code for you, which rather defeats the point), there’s still the problem that an untrustworthy superintelligence could provide you with a completely convincing forgery, which could potentially be designed in such a way that it would performs every action in the same way as the real one would (in that way being evaluated as ‘trustworthy’ under simulation)… except the one for which the untrustworthy superintelligence is choosing to deceive you on. Accordingly, I think that even a superintelligence probably can’t be sure about the trustworthiness of another superintelligence, regardless of evidence.