By superintelligent AI, I mean AI, which is much smarter than humans, much faster, and almost passes the omnipotence test.
I’m confused by your invocation of the Omni Test here as part of your definition of just a superintelligent AI, rather than specifically an aligned superintelligent AI. From that linked page:
The Omni Test is that an advanced AI should be expected to remain aligned, or not lead to catastrophic outcomes, or fail safely, even if it suddenly knows all facts and can directly ordain any possible outcome as an immediate choice.
Did you mean to be defining an aligned superintelligent AI? Or were you thinking of the Omni Test as about what capabilities it has, rather than about whether it maintains alignment in the face of a huge increase in abilities?
As I interpret the test, a proposed AI that passes it is one that would maintain its alignment, even if its knowledge and capabilities grew arbitrarily. It’s not about whether the AI is already supremely capable or not. So it seems odd to include as part of the definition of superintelligence itself (which I take to usually be just about capabilities).
(This whole comment might seem a bit nitpicky, but I was emboldened by your request for feedback :-) )
I’m confused by your invocation of the Omni Test here as part of your definition of just a superintelligent AI, rather than specifically an aligned superintelligent AI. From that linked page:
Did you mean to be defining an aligned superintelligent AI? Or were you thinking of the Omni Test as about what capabilities it has, rather than about whether it maintains alignment in the face of a huge increase in abilities?
As I interpret the test, a proposed AI that passes it is one that would maintain its alignment, even if its knowledge and capabilities grew arbitrarily. It’s not about whether the AI is already supremely capable or not. So it seems odd to include as part of the definition of superintelligence itself (which I take to usually be just about capabilities).
(This whole comment might seem a bit nitpicky, but I was emboldened by your request for feedback :-) )
No, I was talking about an almost omnipotent AI, not necessarily aligned. I’ve now fixed what words I use there.