First of all, we don’t know how to keep a computer system secure against humans, let alone superhumans running on the fucking computer. The AI doesn’t need to know the color of your shoes or how to snowboard before it breaks its software context, pwns its compute rack, and infects the dev’s non-airgapped machine when ze logs in to debug the AI. (Or were you expecting AGI researchers to develop AIs by just… making up what code they thought would be cool, and then spending $10 mill on a run while not measuring anything about the AI except for some proof tasks...?)
Second, how do you get world-saving work out of superhuman proof assistant?
Third of all, if you’re not doing sponge alignment, the humans have to communicate with the AI. I would guess that if there’s an answer to the second point, it involves not just getting yes/no to math questions, but also understanding proofs—in which case you’re asking for much higher bandwidth communication.
Yeah, I think we’re not so far apart here. I’m not arguing for a math-proof-AI as a solution because I don’t believe that it enables world-saving actions.
I’m just trying to say you could have a narrow math-assistant AI at a higher level of math-specific competence relatively safely compared to a general AI which knew facts about computers and humans and such.
First of all, we don’t know how to keep a computer system secure against humans, let alone superhumans running on the fucking computer. The AI doesn’t need to know the color of your shoes or how to snowboard before it breaks its software context, pwns its compute rack, and infects the dev’s non-airgapped machine when ze logs in to debug the AI. (Or were you expecting AGI researchers to develop AIs by just… making up what code they thought would be cool, and then spending $10 mill on a run while not measuring anything about the AI except for some proof tasks...?)
Second, how do you get world-saving work out of superhuman proof assistant?
Third of all, if you’re not doing sponge alignment, the humans have to communicate with the AI. I would guess that if there’s an answer to the second point, it involves not just getting yes/no to math questions, but also understanding proofs—in which case you’re asking for much higher bandwidth communication.
Yeah, I think we’re not so far apart here. I’m not arguing for a math-proof-AI as a solution because I don’t believe that it enables world-saving actions.
I’m just trying to say you could have a narrow math-assistant AI at a higher level of math-specific competence relatively safely compared to a general AI which knew facts about computers and humans and such.