An interesting conundrum: one of the main challenges of designing useful regulation for AI is that we don’t have any cheap and robust way to distinguish a dangerous neural net from a non-dangerous net (or, more generally, a dangerous program from a non-dangerous program). This is an area where technical research could, in principle, help a lot.
The problem is, if there were some robust metric for how dangerous a net is, and that metric were widely known and recognized (as it would probably need to be in order to be used for regulatory purposes), then someone would probably train a net to maximize that metric directly.
This seems to lead to the solution of trying to make your metric one-way, in the sense that your metric should
Provide an upper-bound on the dangerousness of your network
Compress the space of networks which map to approximately the same dangerousness level on the low end of dangerousness, and expand the space of networks which map to approximately the same dangerousness level on the upper end of dangerous, so that you can train your network to minimize the metric, but when you train your network to maximize the metric you end up in a degenerate are with technically very high measured danger levels but in actuality very low levels of dangerousness.
We can hope (or possibly prove) that as you optimize upwards on the metric you get subject to goodheart’s curse, but the opposite occurs on the lower end.
Sure, even seems a bit tautological: any such metric, to be robust, would need to contain in itself a definition of a dangerously-capable AI, so you probably wouldn’t even need to train a model to maximize it. You’d be able to just lift the design from the metric directly.
Do you have any thoughts on a softer version of this problem, where the metric can’t be maximized directly, but gives a concrete idea of what sort of challenge your AI needs to beat to qualify as AGI? (And therefore in which direction in the architectural-design-space you should be moving.)
Some variation on this seems like it might work as a “fire alarm” test set, but as you point out, inasmuch as it’s recognized, it’ll be misapplied for benchmarking instead.
(I suppose the ideal way to do it would be to hand it off to e. g. ARC, so they can use it if OpenAI invites them for safety-testing again. This way, SOTA models still get tested, but the actors who might misuse it aren’t aware of the testing’s particulars until they succeed anyway...)
An interesting conundrum: one of the main challenges of designing useful regulation for AI is that we don’t have any cheap and robust way to distinguish a dangerous neural net from a non-dangerous net (or, more generally, a dangerous program from a non-dangerous program). This is an area where technical research could, in principle, help a lot.
The problem is, if there were some robust metric for how dangerous a net is, and that metric were widely known and recognized (as it would probably need to be in order to be used for regulatory purposes), then someone would probably train a net to maximize that metric directly.
This seems to lead to the solution of trying to make your metric one-way, in the sense that your metric should
Provide an upper-bound on the dangerousness of your network
Compress the space of networks which map to approximately the same dangerousness level on the low end of dangerousness, and expand the space of networks which map to approximately the same dangerousness level on the upper end of dangerous, so that you can train your network to minimize the metric, but when you train your network to maximize the metric you end up in a degenerate are with technically very high measured danger levels but in actuality very low levels of dangerousness.
We can hope (or possibly prove) that as you optimize upwards on the metric you get subject to goodheart’s curse, but the opposite occurs on the lower end.
Sure, even seems a bit tautological: any such metric, to be robust, would need to contain in itself a definition of a dangerously-capable AI, so you probably wouldn’t even need to train a model to maximize it. You’d be able to just lift the design from the metric directly.
Do you have any thoughts on a softer version of this problem, where the metric can’t be maximized directly, but gives a concrete idea of what sort of challenge your AI needs to beat to qualify as AGI? (And therefore in which direction in the architectural-design-space you should be moving.)
Some variation on this seems like it might work as a “fire alarm” test set, but as you point out, inasmuch as it’s recognized, it’ll be misapplied for benchmarking instead.
(I suppose the ideal way to do it would be to hand it off to e. g. ARC, so they can use it if OpenAI invites them for safety-testing again. This way, SOTA models still get tested, but the actors who might misuse it aren’t aware of the testing’s particulars until they succeed anyway...)