Defining the metric for cooperation robustly enough that you could unleash the resulting evolved AI on the real world might not be any easier than figuring out what an FAI’s utility function should be directly.
Also, a sufficiently intelligent AI may be able to hijack the game before we could decide whether it was ready to be released.
Defining the metric for cooperation robustly enough that you could unleash the resulting evolved AI on the real world might not be any easier than figuring out what an FAI’s utility function should be directly.
Also, a sufficiently intelligent AI may be able to hijack the game before we could decide whether it was ready to be released.