The OP framed the scenario in terms of directing the AI to design a FAI, but the technique is more general. It’s possibly safe for all problems with a verifiable solution.
People I don’t trust but don’t want to kill (or modify to cripple). A non-compliant transhuman with self modification ability may not be able to out-compete an FAI but if it is not quarantined it could force the FAI to burn resources to maintain dominance.
But it is something we can let the FAI build for us.
Shrug. For the purposes here they could be called froogles for all I care. The quarantine could occur in either stage depending on the preferences being implemented.
For what?
The OP framed the scenario in terms of directing the AI to design a FAI, but the technique is more general. It’s possibly safe for all problems with a verifiable solution.
People I don’t trust but don’t want to kill (or modify to cripple). A non-compliant transhuman with self modification ability may not be able to out-compete an FAI but if it is not quarantined it could force the FAI to burn resources to maintain dominance.
But it is something we can let the FAI build for us.
At what point does a transhuman become posthuman?
Shrug. For the purposes here they could be called froogles for all I care. The quarantine could occur in either stage depending on the preferences being implemented.
You mean posthuman?