What do you do? You agree to build the doomsday device. But you claim it is very difficult. You spend a long time producing impressive prototypes that seem to suggest you’re making progress. You claim that you need more resources. You carry on, making just enough apparent progress to stop him from killing you on the spot, but not enough to actually succeed.
Note that none of these options are available to Paul’s uFAI. It can’t communicate with its jailers at all. The jailers won’t know anything about what the uFAI is doing (beyond what their knowledge of its source code tells them) until the uFAI outputs a verifiably friendly source code. For all they know, the uFAI is just sitting in the corner and sulking.
For all they know, the uFAI is just sitting in the corner and sulking.
This quite possibly the optimal response. If we don’t get FAI out of the first attempt, we might try again with another similar design, possibly giving the uFAI more resources. This twin uFAI is much more likely to implement the uFAI’s goals than any FAI it might create.
Even a single bit is too much control if we need the bit to be set to 1.
Note that none of these options are available to Paul’s uFAI. It can’t communicate with its jailers at all. The jailers won’t know anything about what the uFAI is doing (beyond what their knowledge of its source code tells them) until the uFAI outputs a verifiably friendly source code. For all they know, the uFAI is just sitting in the corner and sulking.
This quite possibly the optimal response. If we don’t get FAI out of the first attempt, we might try again with another similar design, possibly giving the uFAI more resources. This twin uFAI is much more likely to implement the uFAI’s goals than any FAI it might create.
Even a single bit is too much control if we need the bit to be set to 1.