Also I expect we’re going to have to make some assumption that the problem is “generic” (or else be careful about what daemon means), ruling out problems with the consequentialism embedded in them.
I agree. The following is an attempt to show that if we don’t rule out problems with the consequentialism embedded in them then the answer is trivially “no” (i.e. minimal circuits may contain consequentialists).
Let c be a minimal circuit that takes as input a string of length 10100 that encodes a Turing machine, and outputs a string that is the concatenation of the first 10100 configurations in the simulation of that Turing machine (each configuration is encoded as a string).
Now consider a string x′ that encodes a Turing machine that simulates some consequentialist (e.g. a human upload). For the input x′, the computation of the output of c simulates a consequentialist; and c is a minimal circuit.
I agree. The following is an attempt to show that if we don’t rule out problems with the consequentialism embedded in them then the answer is trivially “no” (i.e. minimal circuits may contain consequentialists).
Let c be a minimal circuit that takes as input a string of length 10100 that encodes a Turing machine, and outputs a string that is the concatenation of the first 10100 configurations in the simulation of that Turing machine (each configuration is encoded as a string).
Now consider a string x′ that encodes a Turing machine that simulates some consequentialist (e.g. a human upload). For the input x′, the computation of the output of c simulates a consequentialist; and c is a minimal circuit.