I don’t know what the statement of the theorem would be. I don’t really think we’d have a clean definition of “contains daemons” and then have a proof that a particular circuit doesn’t contain daemons.
Also I expect we’re going to have to make some assumption that the problem is “generic” (or else be careful about what daemon means), ruling out problems with the consequentialism embedded in them.
(Also, see the comment thread with Wei Dai above, clearly the plausible version of this involves something more specific than daemons.)
Also I expect we’re going to have to make some assumption that the problem is “generic” (or else be careful about what daemon means), ruling out problems with the consequentialism embedded in them.
I agree. The following is an attempt to show that if we don’t rule out problems with the consequentialism embedded in them then the answer is trivially “no” (i.e. minimal circuits may contain consequentialists).
Let c be a minimal circuit that takes as input a string of length 10100 that encodes a Turing machine, and outputs a string that is the concatenation of the first 10100 configurations in the simulation of that Turing machine (each configuration is encoded as a string).
Now consider a string x′ that encodes a Turing machine that simulates some consequentialist (e.g. a human upload). For the input x′, the computation of the output of c simulates a consequentialist; and c is a minimal circuit.
I don’t know what the statement of the theorem would be. I don’t really think we’d have a clean definition of “contains daemons” and then have a proof that a particular circuit doesn’t contain daemons.
Also I expect we’re going to have to make some assumption that the problem is “generic” (or else be careful about what daemon means), ruling out problems with the consequentialism embedded in them.
(Also, see the comment thread with Wei Dai above, clearly the plausible version of this involves something more specific than daemons.)
I agree. The following is an attempt to show that if we don’t rule out problems with the consequentialism embedded in them then the answer is trivially “no” (i.e. minimal circuits may contain consequentialists).
Let c be a minimal circuit that takes as input a string of length 10100 that encodes a Turing machine, and outputs a string that is the concatenation of the first 10100 configurations in the simulation of that Turing machine (each configuration is encoded as a string).
Now consider a string x′ that encodes a Turing machine that simulates some consequentialist (e.g. a human upload). For the input x′, the computation of the output of c simulates a consequentialist; and c is a minimal circuit.