(Eli’s personal “trying to have thoughts” before reading the other comments. Probably incoherent. Possibly not even on topic. Respond iff you’d like.)
(Also, my thinking here is influenced by having read this report recently.)
On the one hand, I can see the intuition that if a daemon is solving a problem, there is some part of the system that is solving the problem, and there is another part that is working to (potentially) optimize against you. In theory, we could “cut out” the part that is the problematic agency, preserving the part that solves the problem. And that circuit would be smaller.
Does that argument apply in the evolution/human case?
Could I “cut away” everything that isn’t solving the problem of inclusive genetic fitness and end up with a smaller “inclusive genetic fitness maximizer”?
On the on hand, this seems like a kind of confusing frame. If some humans do well on the metric of inclusive genetic fitness (in the ancestral environment), this isn’t because there’s a part of the human that’s optimizing for that and then another part that’s patiently waiting and watching for a context shift in order to pull a treacherous turn on evolution. The human is just pursuing its goals, and as a side effect, does well at the IGF metric.
But it also seems like you could, in principle, build an Inclusive Genetic Fitness Maximizer out of human neuro-machinery: a mammal-like brain that does optimize for spreading its genes.
Would such an entity be computationally smaller than a human?
Maybe? I don’t have a strong intuition either way. It really doesn’t seem like much of the “size” of the system is due to the encoding of the goals. Approximately 0 of the difference in size is due to the goals?
A much better mind design might be much smaller, but that wouldn’t make it any less daemonic.
And if, in fact, the computationally smallest way to solve the IGF problem is as a side-effect of some processes optimizing for some other goal, then the minimum circuit is not daemon-free.
Though I don’t know of any good reason why is should be the case that not optimizing directly for the metric works better than optimizing directly for it. True, evolution “chose” to design human as adaptation-executors, but this seems due to evolution’s constraints in searching the space, not due to indirectness having any virtue over directness. Right?
(Eli’s personal “trying to have thoughts” before reading the other comments. Probably incoherent. Possibly not even on topic. Respond iff you’d like.)
(Also, my thinking here is influenced by having read this report recently.)
On the one hand, I can see the intuition that if a daemon is solving a problem, there is some part of the system that is solving the problem, and there is another part that is working to (potentially) optimize against you. In theory, we could “cut out” the part that is the problematic agency, preserving the part that solves the problem. And that circuit would be smaller.
________________________________________________________________________
Does that argument apply in the evolution/human case?
Could I “cut away” everything that isn’t solving the problem of inclusive genetic fitness and end up with a smaller “inclusive genetic fitness maximizer”?
On the on hand, this seems like a kind of confusing frame. If some humans do well on the metric of inclusive genetic fitness (in the ancestral environment), this isn’t because there’s a part of the human that’s optimizing for that and then another part that’s patiently waiting and watching for a context shift in order to pull a treacherous turn on evolution. The human is just pursuing its goals, and as a side effect, does well at the IGF metric.
But it also seems like you could, in principle, build an Inclusive Genetic Fitness Maximizer out of human neuro-machinery: a mammal-like brain that does optimize for spreading its genes.
Would such an entity be computationally smaller than a human?
Maybe? I don’t have a strong intuition either way. It really doesn’t seem like much of the “size” of the system is due to the encoding of the goals. Approximately 0 of the difference in size is due to the goals?
A much better mind design might be much smaller, but that wouldn’t make it any less daemonic.
And if, in fact, the computationally smallest way to solve the IGF problem is as a side-effect of some processes optimizing for some other goal, then the minimum circuit is not daemon-free.
Though I don’t know of any good reason why is should be the case that not optimizing directly for the metric works better than optimizing directly for it. True, evolution “chose” to design human as adaptation-executors, but this seems due to evolution’s constraints in searching the space, not due to indirectness having any virtue over directness. Right?