Rather than saying that most likely-to-work plans for curing cancer route through consequentialism, I think it would be more precise to say that most simple likely-to-work plans route through consequentialism.
For every plan that can be summarized as “build a powerful consequentialist and then delegate the problem to it”, it seems like there should be a corresponding (perhaps very complicated) plan that can be summarized as “directly execute the plan that that consequentialist would have used if you had built it.”
The size of that complexity penalty varies depending on the nature of the problem. There’s maybe a useful sense in which the “hard problems” are exactly the ones where the complexity penalty for avoiding the consequentialist is large.
If your planning space is finite, it doesn’t contain sufficiently complex plans for your argument to go through. If your planning space is infinite, you need to select some measure to be able to talk about “most”, and then that measure will capture the “simple” aspect.
Rather than saying that most likely-to-work plans for curing cancer route through consequentialism, I think it would be more precise to say that most simple likely-to-work plans route through consequentialism.
For every plan that can be summarized as “build a powerful consequentialist and then delegate the problem to it”, it seems like there should be a corresponding (perhaps very complicated) plan that can be summarized as “directly execute the plan that that consequentialist would have used if you had built it.”
The size of that complexity penalty varies depending on the nature of the problem. There’s maybe a useful sense in which the “hard problems” are exactly the ones where the complexity penalty for avoiding the consequentialist is large.
If your planning space is finite, it doesn’t contain sufficiently complex plans for your argument to go through. If your planning space is infinite, you need to select some measure to be able to talk about “most”, and then that measure will capture the “simple” aspect.