I agree with a). c) seems to me to be very optimistic, but that’s mostly an intuition, I don’t have a strong argument against it (and I wouldn’t discourage people who are enthusiastic about it from working on it).
The argument in b) makes sense; I think the part that I disagree with is:
moving from utility maximizes to other types of AIs is just replacing something that is relatively easy to reason about with something that it is harder to reason about, thereby obscuring the problems (that are still there).
The counterargument is “current AI systems don’t look like long term planners”, but of course it is possible to respond to that with “AGI will be very different from current AI systems”, and then I have nothing to say beyond “I think AGI will be like current AI systems”.
Well, any system that satisfies the Minimal Requirement is doing long term planning on some level. For example, if your AI is approval directed, it still needs to learn how to make good plans that will be approved. Once your system has a superhuman capability of producing plans somewhere inside, you should worry about that capability being applied in the wrong direction (in particular due to mesa-optimization / daemons). Also, even without long term planning, extreme optimization is dangerous (for example an approval directed AI might create some kind of memetic supervirus).
But, I agree that these arguments are not enough to be confident of the strong empirical claim.
I agree with a). c) seems to me to be very optimistic, but that’s mostly an intuition, I don’t have a strong argument against it (and I wouldn’t discourage people who are enthusiastic about it from working on it).
The argument in b) makes sense; I think the part that I disagree with is:
The counterargument is “current AI systems don’t look like long term planners”, but of course it is possible to respond to that with “AGI will be very different from current AI systems”, and then I have nothing to say beyond “I think AGI will be like current AI systems”.
Well, any system that satisfies the Minimal Requirement is doing long term planning on some level. For example, if your AI is approval directed, it still needs to learn how to make good plans that will be approved. Once your system has a superhuman capability of producing plans somewhere inside, you should worry about that capability being applied in the wrong direction (in particular due to mesa-optimization / daemons). Also, even without long term planning, extreme optimization is dangerous (for example an approval directed AI might create some kind of memetic supervirus).
But, I agree that these arguments are not enough to be confident of the strong empirical claim.