AIXI would be very good at making complex plans and doing well first time.
Agreed, I claim we have no clue at how to make anything remotely like AIXI in the real world.
Humans have, at least somewhat, the ability to notice that there should be a good plan in this region, find and execute that plan successfully.
Agreed, in a CAIS world, the system of interacting services would probably notice the plan but not execute it because of some service that is meant to prevent it from doing crazy things that humans would not want.
What I am saying is that this form of AI is sufficiently limited that there are still large incentives to make AGI and the CAIS can’t protect us from making an unfriendly AGI.
This definitely seems like the crux for many people. I’m quite unsure about this point; it seems plausible to me that CAIS could in fact do most things such that there aren’t very large incentives, especially if the Factored Cognition hypothesis is true.
I’m also not sure how strong the self improvement can be when the service maker service is only making little tweaks to existing algorithms rather than designing strange new algorithms.
I don’t see why it would have to be little tweaks to existing algorithms, it seems plausible to have the R&D services consider entirely new algorithms as well.
Agreed, I claim we have no clue at how to make anything remotely like AIXI in the real world.
Agreed, in a CAIS world, the system of interacting services would probably notice the plan but not execute it because of some service that is meant to prevent it from doing crazy things that humans would not want.
This definitely seems like the crux for many people. I’m quite unsure about this point; it seems plausible to me that CAIS could in fact do most things such that there aren’t very large incentives, especially if the Factored Cognition hypothesis is true.
I don’t see why it would have to be little tweaks to existing algorithms, it seems plausible to have the R&D services consider entirely new algorithms as well.