In human society and at the highest scale, we solve the agent-principal problem by separation of powers—legislative, executive, and judiciary powers of state typically are divided in independent branches. This naturally leads to a categorization of AI-capabilities:
AI with legislative power (the power to make new rules)
AI with with high-level executive power (the power to make decisions)
AI with with low-level executive power (to carry out orders)
AI with a rule-enforcing power
AI with a power to create new knowledge / make suggestions for decisions
What Bostrom & co shows is that the seemingly innocent powers to create new knownledge and carry out low-level, well-specified tasks are in fact very unsafe. (The Riemann’s hypothesis solver, the paperclip maximizer).
What Bostrom implicitly assumes is that the higher levels of powers do not bring any important new dangers, and might, in fact, be better for the humanity. (The example of an all-powerful sovereign that decides and enforces moral laws in a way that makes them similar to physical laws.) I feel that this point requires more analysis. In general, each new capability brings more ways how to be unfriendly.
In human society and at the highest scale, we solve the agent-principal problem by separation of powers—legislative, executive, and judiciary powers of state typically are divided in independent branches. This naturally leads to a categorization of AI-capabilities:
AI with legislative power (the power to make new rules)
AI with with high-level executive power (the power to make decisions)
AI with with low-level executive power (to carry out orders)
AI with a rule-enforcing power
AI with a power to create new knowledge / make suggestions for decisions
What Bostrom & co shows is that the seemingly innocent powers to create new knownledge and carry out low-level, well-specified tasks are in fact very unsafe. (The Riemann’s hypothesis solver, the paperclip maximizer).
What Bostrom implicitly assumes is that the higher levels of powers do not bring any important new dangers, and might, in fact, be better for the humanity. (The example of an all-powerful sovereign that decides and enforces moral laws in a way that makes them similar to physical laws.) I feel that this point requires more analysis. In general, each new capability brings more ways how to be unfriendly.