Instead, there already exist non-human value systems generated by humans threatening human well-being/existence and we don’t know how to fix that.
First, I am unaware of evidence (though I am aware of a lot of loud screaming) that human institutions pose an existential risk to humanity. I think the closest we come to that is the capability of US and Russia to launch an all-out nuclear exchange.
Second, the whole “non-human value systems” is much too fuzzy for my liking. Is self-preservation a human value? Let’s take an entity, say a large department within a governmental bureaucracy, the major values of which are self-preservation and the accrual of benefits (of various kinds) to its leadership. Is that a “non-human value system”? Should we call it “hostile AI” and be worried about it?
Third, the global financial system (or the “industrial capitalism”) is not an institution. It’s an ecosystem where many different entities coexist, fight, live, and die. I am not sure ecosystems have agency.
Fourth, it looks to me like his argument would shortcut to either a revolution or more malaria nets.
First, I am unaware of evidence (though I am aware of a lot of loud screaming) that human institutions pose an existential risk to humanity. I think the closest we come to that is the capability of US and Russia to launch an all-out nuclear exchange.
Second, the whole “non-human value systems” is much too fuzzy for my liking. Is self-preservation a human value? Let’s take an entity, say a large department within a governmental bureaucracy, the major values of which are self-preservation and the accrual of benefits (of various kinds) to its leadership. Is that a “non-human value system”? Should we call it “hostile AI” and be worried about it?
Third, the global financial system (or the “industrial capitalism”) is not an institution. It’s an ecosystem where many different entities coexist, fight, live, and die. I am not sure ecosystems have agency.
Fourth, it looks to me like his argument would shortcut to either a revolution or more malaria nets.