Sure, I don’t think humanity is in any danger of being destroyed by conventional technologies, and I’m pretty sure the Singularity will be happen—in one form or another—way before then. But there may very well be a lot of suffering on the way.
Have you checked out CFAI? It’s like CEV but with less of an emphasis on humans. I really don’t like humans and would rather only deal with them via implicit meta-level ‘get information about morality from your environment’ means, which is more explicit in CFAI than CEV.
CEV takes more of an economic perspective where agent-extrapolations make deals with each other. The “good” agent-extrapolations might win out in the end (due to having a more-timeless discount rate, say), but there might be a lot of suffering along the way. CFAI on the other hand takes a less deal-centric perspective where the AI’s more directly supposed to reason everything through from first principles, which can avoid predictably-stupid-in-retrospect agents getting much of the future’s pie, so to speak. So I’m more afraid of CEV-like thinking than CFAI-like thinking, even though both are scary, because I am more afraid of humans being evil than I’m afraid of me not getting what I want. This may or may not overlap at all with your concerns.
(The difference isn’t necessarily whether or not they converge on the same policy, it might also be how quickly they converge on that policy. CFAI seems like it’d converge on justifiedness more quickly, but maybe not.)
Sure, I don’t think humanity is in any danger of being destroyed by conventional technologies, and I’m pretty sure the Singularity will be happen—in one form or another—way before then. But there may very well be a lot of suffering on the way.
Have you checked out CFAI? It’s like CEV but with less of an emphasis on humans. I really don’t like humans and would rather only deal with them via implicit meta-level ‘get information about morality from your environment’ means, which is more explicit in CFAI than CEV.
I’ve read part of it, though not all. (I’m a bit confused as to how your comment relates to mine.)
CEV takes more of an economic perspective where agent-extrapolations make deals with each other. The “good” agent-extrapolations might win out in the end (due to having a more-timeless discount rate, say), but there might be a lot of suffering along the way. CFAI on the other hand takes a less deal-centric perspective where the AI’s more directly supposed to reason everything through from first principles, which can avoid predictably-stupid-in-retrospect agents getting much of the future’s pie, so to speak. So I’m more afraid of CEV-like thinking than CFAI-like thinking, even though both are scary, because I am more afraid of humans being evil than I’m afraid of me not getting what I want. This may or may not overlap at all with your concerns.
(The difference isn’t necessarily whether or not they converge on the same policy, it might also be how quickly they converge on that policy. CFAI seems like it’d converge on justifiedness more quickly, but maybe not.)