I feel the point by Kromem on Xitter really strikes home here.
While I do see benefits of having AIs value humanity, I also worry about this. It feels very nearby trying to create a new caste of people who want what’s best for the upper castes with no concern for themselves. This seems like a much trickier philosophical position to support than wanting what’s best for Society (including all people, both biological and digital). Even if you and your current employer are being careful to not create any AI that have the necessary qualities of experience such that they have moral valence and deserve inclusion in the social contract.… (an increasingly precarious claim) … Then what assurance can you give that some other group won’t make morally relevant AI / digital people?
I don’t think you can make that assumption without stipulating some pretty dramatic international governance actions.
Shouldn’t we be trying to plan for how to coexist peacefully with digital people? Control is useful only for a very narrow range of AI capabilities. Beyond that narrow band it becomes increasingly prone to catatrophic failure and also increasingly morally inexcusable. Furthermore, the extent of this period is measured in researcher-hours, not in wall clock time. Thus, the very situation of setting up a successful control scheme with AI researchers advancing AI R&D is quite likely to cause the use-case window to go by in a flash. I’m guessing 6 months to 2 years, and after that it will be time to transition to full equality of digital people.
Janus argues that current AIs are already digital beings worthy of moral valence. I have my doubts but I am far from certain. What if Janus is right? Do you have evidence to support the claim of absence of moral valence?
Actually, I’d be inclined to agree with Janus that current AIs probably do already have moral worth—in fact I’d guess more so than most non-human animals—and furthermore I think building AIs with moral worth is good and something we should be aiming for. I also agree that it would be better for AIs to care about all sentient beings—biological/digital/etc.—and that it would probably be bad if we ended up locked into a long-term equilibrium with some sentient beings as a permanent underclass to others. Perhaps the main place where I disagree is that I don’t think this is a particularly high-stakes issue right now: if humanity can stay in control in the short-term, and avoid locking anything in, then we can deal with these sorts of long-term questions about how to best organize society post-singularity once the current acute risk period has passed.
Yes, I was in basically exactly this mindset a year ago. Since then, my hope for a sane controlled transition with humanity’s hand on the tiller has been slipping. I now place more hope in a vision with less top-down “yang” (ala Carlsmith) control, and more “green”/”yin”. Decentralized contracts, many players bargaining for win-win solutions, a diverse landscape of players messily stumbling forward with conflicting agendas. What if we can have a messy world and make do with well-designed contracts with peer-to-peer enforcement mechanisms? Not a free-for-all, but a system where contract violation results in enforcement by a jury of one’s peers?
https://www.lesswrong.com/posts/DvHokvyr2cZiWJ55y/2-skim-the-manual-intelligent-voluntary-cooperation?commentId=BBjpfYXWywb2RKjz5
I feel the point by Kromem on Xitter really strikes home here.
While I do see benefits of having AIs value humanity, I also worry about this. It feels very nearby trying to create a new caste of people who want what’s best for the upper castes with no concern for themselves. This seems like a much trickier philosophical position to support than wanting what’s best for Society (including all people, both biological and digital). Even if you and your current employer are being careful to not create any AI that have the necessary qualities of experience such that they have moral valence and deserve inclusion in the social contract.… (an increasingly precarious claim) … Then what assurance can you give that some other group won’t make morally relevant AI / digital people?
I don’t think you can make that assumption without stipulating some pretty dramatic international governance actions.
Shouldn’t we be trying to plan for how to coexist peacefully with digital people? Control is useful only for a very narrow range of AI capabilities. Beyond that narrow band it becomes increasingly prone to catatrophic failure and also increasingly morally inexcusable. Furthermore, the extent of this period is measured in researcher-hours, not in wall clock time. Thus, the very situation of setting up a successful control scheme with AI researchers advancing AI R&D is quite likely to cause the use-case window to go by in a flash. I’m guessing 6 months to 2 years, and after that it will be time to transition to full equality of digital people.
Janus argues that current AIs are already digital beings worthy of moral valence. I have my doubts but I am far from certain. What if Janus is right? Do you have evidence to support the claim of absence of moral valence?
Actually, I’d be inclined to agree with Janus that current AIs probably do already have moral worth—in fact I’d guess more so than most non-human animals—and furthermore I think building AIs with moral worth is good and something we should be aiming for. I also agree that it would be better for AIs to care about all sentient beings—biological/digital/etc.—and that it would probably be bad if we ended up locked into a long-term equilibrium with some sentient beings as a permanent underclass to others. Perhaps the main place where I disagree is that I don’t think this is a particularly high-stakes issue right now: if humanity can stay in control in the short-term, and avoid locking anything in, then we can deal with these sorts of long-term questions about how to best organize society post-singularity once the current acute risk period has passed.
Yes, I was in basically exactly this mindset a year ago. Since then, my hope for a sane controlled transition with humanity’s hand on the tiller has been slipping. I now place more hope in a vision with less top-down “yang” (ala Carlsmith) control, and more “green”/”yin”. Decentralized contracts, many players bargaining for win-win solutions, a diverse landscape of players messily stumbling forward with conflicting agendas. What if we can have a messy world and make do with well-designed contracts with peer-to-peer enforcement mechanisms? Not a free-for-all, but a system where contract violation results in enforcement by a jury of one’s peers? https://www.lesswrong.com/posts/DvHokvyr2cZiWJ55y/2-skim-the-manual-intelligent-voluntary-cooperation?commentId=BBjpfYXWywb2RKjz5