We’re not getting the CEV of humanity even with aligned AI
I agree. I defended almost this exact same thesis too in a recent post.
In keeping with this long tradition of human selfishness, it seems obvious that, if we succeed at aligning AI, the vast, vast majority of its output will get directed toward satisfying the preferences and values of the people controlling it (or possessing leverage over its continued operation) not the “CEV of all humans”, let alone the “CEV of all extant moral persons”
I agree with this part too. But I’d add that the people who “control” AIs won’t necessarily be the people who build them. Mostly I think AI values will be determined by a variety of forces—including AI developers—but mostly market and regulatory forces outside of the control of AI developers. As I said in another recent post,
I don’t think that “the team of humans that succeeds in building the first AGI” will likely be the primary force in the world responsible for shaping the values of future AIs. Instead, I think that (1) there isn’t likely to be a “first AGI” in any meaningful sense, and (2) AI values will likely be shaped more by market forces and regulation than the values of AI developers, assuming we solve the technical problems of AI alignment.
In general, companies usually cater to what their customers want, and when they don’t do that, they’re generally outcompeted by companies who will do what customers want instead. Companies are also heavily constrained by laws and regulations. I think these constraints—market forces and regulation—will apply to AI companies too. Indeed, we have already seen these constraints play a role shaping the commercialization of existing AI products, such as GPT-4. It seems best to assume that this situation will largely persist into the future, and I see no strong reason to think there will be a fundamental discontinuity with the development of AGI.
In the longer term, I expect even “aligned AI” values will evolve outside the bounds of human intentions, but this won’t necessarily be bad for humans if we can stay rich—kept afloat by strong property rights and respect for the rule of law—even as our values decline in relative influence with respect to the rest of the universe.
I agree with this part too. But I’d add that the people who “control” AIs won’t necessarily be the people who build them.
I agree, I used the general term to avoid implying necessarily that OpenAI et. al. will get to decide, though I think the implicit goal of most AGI developers is to get as much control over the lightcone as possible and that deliberately working towards that particular goal counts for a lot.
I think the implicit goal of most AGI developers is to get as much control over the lightcone as possible and that deliberately working towards that particular goal counts for a lot.
That seems right. I’d broaden this claim a bit: most people in general, want to be rich, i.e. “get control over the lightcone”. People vary greatly in their degree of rapaciousness, and how hard they work to become rich, but to a first approximation, people really do care a lot about earning a high income. For example, most people are willing to work ~40 hours a week for ~40 years of their life even though a modern wage in a developed country is perfectly capable of sustaining life at a fraction of the cost in time.
I agree. I defended almost this exact same thesis too in a recent post.
I agree with this part too. But I’d add that the people who “control” AIs won’t necessarily be the people who build them. Mostly I think AI values will be determined by a variety of forces—including AI developers—but mostly market and regulatory forces outside of the control of AI developers. As I said in another recent post,
In the longer term, I expect even “aligned AI” values will evolve outside the bounds of human intentions, but this won’t necessarily be bad for humans if we can stay rich—kept afloat by strong property rights and respect for the rule of law—even as our values decline in relative influence with respect to the rest of the universe.
I agree, I used the general term to avoid implying necessarily that OpenAI et. al. will get to decide, though I think the implicit goal of most AGI developers is to get as much control over the lightcone as possible and that deliberately working towards that particular goal counts for a lot.
That seems right. I’d broaden this claim a bit: most people in general, want to be rich, i.e. “get control over the lightcone”. People vary greatly in their degree of rapaciousness, and how hard they work to become rich, but to a first approximation, people really do care a lot about earning a high income. For example, most people are willing to work ~40 hours a week for ~40 years of their life even though a modern wage in a developed country is perfectly capable of sustaining life at a fraction of the cost in time.