I’m not sure I understand the distinction you’re drawing between risk factors that compound the risks that you’re describing vs. different problems not related to intent alignment per se. It seems to me like “AI-powered economies have much higher economies of scale because AIs don’t suffer from the kind of coordination costs that humans have (e.g., they can merge their utility functions and become clones of each other)” is a separate problem from solving intent alignment, whereas “AI-powered memetic warfare makes all humans effectively insane” is kind of an extreme case of “machine learning will increase our ability to ‘get what we can measure’” which seems to be the opposite of how you classify them.
What do you think are the implications of something belonging to one category versus another (i.e., is there something we should do differently depending on which of these categories a risk factor / problem belongs to)?
I think the more general point is: if you think AI progress is likely to drive many of the biggest upcoming changes in the world, then there will be lots of risks associated with AI. Here I’m just trying to clarify what happens if we fail to solve intent alignment.
Ah, when I read “I think this is probably not what failure will look like” I interpreted that to mean “failure to prevent AI risk”, and then I missed the clarification “these are the most important problems if we fail to solve intent alignment” that came later in the post, in part because of a bug in GW that caused the post to be incorrectly formatted.
Aside from that, I’m worried about telling a vivid story about one particular AI risk, unless you really hammer the point that it’s just one risk out of many, otherwise it seems too easy for the reader to get that story stuck in their mind and come to think that this is the main or only thing they have to worry about as far as AI is concerned.
I’m not sure I understand the distinction you’re drawing between risk factors that compound the risks that you’re describing vs. different problems not related to intent alignment per se. It seems to me like “AI-powered economies have much higher economies of scale because AIs don’t suffer from the kind of coordination costs that humans have (e.g., they can merge their utility functions and become clones of each other)” is a separate problem from solving intent alignment, whereas “AI-powered memetic warfare makes all humans effectively insane” is kind of an extreme case of “machine learning will increase our ability to ‘get what we can measure’” which seems to be the opposite of how you classify them.
What do you think are the implications of something belonging to one category versus another (i.e., is there something we should do differently depending on which of these categories a risk factor / problem belongs to)?
Ah, when I read “I think this is probably not what failure will look like” I interpreted that to mean “failure to prevent AI risk”, and then I missed the clarification “these are the most important problems if we fail to solve intent alignment” that came later in the post, in part because of a bug in GW that caused the post to be incorrectly formatted.
Aside from that, I’m worried about telling a vivid story about one particular AI risk, unless you really hammer the point that it’s just one risk out of many, otherwise it seems too easy for the reader to get that story stuck in their mind and come to think that this is the main or only thing they have to worry about as far as AI is concerned.