I’m very glad that this was written. It exceeded my expectations of OpenAI. One small problem that I have not seen anyone else bring up:
”We want AGI to empower humanity to maximally flourish in the universe.”
If this type of language ends up informing the goals of an AGI, we could see some problems here. In general, we probably won’t want our agentic AI’s to be maximizers for anything, even if it sounds good. Even in the best case scenario where this really does cause humanity to flourish in a way that we would recognize as such, what about when human flourishing necessitates the genocide of less advanced alien life in the universe?
Truth be told, I’m actually sort of fine. That’s because right now we have to focus, and not get distracted by neat side goals, and whilst I expect it to be imperfect, right now I just want to care about the alignment problem right now and put off the concerns of technical alignment and maximization for later.
I understand that perspective, but I think it’s a small cost to Sam to change the way he’s framing his goals. Small nudge now, to build good habits for when specifying goals becomes, not just important, but the most important thing in all of human history.
I’m very glad that this was written. It exceeded my expectations of OpenAI. One small problem that I have not seen anyone else bring up:
”We want AGI to empower humanity to maximally flourish in the universe.”
If this type of language ends up informing the goals of an AGI, we could see some problems here. In general, we probably won’t want our agentic AI’s to be maximizers for anything, even if it sounds good. Even in the best case scenario where this really does cause humanity to flourish in a way that we would recognize as such, what about when human flourishing necessitates the genocide of less advanced alien life in the universe?
Truth be told, I’m actually sort of fine. That’s because right now we have to focus, and not get distracted by neat side goals, and whilst I expect it to be imperfect, right now I just want to care about the alignment problem right now and put off the concerns of technical alignment and maximization for later.
I understand that perspective, but I think it’s a small cost to Sam to change the way he’s framing his goals. Small nudge now, to build good habits for when specifying goals becomes, not just important, but the most important thing in all of human history.