Optimization power is the source of the danger, not agency. Agents merely wield optimality to achieve their goals.
Agency is orthogonal to optimization power
@All: It seems we agreethat optimality, when pursued blindly, is about extreme optimization that can lead to dangerous outcomes.
Could it be that we are overlooking the potential for a (superintelligent) system to prioritize what matters moreâthe effectiveness of a decisionârather than simply optimizing for a single goal? đ¤
For example, optimizing too much for a single goal (getting the most paperclips) might overlook ethical or long-term considerations which may contribute to the greater good for all Beings.
Final question: Under what circumstances might you prefer a (superintelligent) system to reject the paperclip request and suggest alternative solutions, or seek to understand the requesterâs underlying needs and motivations?
I would love to hear additional comments or feedback on when to prioritize effectiveness, as I am still trying to understand decision-making better đ¤
Fundamentally, the story was about the failure cases of trying to make capable systems that donât share your values safe by preventing specific means by which its problem solving capabilities express themselves in scary ways. This is different to what you are getting at here, which is having those systems actually operationally share your values. A well aligned system, in the traditional âFriendly AIâ sense of alignment, simply wonât make the choices that the one in the story did.
Thank you đ @mesaoptimizer for the summary!
@All: It seems we agree that optimality, when pursued blindly, is about extreme optimization that can lead to dangerous outcomes.
Could it be that we are overlooking the potential for a (superintelligent) system to prioritize what matters moreâthe effectiveness of a decisionârather than simply optimizing for a single goal? đ¤
For example, optimizing too much for a single goal (getting the most paperclips) might overlook ethical or long-term considerations which may contribute to the greater good for all Beings.
Final question:
Under what circumstances might you prefer a (superintelligent) system to reject the paperclip request and suggest alternative solutions, or seek to understand the requesterâs underlying needs and motivations?
I would love to hear additional comments or feedback on when to prioritize effectiveness, as I am still trying to understand decision-making better đ¤
Fundamentally, the story was about the failure cases of trying to make capable systems that donât share your values safe by preventing specific means by which its problem solving capabilities express themselves in scary ways. This is different to what you are getting at here, which is having those systems actually operationally share your values. A well aligned system, in the traditional âFriendly AIâ sense of alignment, simply wonât make the choices that the one in the story did.