M. Y. Zuo comments on How could you possibly choose what an AI wants?

[M. Y. Zuo]

… then it realizes that it wants to help in the quest of making the future wonderful and fun. That’s possible, but by no means guaranteed.

Possible based on what assumptions?