Just to be clear, this isn’t in response to something I wrote, right? (I’m definitely not advocating any kind of “utilitarian TAI project” and would be quite scared of such a project myself.)
No! Sorry, if I gave that impression.
So what are you (and them) then? What would your utopia look like?
Well, I linked my toy model of partiality before. Are you asking about something more concrete?
I have low confidence about this, but my best guess personal utopia would be something like: A lot of cool and interesting things are happening. Some of them are good, some of them are bad (a world in which nothing bad ever happens would be boring). However, there is a limit on how bad something is allowed to be (for example, true death, permanent crippling of someone’s mind and eternal torture are over the line), and overall “happy endings” are more common than “unhappy endings”. Moreover, since it’s my utopia (according to my understanding of the question, we are ignoring the bargaining process and acausal cooperation here), I am among the top along those desirable dimensions which are zero-sum (e.g. play an especially important / “protagonist” role in the events to the extent that it’s impossible for everyone to play such an important role, and have high status to the extent that it’s impossible for everyone to have such high status).
No! Sorry, if I gave that impression.
Well, I linked my toy model of partiality before. Are you asking about something more concrete?
Yeah, I mean aside from how much you care about various other people, what concrete things do you want in your utopia?
I have low confidence about this, but my best guess personal utopia would be something like: A lot of cool and interesting things are happening. Some of them are good, some of them are bad (a world in which nothing bad ever happens would be boring). However, there is a limit on how bad something is allowed to be (for example, true death, permanent crippling of someone’s mind and eternal torture are over the line), and overall “happy endings” are more common than “unhappy endings”. Moreover, since it’s my utopia (according to my understanding of the question, we are ignoring the bargaining process and acausal cooperation here), I am among the top along those desirable dimensions which are zero-sum (e.g. play an especially important / “protagonist” role in the events to the extent that it’s impossible for everyone to play such an important role, and have high status to the extent that it’s impossible for everyone to have such high status).