The question is not only about their position in mindspace. Surely there may be as many possible minds (not just humans) which believe they can simulate 3^^^3 people and torture them as there those that do not believe so. But this does not mean that there are as many possible minds which actually could really do it. So I shouldn’t use a maximum entropy prior for my belief in their ability to do it, but for my belief in their belief in their ability to do it!
The question is not only about their position in mindspace. Surely there may be as many possible minds (not just humans) which believe they can simulate 3^^^3 people and torture them as there those that do not believe so. But this does not mean that there are as many possible minds which actually could really do it. So I shouldn’t use a maximum entropy prior for my belief in their ability to do it, but for my belief in their belief in their ability to do it!