The best “priors” about the universe are 1 for what that universe right around you is, and 0 for everything else. Other priors are a compromise, an engineering decision.
What I am thinking is that
there is a considerably better way to assign priors which we do not know of yet—the way which will assign equal probabilities to each side of a die if it has no reason to prefer one over the other—the way that does correspond to symmetries in the evidence.
We don’t know that there will still be same problem when we have a non-stupid way to assign priors (especially as the non-stupid way ought to be considerably more symmetric). And it may be that some value systems are intrinsically incoherent. Suppose you wanted to maximize blerg without knowing what blerg even really is. That wouldn’t be possible, you can’t maximize something without having a measure of it. But I still can tell you i’d give you 3^^^^3 blergs for a dollar, without either of us knowing what blerg is supposed to be or whenever 3^^^^3 blergs even make sense (if blerg is an unique good book of up to 1000 page length, it doesn’t because duplicates aren’t blerg).
The best “priors” about the universe are 1 for what that universe right around you is, and 0 for everything else. Other priors are a compromise, an engineering decision.
What I am thinking is that
there is a considerably better way to assign priors which we do not know of yet—the way which will assign equal probabilities to each side of a die if it has no reason to prefer one over the other—the way that does correspond to symmetries in the evidence.
We don’t know that there will still be same problem when we have a non-stupid way to assign priors (especially as the non-stupid way ought to be considerably more symmetric). And it may be that some value systems are intrinsically incoherent. Suppose you wanted to maximize blerg without knowing what blerg even really is. That wouldn’t be possible, you can’t maximize something without having a measure of it. But I still can tell you i’d give you 3^^^^3 blergs for a dollar, without either of us knowing what blerg is supposed to be or whenever 3^^^^3 blergs even make sense (if blerg is an unique good book of up to 1000 page length, it doesn’t because duplicates aren’t blerg).