The general reason Eliezer advocates not naming the highest virtue (as I understand it) is that there may be some type of problem for which bayesian updating (and the scoring rule referred to) yields the wrong answer. This idea sounds rather improbable to me, but there is a non-negligible probability that bayes will yield a wrong answer on some question. Not naming the virtue is supposed to be a reminder that if bayes ever gives the wrong answer, we go with the right answer, not bayes.
I think we’ve already found a type of problem where bayesian updating breaks. Namely, all the anthropic problems that UDT solves. (UDT doesn’t say that bayes gives the wrong answer in those cases, but it does say that asking for a probability is the wrong question.)
The general reason Eliezer advocates not naming the highest virtue (as I understand it) is that there may be some type of problem for which bayesian updating (and the scoring rule referred to) yields the wrong answer. This idea sounds rather improbable to me, but there is a non-negligible probability that bayes will yield a wrong answer on some question. Not naming the virtue is supposed to be a reminder that if bayes ever gives the wrong answer, we go with the right answer, not bayes.
I think we’ve already found a type of problem where bayesian updating breaks. Namely, all the anthropic problems that UDT solves. (UDT doesn’t say that bayes gives the wrong answer in those cases, but it does say that asking for a probability is the wrong question.)
This makes sense to me. I retract my claims.