Eliezer has written about using the length of the program required to produce it, but this doesn’t seem to be unique; you could have languages that are very efficient for one thing, but long-winded for another. And quantum computing seems to make it even more confusing.
The method that Eliezer is referring to is known as Solomonoff induction which relies on programs as defined by Turing machines. Quantum computing doesn’t come into this issue since these formulations just talk about length of specification, not efficiency of computation. There are theorems that also show that for any given Turing complete well-behaved language, the minimum size of program can’t be differ by more than a constant. So changing the language won’t alter the priors other than a fixed amount. Taken together with Aumann’s Agreement Theorem, the level of disagreement about estimated probability should go to zero in the limiting case (disclaimer I haven’t seen a proof of that last claim, but I suspect it would be a consequence of using a Solomonoff style system for your priors).
Do we have a unique method for generating priors?
Eliezer has written about using the length of the program required to produce it, but this doesn’t seem to be unique; you could have languages that are very efficient for one thing, but long-winded for another. And quantum computing seems to make it even more confusing.
The method that Eliezer is referring to is known as Solomonoff induction which relies on programs as defined by Turing machines. Quantum computing doesn’t come into this issue since these formulations just talk about length of specification, not efficiency of computation. There are theorems that also show that for any given Turing complete well-behaved language, the minimum size of program can’t be differ by more than a constant. So changing the language won’t alter the priors other than a fixed amount. Taken together with Aumann’s Agreement Theorem, the level of disagreement about estimated probability should go to zero in the limiting case (disclaimer I haven’t seen a proof of that last claim, but I suspect it would be a consequence of using a Solomonoff style system for your priors).