Konrad: In computational terms, you can’t avoid using a ‘hack’. Maybe not the hack you described, but something, somewhere has to be hard-coded.
Well, yes. The alternative to code is not solipsism, but a rock, and even a rock can be viewed as being hard-coded as a rock. But we would prefer that the code be elegant and make sense, rather than using a local patch to fix specific problems as they come to mind, because the latter approach is guaranteed to fail if the AI becomes more powerful than you and refuses to be patched.
Andrew: You’re saying that your priors have to come from some rigorous procedure
The priors have to come from some computable procedure. We would prefer it to be a good one, as agents with nonsense priors will not attain sensible posteriors.
but your utility comes from simply transcribing what some dude says to you.
No. Certain hypothetical scenarios, which we describe using the formalism of Turing machines, have fixed utilities—that is, if some description of the universe is true, it has a certain utility.
The problem with this scenario is not that we believe everything the dude tells us. The problem is that the description of a certain very large universe with a very large utility, does not have a correspondingly tiny prior probability if we use Solmonoff’s prior. And then as soon as we see any evidence, no matter how tiny, anything whose entanglement is not as tiny as the very large universe is large, that expected utility differential instantly wipes out all other factors in our decision process.
Second, even if for some reason you really want to work with the utility of 3^^^^3, there’s no good reason for you not to consider the possibility that it’s really −3^^^^3, and so you should be doing the opposite.
A Solomonoff inductor might indeed consider it, though there’s the problem of any bounded rationalist not being able to consider all computations. It seems “reasonable” for a bounded mind to consider it here; you did, after all.
The issue is not that two huge numbers will exactly cancel out; the point is that you’re making up all the numbers here but are artificially constraining the expected utility differential to be positive.
Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won’t want the differential to be zero, it will just compute it.
Konrad: In computational terms, you can’t avoid using a ‘hack’. Maybe not the hack you described, but something, somewhere has to be hard-coded.
Well, yes. The alternative to code is not solipsism, but a rock, and even a rock can be viewed as being hard-coded as a rock. But we would prefer that the code be elegant and make sense, rather than using a local patch to fix specific problems as they come to mind, because the latter approach is guaranteed to fail if the AI becomes more powerful than you and refuses to be patched.
Andrew: You’re saying that your priors have to come from some rigorous procedure
The priors have to come from some computable procedure. We would prefer it to be a good one, as agents with nonsense priors will not attain sensible posteriors.
but your utility comes from simply transcribing what some dude says to you.
No. Certain hypothetical scenarios, which we describe using the formalism of Turing machines, have fixed utilities—that is, if some description of the universe is true, it has a certain utility.
The problem with this scenario is not that we believe everything the dude tells us. The problem is that the description of a certain very large universe with a very large utility, does not have a correspondingly tiny prior probability if we use Solmonoff’s prior. And then as soon as we see any evidence, no matter how tiny, anything whose entanglement is not as tiny as the very large universe is large, that expected utility differential instantly wipes out all other factors in our decision process.
Second, even if for some reason you really want to work with the utility of 3^^^^3, there’s no good reason for you not to consider the possibility that it’s really −3^^^^3, and so you should be doing the opposite.
A Solomonoff inductor might indeed consider it, though there’s the problem of any bounded rationalist not being able to consider all computations. It seems “reasonable” for a bounded mind to consider it here; you did, after all.
The issue is not that two huge numbers will exactly cancel out; the point is that you’re making up all the numbers here but are artificially constraining the expected utility differential to be positive.
Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won’t want the differential to be zero, it will just compute it.