“Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won’t want the differential to be zero, it will just compute it.”
How can a computation arrive at a nonzero differential, starting with zero data? If I ask a rational AI to calculate the probability of me typing “QWERTYUIOP” saving 3^^^^3 human lives, it knows literally nothing about the causal interactions between me and those lives, because they are totally unobservable.
“Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won’t want the differential to be zero, it will just compute it.”
How can a computation arrive at a nonzero differential, starting with zero data? If I ask a rational AI to calculate the probability of me typing “QWERTYUIOP” saving 3^^^^3 human lives, it knows literally nothing about the causal interactions between me and those lives, because they are totally unobservable.