It seems like the simplest algorithm that makes good predictions and runs on your computer is going to involve e.g. reasoning about what aspects of reality are important to making good predictions and then attending to those. But that doesn’t mean that I think reality probably works that way. So I don’t see how to salvage this kind of argument.
I think it works differently. What you should get is an infra-Bayesian hypothesis which models only those parts of reality that can be modeled within the given computing resources. More generally, if you don’t endorse the predictions of the prediction algorithm than either you are wrong or you should use a different prediction algorithm.
How the can the laws of physics be extra-compressible within the context of a simulation hypothesis? More compression means more explanatory power. I think that is must look something like, we can use the simulation hypothesis to predict the values of some of the physical constants. But, it would require a very unlikely coincidence for physical constants to have such values unless we are actually in a simulation.
It seems to me like this requires a very strong match between the priors we write down and our real priors. I’m kind of skeptical about that a priori, but then in particular we can see lots of ways in which attackers will be exploiting failures in the prior we write down (e.g. failure to update on logical facts they observe during evolution, failure to make the proper anthropic update, and our lack of philosophical sophistication meaning that we write down some obviously “wrong” universal prior).
I agree that we won’t have a perfect match but I think we can get a “good enough” match (similarly to how any two UTMs that are not too crazy give similar Solomonoff measures.) I think that infra-Bayesianism solves a lot of philosophical confusions, including anthropics and logical uncertainty, although some of the details still need to be worked out. (But, I’m not sure what specifically do you mean by “logical facts they observe during evolution”?) Ofc this doesn’t mean I am already able to fully specify the correct infra-prior: I think that would take us most of the way to AGI.
Do we have any idea how to write down such an algorithm though?
I have all sorts of ideas, but still nowhere near the solution ofc. We can do deep learning while randomizing initial conditions and/or adding some noise to gradient descent (e.g. simulated annealing), producing a population of networks that progresses in an evolutionary way. We can, for each prediction, train a model that produces the opposite prediction and compare it to the default model in terms of convergence time and/or weight magnitudes. We can search for the algorithm using meta-learning. We can do variational Bayes with a “multi-modal” model space: mixtures of some “base” type of model. We can do progressive refinement of infra-Bayesian hypotheses, s.t. the plausible hypotheses at any given moment are the leaves of some tree.
moreover it’s not clear to me the malign hypothesis faces a similar version of this problem since it’s just thinking about a small list of hypotheses rather than trying to maintain a broad enough distribution to find all of them
Well, we also don’t have to find all of them: we just have to make sure we don’t miss the true one. So, we need some kind of transitivity: if we find a hypothesis which itself finds another hypothesis (in some sense) then we also find the other hypothesis. I don’t know how to prove such a principle, but it doesn’t seem implausible that we can.
it may just be reasoning deductively about properties of the space of hypotheses rather than using a simple algorithm we can write down.
Why do you think “reasoning deductively” implies there is no simple algorithm? In fact, I think infra-Bayesian logic might be just the thing to combine deductive and inductive reasoning.
I think it works differently. What you should get is an infra-Bayesian hypothesis which models only those parts of reality that can be modeled within the given computing resources. More generally, if you don’t endorse the predictions of the prediction algorithm than either you are wrong or you should use a different prediction algorithm.
How the can the laws of physics be extra-compressible within the context of a simulation hypothesis? More compression means more explanatory power. I think that is must look something like, we can use the simulation hypothesis to predict the values of some of the physical constants. But, it would require a very unlikely coincidence for physical constants to have such values unless we are actually in a simulation.
I agree that we won’t have a perfect match but I think we can get a “good enough” match (similarly to how any two UTMs that are not too crazy give similar Solomonoff measures.) I think that infra-Bayesianism solves a lot of philosophical confusions, including anthropics and logical uncertainty, although some of the details still need to be worked out. (But, I’m not sure what specifically do you mean by “logical facts they observe during evolution”?) Ofc this doesn’t mean I am already able to fully specify the correct infra-prior: I think that would take us most of the way to AGI.
I have all sorts of ideas, but still nowhere near the solution ofc. We can do deep learning while randomizing initial conditions and/or adding some noise to gradient descent (e.g. simulated annealing), producing a population of networks that progresses in an evolutionary way. We can, for each prediction, train a model that produces the opposite prediction and compare it to the default model in terms of convergence time and/or weight magnitudes. We can search for the algorithm using meta-learning. We can do variational Bayes with a “multi-modal” model space: mixtures of some “base” type of model. We can do progressive refinement of infra-Bayesian hypotheses, s.t. the plausible hypotheses at any given moment are the leaves of some tree.
Well, we also don’t have to find all of them: we just have to make sure we don’t miss the true one. So, we need some kind of transitivity: if we find a hypothesis which itself finds another hypothesis (in some sense) then we also find the other hypothesis. I don’t know how to prove such a principle, but it doesn’t seem implausible that we can.
Why do you think “reasoning deductively” implies there is no simple algorithm? In fact, I think infra-Bayesian logic might be just the thing to combine deductive and inductive reasoning.