Infra-Bayesian physicalism: a formal theory of naturalized induction

[Vanessa Kosoy]

This is joint work by Vanessa Kosoy and Alexander “Diffractor” Appel. For the proofs, see 1 and 2.

TLDR: We present a new formal decision theory that realizes naturalized induction. Our agents reason in terms of infra-Bayesian hypotheses, the domain of which is the cartesian product of computations and physical states, where the ontology of “physical states” may vary from one hypothesis to another. The key mathematical building block is the “bridge transform”, which, given such a hypothesis, extends its domain to “physically manifest facts about computations”. Roughly speaking, the bridge transforms determines which computations are executed by the physical universe. In particular, this allows “locating the agent in the universe” by determining on which inputs its own source is executed.

0. Background

The “standard model” of ideal agency is Bayesian reinforcement learning, and more specifically, AIXI. We challenged this model before due to its problems with non-realizability, suggesting infra-Bayesianism as an alternative. Both formalisms assume the “cartesian cybernetic framework”, in which (i) the universe is crisply divided into “agent” and “environment” and (ii) the two parts interact solely via the agent producing actions which influence the environment and the environment producing observations for the agent. This is already somewhat objectionable on the grounds that this division is not a clearly well-defined property of the physical universe. Moreover, once we examine the structure of the hypothesis such an agent is expected to learn (at least naively), we run into some concrete problems.

The modern understanding of the universe is that no observer plays a privileged role^[1]. Therefore, the laws of physics are insufficient to provide a cartesian description of the universe, and must, to this end, be supplemented with “bridge rules” that specify the agent’s location inside the universe. That is, these bridge rules need to translate the fundamental degrees of freedom of a physical theory (e.g. quantum wavefunction) to the agent’s observations (e.g. values of pixels on a camera), and translate the agent’s actions (e.g. signal to robot manipulators) in the other direction^[2]. The cost of this is considerable growth in the description complexity of the hypothesis.

A possible retort is something like “how many bits can it really take to pick out the computer within the universe state and describe how to translate universe state to observations? Sure, it might be a considerable chunk of data. But, since Solomonoff induction only needs to make a kilobyte worth of predictive mistakes to learn to predict the inputs as well as any thousand-bit predictor, it should learn this sort of stuff pretty fast.” This objection is addressed more later on in this post. A first counter-objection is that, for practical algorithms, this can be a bigger obstacle, especially when sample complexity scales superlinearly with description complexity. For example, the Russo-Van Roy regret bound for Thompson sampling in multi-armed bandits has the time horizon necessary to get a particular regret bound scale as the square of entropy, and the computational complexity cost can also go way up as the description complexity rises, because you need to evaluate more hypotheses more times to test them. For reinforcement learning it’s even worse, as in the case of a game where an agent must enter an n-digit password and if it gets it right, it gets reward, and if it gets it wrong, it does not get reward and can try again. The learning time for this game scales as $\Omega \left(2^n\right)$, far superlinearly with description complexity.

Another assumption of AIXI is the simplicity prior, and we expect some form of this assumption to persist in computable and infra-Bayesian analogues. This reflects the intuitive idea that we expect the world to follow simple laws (or at least contain simple patterns). However, from the cartesian perspective, the “world” (i.e. the environment) is, prima facie, not simple at all (because of bridge rules)! Admittedly, the increase in complexity from the bridge rule is low compared to the cost of specifying the universe state, but once the agent learns the transition rules and bridge rule for the universe it’s in, learning the state of the universe in addition doesn’t seem to yield any particular unforeseen metaphysical difficulties. Further, the description complexity cost of the bridge rule seems likely to be above the description complexity of the laws of physics.

Hence, there is some disconnect between the motivation for using a simplicity prior and its implementation in a cartesian framework.

Moreover, if the true hypothesis is highly complex, it implies that the sample complexity of learning it is very high. And, as previously mentioned, the sample complexity issues are worse in practice than Solomonoff suggests. This should make us suspect that such a learning process is not properly exploiting Occam’s razor. Intuitively, such an agent a-priori considers it equally plausible to discover itself to be a robot or to discover itself to be a random clump of dust in outer space, since it’s about as hard to specify a bridge rule interface between the computer and observations as it is to specify a bridge rule interface between the dust clump and observations and needs a lot of data to resolve all those possibilities for how its observations connect to a world. Also, though Solomonoff is extremely effective at slicing through the vast field of junk hypotheses that do not describe the thing being predicted, once it’s whittled things down to a small core of hypotheses that do predict things fairly accurately, the data to further distinguish between them may be fairly slow in coming. If there’s a simple predictor of events occurring in the world but it’s running malign computation, then you don’t have the luxury of 500 bits of complexity wiggle room (to quickly knock out this hypothesis), because that’s a factor of $2^{500}$ probability difference. Doing worst-case hypothesis testing as in KWIK learning would require a very aggressive threshold indeed, and mispredictions can be rare but important.

Furthermore, some events are simple to describe from the subjective (cartesian) point of view, but complex to describe from an objective (physical) point of view. (For example, all the pixels of the camera becoming black.) Modifying a hypothesis by positing exceptional behavior following a simple event only increases its complexity by the difficulty of specifying the event and what occurs afterwards, which could be quite low. Hence, AIXI-like agents would have high uncertainty about the consequences of observationally simple events. On the other hand, from an objective perspective such uncertainty seems irrational. (Throwing a towel on the camera should not break physics.) In other words, cartesian reasoning is biased to privilege the observer.

Yet another failure of cartesian agents is the inability to reason about origin theories. When we learn that a particular theory explains our own existence (e.g. evolution), this serves as a mark in favor of the theory. We can then exploit this theory to make useful predictions or plans (e.g. anticipate that using lots of antibiotics will cause bacteria to develop resistance). However, for a cartesian agent the question of origins is meaningless. Such an agent perceives its own existence as axiomatic, hence there is nothing to explain.

Finally, cartesian agents are especially vulnerable to acausal attacks. Suppose we deploy a superintelligent Cartesian AI called Kappa. And, imagine a superintelligent agent Mu that inhabits some purely hypothetical universe. If Mu is motivated to affect our own (real) universe, it can run simulations of Kappa’s environment. Kappa, who doesn’t know a priori whether it exists in our universe or in Mu’s universe, will have to seriously consider the hypothesis it is inside such a simulation. And, Mu will deploy the simulation in such manner as to make the simulation hypothesis much simpler, thanks to simpler bridge rules. This will cause Kappa to become overwhelmingly confident that it is in a simulation. If this is achieved, Mu can cause the simulation to diverge from our reality in a strategically chosen point such that Kappa is induced to take an irreversible action in Mu’s favor (effectively a treacherous turn). Of course, this requires Kappa to predict Mu’s motivations in some detail. This is possible if Kappa develops a good enough understanding of metacosmology.

An upcoming post by Diffractor will discuss acausal attacks in more detail.

In the following sections, we will develop a “physicalist” formalism that entirely replaces the cartesian framework, curing the abovementioned ills, though we have not yet attained the stage of proving improved regret bounds with it, just getting the basic mathematical properties of it nailed down. As an additional benefit, it allows naturally incorporating utility functions that depend on unobservables, thereby avoiding the problem of “ontological crises”. At the same time, it seems to impose some odd constraints on the utility function. We discuss the possible interpretations of this.

1. Formalism

Notation

It will be more convenient to use ultradistributions rather than infradistributions. This is a purely notational choice: the decision theory is unaffected, since we are going to apply these ultradistributions to loss functions rather than utility functions. As support for this claim, Diffractor originally wrote down most of the proofs in infradistribution form, and then changing their form for this post was rather straightforward to do. In addition, for the sake of simplicity, we will stick to finite sets: more general spaces will be treated in a future article. So far, we’re up to countable products of finite sets.

We denote $R_{+}:=[0,\infty )$. Given a finite set $X$, a contribution on $X$ is $\theta :X\to R_{+}$ s.t. ${\sum }_x\theta \left(x\right)\le 1$ (it’s best to regard it as a measure on $X$). The space of contributions is denoted ${\Delta }^{c}X$. Given $f:X\to R$ and $\theta \in {\Delta }^{c}X$, we denote $\theta \left(f\right):={\sum }_x\theta \left(x\right)f\left(x\right)$. There is a natural partial order on contributions: ${\theta }_1\le {\theta }_2$ when $\forall x\in X:{\theta }_1\left(x\right)\le {\theta }_2\left(x\right)$. Naturally, any distribution is in particular a contribution, so $\Delta X\subseteq {\Delta }^{c}X$. A homogenous ultracontribution (HUC) on $X$ is non-empty closed convex $\Theta \subseteq {\Delta }^{c}X$ which is downward closed w.r.t. the partial order on ${\Delta }^{c}X$. The space of HUCs on $X$ is denoted ${\square }^{c}X$. A homogenous ultradistribution (HUD) on $X$ is a HUC $\Theta$ s.t. $\Theta \cap \Delta X\ne \varnothing$. The space of HUDs on $X$ is denoted $\square X$. Given $f:X\to R$ and $\Theta \in {\square }^{c}X$, we denote $\Theta \left(f\right):={max}_{\theta \in \Theta }\theta \left(f\right)$.

Given $s:X\to Y$, $s_{\ast }:{\square }^{c}X\to {\square }^{c}Y$ is the pushforward by $s$:

$$s_{\ast }\Theta :=\{s_{\ast }\theta \mid \theta \in \Theta \}$$

$$\left(s_{\ast }\theta \right)\left(y\right):=\sum _{x\in s^{-1}\left(y\right)}\theta \left(x\right)$$

Given $\Xi :X\to {\square }^{c}Y$, ${\Xi }_{\ast }:{\square }^{c}X\to {\square }^{c}Y$ is the pushforward by $\Xi$:

$${\Xi }_{\ast }\Theta :=\{{\kappa }_{\ast }\theta \mid \theta \in \Theta ,\kappa :X\to {\Delta }^{c}Y,\forall x\in X:\kappa (x)\in \Xi (x\left)\right\}$$

$$\left({\kappa }_{\ast }\theta \right)\left(y\right):=\sum _{x\in X}\theta \left(x\right)\kappa (x;y)$$

${pr}_X:X\times Y\to X$ is the projection mapping and ${pr}_{-Y}:={pr}_X$. We slightly abuse notation by omitting the asterisk in pushforwards by these.

Given $\Theta \in {\square }^{c}X$ and $\Xi :X\to {\square }^{c}Y$, $\Theta ⋉\Xi \in {\square }^c(X\times Y)$ is the semidirect product:

$$\Theta ⋉\Xi :=\{\kappa ⋉\theta \mid \theta \in \Theta ,\kappa :X\to {\Delta }^{c}Y,\forall x\in X:\kappa (x)\in \Xi (x\left)\right\}$$

$$(\kappa ⋉\theta )(x,y):=\theta \left(x\right)\kappa (x;y)$$

We will also use the notation $\Xi ⋊\Theta \in {\square }^c(Y\times X)$ for the same HUC with $X$ and $Y$ flipped. And, for $\Lambda \in {\square }^{c}Y$, $\Theta ⋉\Lambda \in {\square }^c(X\times Y)$ is the semidirect product of $\Theta$ with the constant ultrakernel whose value is $\Lambda$^[3].

For more discussion of HUDs, see previous article, where we used the equivalent concept of “cohomogenous infradistribution”.

Notation Reference

If you got lost somewhere and wanted to scroll back to see some definition, or see how the dual form of this with infradistributions works, that’s what this section is for.

$\theta$ is a contribution, a measure with 1 or less measure in total. The dual concept is an a-measure $(\lambda \mu ,b)$ with $\lambda +b=1$.

$\Theta$ is a HUC (homogenous ultracontribution) or HUD (homogenous ultradistribution), a closed convex downwards closed set of contributions. The dual concepts are cohomogenous inframeasure and cohomogenous infradistribution, respectively.

${\Delta }^{c}X,{\square }^{c}X,\square X$ are the spaces of contributions, homogenous ultracontributions, and homogenous ultradistributions respectively.

$\theta \left(f\right),\Theta \left(f\right)$ are the expectations of functions $f:X\to [0,1]$, defined in the usual way. For $\theta$, it’s just the expectation of a function w.r.t. a measure, and for $\Theta$, it’s $\Theta \left(f\right):={max}_{\theta \in \Theta }\theta \left(f\right)$, to perfectly parallel a-measures evaluating functions by just taking expectation, and inframeasures evaluating functions via ${min}_{(m,b)\in \Psi }m\left(f\right)+b$.

$\le$ is the ordering on contributions/​HUC’s/​HUD’s, which is the function ordering, where $\theta \le {\theta }^{\prime }$ iff for all $f:X\to [0,1]$, $\theta \left(f\right)\le {\theta }^{\prime }\left(f\right)$, and similar for the HUC’s. Inframeasures are equipped with the opposite ordering.

$s_{\ast }\Theta$ is the pushforward along the function $s:X\to Y$. This is a standard probability theory concept which generalizes to all the infra and ultra stuff.

${\Xi }_{\ast }\Theta$ is the pushforward of $\Theta$ along the ultrakernel $\Xi :X\to {\square }^{c}Y$. This is just the generalization to infra and ultra stuff of the ability to push a probability distribution on $X$ through a probabilistic function $X\to \Delta Y$ to get a probability distribution on $Y$.

$\Theta ⋉\Xi$ is the semidirect product of $\Theta$ and $\Xi$, an element of ${\square }^c(X\times Y)$. This is the generalization of the ability to take a probability distribution on $X$, and a probabilistic kernel $X\to \Delta Y$, and get a joint distribution on $X\times Y$.

$A,O$ are the set of actions and observations.

$N$ is the time horizon.

$R$ is the space of programs, $\Sigma$ is the space of outputs.

$\Gamma ={\Sigma }^R$ is the space of functions from programs to the results they output. An element of $\Gamma$ can be thought of as the state of the “computational universe”, for it specifies what all the programs do.

$H$ is the space of histories, action-observation sequences that can end at any point, ending with an observation.

$D$ is the space of destinies, action-observation sequences that are as long as possible, going up to the time horizon.

$C$ is a relation on $\Gamma \times D$ that says whether a computational universe is consistent with a given destiny. A very important note is that this is not the same thing as “the policy is consistent with the destiny” (the policy’s actions are the same as what the destiny advises). This is saying something more like “if the destiny has an observation that the computer spat out result 1 when run on computation A, then that is only consistent with mathematical universes which have computation A outputting result 1″. Except we don’t want to commit to the exact implementation details of it, so we’re leaving it undefined besides just “it’s a relation”

$\Phi$ is the space of “physics outcomes”, it can freely vary depending on the hypothesis. It’s not a particular fixed space.

$\Theta$ is the variable typically used for physicalist hypotheses, elements of $\square (\Gamma \times \Phi )$. Your uncertainty over the joint distribution over the computational universe and the physical universe.

$G$ is the code of the program-which-is-the-agent. So, $G\left(h\right)$ would be the computation that runs what the agent does on history $h$, and returns its resulting action.

${\text{el}}^{\Gamma }$ is the subset of the space $\Gamma \times 2^{\Gamma }$ consisting of $(y,\alpha )$ pairs s.t. $y\in \alpha$. The $y$ can be thought of as the mathematical universe, and $\alpha$ can be thought of as the set of mathematical universes that are observationally indistinguishable from it.

${\chi }_A$ is the indicator function that’s 1 on the set $A$, and 0 everywhere else. It can be multiplied by a measure, to get the restriction of a measure to a particular set.

$\text{Br}\left(\Theta \right)$ is the bridge transform of $\Theta$, defined in definition 1.1, an ultracontribution over the space ${\text{el}}^{\Gamma }\times \Phi$.

$H_{y\alpha }$ is the set of instantiated histories, relative to mathematical universe $y$, and set of observationally indistinguishable universes $\alpha$. It’s the set of histories $h$ where all the “math universes” in $\alpha$ agree on how the agent’s source code reacts to all the prefixes of $h$, and where the history $h$ can be extended to some destiny that’s consistent with math universe $y$. Ie, for a history to be in here, all the prefixes have to be instantiated, and it must be consistent with the selected math universe.

Setting

As in the cartesian framework, we fix a finite set $A$ of actions and a finite set $O$ of observations. We assume everything happens within a fixed finite^[4] time horizon $N\in N.$ We assume that our agent has access to a computer^[5] on which it can execute some finite^[4:1] set of programs $R$ with outputs in a finite alphabet $\Sigma$. Let $\Gamma :={\Sigma }^R$ be the set of “possible computational universes”^[6]. We denote $H:=(A\times O{)}^{<N}$ (the set of histories) and $D:=(A\times O{)}^{N-1}\times A$ (the set of “destinies”).

To abstract over the details of how the computer is operated, we assume a relation $C\subseteq D\times \Gamma$ whose semantics is, $dCy$ (our notation for $(d,y)\in C$) if and only if destiny $d$ is a priori consistent with computational universe $y$. For example, suppose some $a\in A$ implies a command to execute program $r\in R,$and if $o\in O$ follows $a$, it implies observing the computer return output $i\in \Sigma$ for $r$. Then, if $d$ contains the substring $ao$ and $dCy$, it must be the case that $y\left(r\right)=i$.

A physicalist hypothesis is a pair $(\Phi ,\Theta$), where $\Phi$ is a finite^[4:2] set representing the physical states of the universe and $\Theta \in \square (\Gamma \times \Phi )$ represents a joint belief about computations and physics. By slight abuse of notation we will refer to such $\Theta$ as a physicalist hypothesis, understanding $\Phi$ to be implicitly specified. Our agent will have a prior over such hypotheses, ranging over different $\Phi$.

Two questions stand out to us at this point. The first is, what is the domain over which our loss function should be defined? The second is, how do we define the counterfactuals corresponding to different policies $\pi :H\to A$? The answers to both questions turn out to require the same mathematical building block.

For the first question, we might be tempted to identify $\Phi$ as our domain. However, prima facie this doesn’t make sense, since $\Phi$ is hypothesis-dependent. This is the ontological crisis problem: we expect the agent’s values to be defined within a certain ontology which is not the best ontology for formulating the laws of the physical universe. For example, a paperclip maximizer might benefit from modeling the universe in terms of quantum fields rather than paperclips. In principle, we can circumvent this problem by requiring our $\Phi$ to be equipped with a mapping $\nu :\Phi \to {\Phi }_0,$where ${\Phi }_0$ is the “axiological” ontology. However, this $\nu$ is essentially a bridge rule, carrying with it all the problems of bridge rules: acausal attack is performed by the adversarial hypothesis imprinting the “axiological rendering” of the target universe on the microscopic degrees of freedom of the source universe in order to have a low-complexity $\Phi \to {\Phi }_0$ function; the analogue of the towel-on-camera issue is that, once you’ve already coded up your uncertainty over math and physics, along with how to translate from physics to the ontology of value, it doesn’t take too much extra complexity to tie “axiologically-simple” results (the analogue of low-complexity observations) to physics-simple results (the analogue of a low-complexity change in what happens), like “if all the paperclips are red, the fine-structure constant doubles in value”.

Instead, we will take a computationalist stance: value is a not property of physical states or processes, but of the computations realized by physical processes. For example, if our agent is “selfish” in the sense that, rewards/​losses are associated purely with subjective histories, the relevant computation is the agent’s own source code. Notice that, for the program-which-is-the-agent $G$, histories are input. Hence, given loss function $l:H\to R$ we can associate the loss $l\left(h\right)$ with the computation $G\left(h\right)$. Admittedly, there is an implicit assumption that the agent has access to its own source code, but modal decision theory made the same assumption. For another example, if our agent is a diamond maximizer, then the relevant computations are simulations of the physics used to define “diamonds”. A more concrete analogue of this is worked out in detail in section 3, regarding Conway’s Game of Life.

For the second question, we might be tempted to follow updateless decision theory: counterfactuals correspond to conditioning on $G=\pi$. Remember, G is the code of the agent. However, this is not “fair” since it requires the agent to be “responsible” for copies of itself instantiated with fake memories. Such a setting admits no learning-theoretic guarantees, since learning requires trusting your own memory. (Moreover, the agent also has to be able to trust the computer.) Therefore our counterfactuals should only impose $G\left(h\right)=\pi \left(h\right)$ when $h$ is a “real memory”, which we against interpret through computationalism: $h$ is real if and only if, $G\left(h^{\prime }\right)$ is physically realized for any prefix $h^{\prime }$ of $h$.

Both of our answers requires a formalization of the notion “assuming hypothesis $\Theta$, this computation is physically realized”. More precisely, we should allow for computations to be realized with certain probabilities, and more generally allow for ultradistributions over which computations are realized. We will now accomplish this formalization.

Bridge transform

Given any set $A$, we denote ${el}^A=\left\{\right(a,B)\in A\times 2^A\mid a\in B\}.$ $supp$ stands for “support” and ${\chi }_A$ is the characteristic function of $A$.

Definition 1.1: Let $\Gamma ,\Phi$ be finite sets and $\Theta \in {\square }^c(\Gamma \times \Phi )$. The bridge transform of $\Theta$ is ${Br}_{\Gamma }\left(\Theta \right)\in {\square }^c(\Gamma \times 2^{\Gamma }\times \Phi )$ s.t.$\theta \in {Br}_{\Gamma }\left(\Theta \right)$ if and only if

• $supp\theta \subseteq {el}^{\Gamma }\times \Phi$

• for any $s:\Gamma \to \Gamma$, ${pr}_{\Gamma \times \Phi }{\chi }_{{el}^{\Gamma }\times \Phi }(s\times {id}_{2^{\Gamma }\times \Phi }{)}_{\ast }\theta \in \Theta$.

We will use the notation $Br\left(\Theta \right)$ when $\Gamma$ is obvious from the context.

Notice that we are multiplying by ${\chi }_{{el}^{\Gamma }\times \Phi }$, not pushforwarding.

The $2^{\Gamma }$ variable of the bridge transform denotes the “facts about computations realized by physics”. In particular, if this $\alpha \in 2^{\Gamma }$ takes the form $\{y\in \Gamma \mid \forall r\in R_0:y(r)=y_0(r\left)\right\}$ for some $R_0\subseteq R$ and $y_0\in {\Sigma }^{R_0}$, then we may say that the computations $R_0$ are “realized” and the computations $R\setminus R_0$ are “not realized”. More generally, talking only about which computations are realized is imprecise since $\alpha$ might involve “partial realization” and/​or “entanglement” between computations (i.e. not be of the form above).

Intuitively, this definition expresses that the “computational universe” can be freely modified as long as the “facts known by physics” are preserved. However, that isn’t what originally motivated the definition. The bulk of its justification comes from its pleasing mathematical properties, discussed in the next section.

A physicalist agent should be equipped with a prior over physicalist hypotheses. For simplicity, suppose it’s a discrete Bayesian prior (it is straightforward to generalize beyond this): hypothesis ${\Theta }_i$ is assigned probability $\zeta \left(i\right)$ and ${\sum }_i\zeta \left(i\right)=1$. Then, we can consider the total bridge transform of the prior. It can’t be given by mixing the hypotheses together and applying the bridge transform, because every hypothesis has its own choice of $\Phi$, its own ontology, so you can’t mix them before applying the bridge transform. You have to apply the bridge transform to each component first, forget about the choice of $\Phi$ via projecting it out, and then mix them afterwards. This receives further support from Proposition 2.13 which takes an alternate possible way of defining the bridge transform for mixtures (extend all the hypotheses from $\Gamma \times {\Phi }_i$ to $\Gamma \times {⨆}_i{\Phi }_i$ in the obvious way so you can mix them first) and shows that it produces the same result.

Definition 1.2: $$Br(\zeta ⋉\Theta ):=\sum _i\zeta \left(i\right){pr}_{{el}^{\Gamma }}Br\left({\Theta }_i\right)\in \square {el}^{\Gamma }$$

Evaluating policies

Given $A\subseteq X$, ${\top }_A\in \square X$ is defined as $\{\theta \in {\Delta }^{c}X\mid supp\theta \subseteq A\}$. I.e., it’s total uncertainty over which point in $A$ will be selected.

We need to assume that $R$ contains programs representing the agent itself. That is, there is some $M\in N$, $dec:{\Sigma }^M\to A$ and for each $h\in H$, $┌G\left(h\right)┐\in R^M$. Pretty much, if you have large numbers of actions available, and a limited number of symbols in your language, actions (and the outputs of other programs with a rich space of outputs) can be represented be m-tuples of programs, like “what’s the first bit of this action choice” and “what’s the second bit of this action choice” and so on. So, $M$ is just how many bits you need, $dec$ is the mapping from program outputs to the actual action, and $┌G\left(h\right)┐$ is the m-tuple of of programs which implements the computation “what does my source code do on input history $h$”. The behavior of the agent in a particular mathematical universe is given by taking each program in $┌G\left(h\right)┐$, using the mathematical universe to figure out what each of the programs outputs, and then using $dec$ to convert that bitstring to an action.

Definition 1.3: Given $y\in \Gamma$ and $h\in H$, we define $G^y\left(h\right)\in A$ to be $dec\left(a\right)$ for $a\in {\Sigma }^M$ given by $a_i:=y(┌G(h\left){┐}_i\right)$.

We can now extract counterfactuals from any $\Lambda \in \square {el}^{\Gamma }.$ Specifically, given any policy $\pi$ we define some $C^{\pi }\subseteq {el}^{\Gamma }$ (the set of stuff consistent with behaving according to $\pi$, for a yet-to-be-defined notion of consistency) and define the counterfactual as $\Lambda \cap {\top }_{C^{\pi }}$. We could use the “naive” definition:

Definition 1.4: $C_{naive}^{\pi }$ is the set of all $(y,\alpha )\in {el}^{\Gamma }$ s.t. for any $h\in H$, $G^y\left(h\right)=\pi \left(h\right)$.

Per discussion above, it seems better to use a different definition. We will use the notation $h⊏g$ to mean ”$h$ is a proper prefix of $g$” and $h⊑g$ to mean ”$h$ is a prefix of $g$”,

Definition 1.5: Given $(y,\alpha )\in {el}^{\Gamma }$, let $H_{y\alpha }$ be the set of all $h\in H$ s.t. the following two conditions hold:

1. For all $ga\in H\times A$ and $y^{\prime }\in \Gamma$ where $ga⊏h$ and $y^{\prime }\in \alpha$, $G^{y^{\prime }}\left(g\right)=a$.

2. $h⊏d$ for some $d\in D$ s.t. $dCy$.

$C_{fair}^{\pi }$ is the set of all $(y,\alpha )\in {el}^{\Gamma }$ s.t. for any $h\in H_{y\alpha }$, $G^y\left(h\right)=\pi \left(h\right)$.

Here, condition 1 says that we only “take responsibility” for the action on a particular history when the history was actually observed (all preceding evaluations of the agent are realized computations). Condition 2 says that we only “take responsibility” when the computer is working correctly.

At this stage, Definition 1.5 should be regarded as tentative, since we only have one result so far that validates this definition, namely that the set of $(y,\alpha )$ in $C_{\text{fair}}^{\pi }$ only depends on what the policy $\pi$ does on possible inputs, instead of having the set depending on what the policy does on impossible inputs where one of the past memorized actions is not what the policy would actually do. We hope to rectify this in future articles.

Putting everything together: given a loss function $L:{el}^{\Gamma }\to R$, which depends on how the mathematical universe is, and which computations are realized or not-realized, the loss of policy $\pi$ is given by just applying the bridge transform to coax the hypotheses into the appropriate form, intersecting with the set of possibilities consistent with the agent behaving according to a policy, and evaluating the expectation of the loss function, as detailed below.

$$L^{\text{pol}}(\pi ,\zeta ):=\left(Br\right(\zeta ⋉\Theta )\cap {\top }_{C_{fair}^{\pi }})\left(L\right)$$

Evaluating agents

So far we regarded the agent’s source code $G$ and the agent’s policy $\pi$ as independent variables, and explained how to evaluate different policies given fixed $G$. However, in reality the policy is determined by the source code. Therefore, it is desirable to have a way to evaluate different codes. We achieve this using an algorithmic information theory approach.

We also want to allow the loss function to depend on $G$. That is, we postulate $L:(R^M{)}^H\times {el}^{\Gamma }\to R$. Specifically, since $R^M$ can be thought of as the space of actions, $(R^M{)}^H$ is basically the space of policies. In section 3 we will see why in some detail, but for now think of the difference between “maximizing my own happiness” and “maximizing Alice’s happiness”: the first is defined relative to the agent (depends on $G$) whereas the second is absolute (doesn’t depend on $G$). In particular, for a selfish agent that just cares about its own observations, its loss function must reference its own source code.

Definition 1.6: Denote $G^{\ast }:H\to A$ the policy actually implemented by $G$. Fix $\xi \in \Delta (A^H$). The physicalist intelligence of $G$ relative to the baseline policy mixture $\xi$, prior $\zeta$ and loss function $L$ is defined by:

$$g(G\mid \xi ;\zeta ,L):=-\log {Pr}_{\pi \sim \xi }\left[L^{\text{pol}}\right(┌G┐,\pi ,\zeta )\le L^{\text{pol}}(┌G┐,G^{\ast },\zeta \left)\right]$$

Notice that $L^{\text{pol}}$ depends on $┌G┐$ in two ways: through the direct dependence of $L$ on $┌G┐$ and through $C_{fair}^{\pi }$.

In particular, it makes sense to choose $\zeta$ and $\xi$ as simplicity priors.

There is no obvious way to define “physicalist AIXI”, since we cannot have $g=\infty$. For one thing, $g$ is not even defined for uncomputable agents. In principle we could define it for non-uniform agents, but then we get a fixpoint problem that doesn’t obviously have a solution: finding a non-uniform agent $G$ s.t. $L^{\text{pol}}(┌G┐,G^{\ast },\zeta )={min}_{\pi }{L}^{\text{pol}}(┌G┐,\pi ,\zeta )$. On the other hand, once we spell out the infinitary ($N=\infty$) version of the formalism, it should be possible to prove the existence of agents with arbitrarily high finite $g$. That’s because our agent can use quining to access its own source code $┌G┐$, and then brute force search a policy ${\pi }_{ϵ}^{\ast }$ with $L^{\text{pol}}(┌G┐,{\pi }_{ϵ}^{\ast },\zeta )<{inf}_{\pi :H\to A}{L}^{\text{pol}}(┌G┐,\pi ,\zeta )+ϵ$.

2. Properties of the bridge transform

In this section we will not need to assume $\Gamma$ is of the form ${\Sigma }^R$: unless stated otherwise, it will be any finite set.

Sanity test

Proposition 2.1: For any $\Gamma$, $\Phi$ and $\Theta \in {\square }^c(\Gamma \times \Phi )$, $Br\left(\Theta \right)$ exists and satisfies ${pr}_{\Gamma \times \Phi }Br\left(\Theta \right)=\Theta$. In particular, if $\Theta \in \square (\Gamma \times \Phi )$ then $Br\left(\Theta \right)\in \square ({el}^{\Gamma }\times \Phi )$.

Downwards closure

Roughly speaking, the bridge transform tell us which computations are physically realized. But actually it only bounds it from one side: some computations are definitely realized but any computation might be realized. One explanation for why it must be so is: if you looked at the world in more detail, you might realize that there are small-scale, previously invisible, features of the world which depend on novel computations. There is a direct tension between bounding both sides (i.e. being able to say definitively that a computation isn’t instantiated) and having the desirable property that learning more about the small-scale structure of the universe narrows down the uncertainty. To formalize this, we require the following definitions:

Definition 2.1: Let $X$ be a partially ordered set (poset). Then the induced partial order on ${\Delta }^{c}X$ is defined as follows. Given $\theta ,\eta \in {\Delta }^{c}X$, $\theta ⪯\eta$ if and only if for any monotonically non-decreasing function $f:X\to R_{+}$, $\theta \left(f\right)\le \eta \left(f\right)$.

This is also called the stochastic order (which is standard mathematical terminology). Intuitively, $\theta ⪯\eta$ means that $\eta$ has its measure further up in the poset than $\theta$ does. To make that intuition formal, we can also characterize the induced order as follows:

Proposition 2.2: Let $X$ be a poset, $\theta ,\eta \in {\Delta }^{c}X$. Then, $\theta ⪯\eta$ if and only if there exists $\kappa :X\to \Delta X$ s.t.:

• For all $x\in X$, $y\in supp\kappa \left(x\right)$: $x\le y$.

• ${\kappa }_{\ast }\theta \le \eta$

Proposition 2.3: Let $X$ be a poset, $\theta ,\eta \in {\Delta }^{c}X$. Then, $\theta ⪯\eta$ if and only if there exists $\kappa :X\to \Delta X$ s.t.:

• For all $x\in X$, $y\in supp\kappa \left(x\right)$: $x\ge y$.

• $\theta \le {\kappa }_{\ast }\eta$

Or, in words, you can always go from $\theta$ to $\eta$ by moving probability mass upwards from where it was, since $\kappa \left(x\right)$ is always supported on the set of points at-or-above $x$.

We can now state the formalization of only bounding one side of the bridge transform. Let ${el}^{\Gamma }\times \Phi$ be equipped with the following order. $(y,\alpha ,x)\le (y^{\prime },{\alpha }^{\prime },x^{\prime })$ if and only if $y=y^{\prime }$, $x=x^{\prime }$ and $\alpha \subseteq {\alpha }^{\prime }$. Then:

Proposition 2.4: For any $\Gamma$, $\Phi$ and $\Theta \in {\square }^c(\Gamma \times \Phi )$, $Br\left(\Theta \right)$ is downwards closed w.r.t. the induced order on ${\Delta }^c({el}^{\Gamma }\times \Phi )$. That is, if $\theta \in Br\left(\Theta \right)$ and $\eta ⪯\theta$ then $\eta \in Br\left(\Theta \right)$.

Simple special case

Let’s consider the special case where there’s only one program, which can produce two possible outputs, a 0 and a 1. And these two outputs map to two different distributions over physics outcomes in $\Phi$. Intuitively, if the computation isn’t realized/​instantiated, the distribution over physics outcome should be identical, while if the computation is realized/​instantiated, it should be possible to look at the physics results to figure out how the computation behaves. The two probability distributions may overlap some intermediate amount, in which case it should be possible to write the two probability distributions as a mixture between a probability distribution that behaves identically regardless of the program output (the “overlap” of the two distributions), and a pair of probability distributions corresponding to the two different program outputs which are disjoint. And the total variation distance ($d_{\text{TV}}\left(\varphi \right(0),\varphi (1\left)\right)$) between the two probability distributions is connected to the size of the distribution overlap. Proposition 2.5 makes this formal.

Proposition 2.5: Consider a finite set $X$, ${\varphi }_{?}\in \Delta X$, ${\varphi }_{!}:\{0,1\}\to \Delta X$, $p\in [0,1]$ and $\varphi :=(1-p){\varphi }_{?}+p{\varphi }_{!}$. Then, $p\ge d_{\text{TV}}\left(\varphi \right(0),\varphi (1\left)\right)$. Conversely, consider any $\varphi :\{0,1\}\to \Delta X$. Then, there exist ${\varphi }_{?}\in \Delta X$ and ${\varphi }_{!}:\{0,1\}\to \Delta X$ s.t.$\varphi =(1-p){\varphi }_{?}+p{\varphi }_{!}$ for $p:=d_{\text{TV}}\left(\varphi \right(0),\varphi (1\left)\right)$.

The bridge transform should replicate this same sort of analysis. We can interpret the case of “total uncertainty over math, but knowing how physics turns out conditional on knowing how math turns out” by ${\top }_{\Gamma }⋉\varphi$ for some $\varphi :\Gamma \to \Delta \Phi$. Taking the special case where $\Gamma =\{0,1\}$ for one program that can output two possible answers, it should return the same sort of result, where the two probability distributions can be decomposed into their common overlap, and non-overlapping pieces, and the overlapping chunk of probability measure should be allocated to the event “the computation is not instantiated”, while the disjoint ones should be allocated to the event “the computation is instantiated”. As it turns out, this does indeed happen, with the same total-variation-distance-based upper bound on the probability that the computation is unrealized (namely, $1-d_{\text{TV}}\left(\varphi \right(0),\varphi (1\left)\right)$)

Proposition 2.6: Consider any $\Phi$ and $\varphi :\{0,1\}\to \Delta \Phi$. Denote $U:=\{0,1\}\times \left\{\right\{0,1\left\}\right\}\times \Phi$ (the event “program is unrealized”). Let $\Lambda :=Br({\top }_{\{0,1\}}⋉\varphi )$. Then,

$$\Lambda \left({\chi }_U\right)=1-d_{\text{TV}}\left(\varphi \right(0),\varphi (1\left)\right)$$

Bound in terms of support

If the physical state is $x\in \Phi$, then it is intuitively obvious that the physically manifest facts about the computational universe $y\in \Gamma$ must include the fact that $(y,x)\in supp\Theta$. Formally:

Proposition 2.7: If $X$ is a poset and $\Theta \in {\square }^{c}X$, then ${\Theta }^{\downarrow }$ will denote downward closure of $\Theta$. For any $\Gamma$, $\Phi$ and $\Theta \in {\square }^c(\Gamma \times \Phi )$ if $(y,\alpha ,x)\in suppBr\left(\Theta \right)$ and $y^{\prime }\in \alpha$, then $(y^{\prime },x)\in supp\Theta$. Moreover, define ${sus}^{\Theta }:\Phi \to 2^{\Gamma }$ by ${sus}^{\Theta }\left(x\right):=\{y\in \Gamma \mid (y,x)\in supp\Theta \}$. Then, $Br\left(\Theta \right)\subseteq (\Theta ⋉{sus}^{\Theta }{)}^{\downarrow }$ (we slightly abuse notation by treating ${sus}^{\Theta }$ as a mapping $\Gamma \times \Phi \to 2^{\Gamma }$ that doesn’t depend on the first argument, and also playing loose with the order of factors in the set on which our HUCs live).

Putting that into words, all the $(\alpha ,x)$ that are in the support of the bridge transform have $\alpha$ being a subset of the support of $\Theta$ restricted to $x$. The bridge transform knows not to provide $\alpha$ sets which are too large. Further, the bridge transform is smaller/​has more information than the transform that just knows about what the support of $\Theta$ is.

Idempotence

Imagine replacing the physical state space $\Phi$ by the space of manifest facts $2^{\Gamma }$. Intuitively, the latter should contain the same information about computations as the former. Therefore, if we apply the bridge transform again after making such a replacement, we should get the same thing. Given any set $X$, we denote ${id}_X:X\to X$ the identity mapping and ${diag}_X:X\to X\times X$ the diagonal mapping.

Proposition 2.8: For any $\Gamma$, $\Phi$ and $\Theta \in {\square }^c(\Gamma \times \Phi )$

$$Br\left({pr}_{\Gamma \times 2^{\Gamma }}Br\right(\Theta \left)\right)=\left[\right({id}_{\Gamma }\times {diag}_{2^{\Gamma }}{)}_{\ast }{pr}_{\Gamma \times 2^{\Gamma }}Br\left(\Theta \right){]}^{\downarrow }$$

Refinement

If we refine a hypothesis (i.e. make it more informative) in the ultradistribution sense, the bridge transform should also be more refined:

Proposition 2.9: For any $\Gamma$, $\Phi$ and ${\Theta }_1,{\Theta }_2\in {\square }^c(\Gamma \times \Phi )$, if ${\Theta }_1\subseteq {\Theta }_2$ then $Br\left({\Theta }_1\right)\subseteq Br\left({\Theta }_2\right)$.

Another way of refining a hypothesis is refining the ontology, i.e. moving to a richer state space. For example, we can imagine one hypothesis that describes the world in terms of macroscopic objects and another hypothesis that describes the worlds in term of atoms which are otherwise perfectly consistent with each other. In this case, we also expect the bridge transform to become more refined. This desiderata is where the downwards closure comes from, because it’s always possible to pass to a more detailed view of the world and new computations could manifest then. It’s also worth noting that, if you reduce your uncertainty about math, or physics, or move to a more detailed state space, that means more computations are instantiated, and the loss goes down as a result.

Proposition 2.10: For any $\Gamma$, ${\Phi }_1$, ${\Phi }_2$, $t:{\Phi }_2\to {\Phi }_1$ and $\Theta \in {\square }^c(\Gamma \times {\Phi }_2)$

$$({id}_{{el}^{\Gamma }}\times t{)}_{\ast }Br(\Theta )\subseteq Br(({id}_{\Gamma }\times t{)}_{\ast }\Theta )$$

In general, when we refine the ontology, the manifest facts can become strictly refined: the richer state “knowns more” about computations than the poorer state. However, there is a special case when we don’t expect this to happen, namely when the rich state depends on the computational universe only through the poor state. Roughly speaking, in this case once we have the poor state we can sample the rich state without evaluating any more programs. To formalize this, it is convenient to introduce pullback of HUCs, which is effectively the probabilistic version of taking a preimage.

Definition 2.2: Let $X,Y$ be finite sets and $t:X\to Y$ a mapping. We define the pullback^[7] operator $t^{\ast }:{\square }^{c}Y\to {\square }^{c}X$ by

$$t^{\ast }\Theta :=\{\theta \in {\Delta }^{c}X\mid t_{\ast }\theta \in \Theta \}$$

Proposition 2.11: Consider any $\Gamma$, ${\Phi }_1$, ${\Phi }_2$, $t:{\Phi }_2\to {\Phi }_1$, $\Xi :{\Phi }_1\to {\square }^c{\Phi }_2$ and $\Theta \in {\square }^c(\Gamma \times {\Phi }_1)$ s.t. $t\Xi ={id}_{{\Phi }_1}$. Then,

$$({id}_{{el}^{\Gamma }}\times \Xi {)}_{\ast }Br(\Theta )\subseteq Br(({id}_{\Gamma }\times \Xi {)}_{\ast }\Theta )\subseteq ({id}_{{el}^{\Gamma }}\times t{)}^{\ast }Br(\Theta )$$

In particular,

$${pr}_{{el}^{\Gamma }}Br\left(\Theta \right)={pr}_{{el}^{\Gamma }}Br\left(\right({id}_{\Gamma }\times \Xi {)}_{\ast }\Theta )$$

In other words, we can think of ${\Phi }_1$ as the coarse-grained version, and ${\Phi }_2$ as the fine-grained version, and $t$ as the coarse-graining function, and $\Xi$ as the function mapping a coarse grained state to some uncertainty over what the corresponding fine-grained state is.$\Theta$ is your starting uncertainty over computations and the coarse-grained state. Then, in order from most to least informative, you could apply $\Xi$ after the bridge transform, apply $\Xi$ before the bridge transform, or pull back along $t$ after the bridge transform (because the pullback is the most uninformative inverse, in a sense). But all this stuff only affects the information about the fine-grained state ${\Phi }_2$. If you were to forget about that, the data about computations and how they’re instantiated is the same no matter whether you do the bridge transform with the coarse state, or try your best to fill in what the fine state is first, because the “fill in the fine state” function doesn’t depend on your uncertainty about computations, so it doesn’t contain any more information about how math is connected to physics than the original coarse-grained view had.

Mixing hypotheses

In general, we cannot expect the bridge transform to commute with taking probabilistic mixtures. For example, let $\Gamma =\{0,1\}$, $\Phi =\{0,1\}$, ${\Theta }_1={\top }_{\{00,11\}}$, ${\Theta }_2={\top }_{\{01,10\}}$. Clearly either for ${\Theta }_1$ or for ${\Theta }_2$, we expect the program to be realized (since the physical state encodes its output). On the other hand, within $\frac{1}{2}{\Theta }_1+\frac{1}{2}{\Theta }_2$ we have both the distribution “50% on 00, 50% on 01” and the distribution “50% on 10, 50% on 11″. The physical state no longer necessarily “knows” the output, since the mixing “washes away” information. However, there’s no reason why mixing would create information, so we can expect the mixture of bridge transforms to be a refinement of the bridge transform of the mixture.

Proposition 2.12: For any $\Gamma$, $\Phi$, ${\Theta }_1,{\Theta }_2\in {\square }^c(\Gamma \times \Phi )$ and $p\in [0,1]$

$$pBr\left({\Theta }_1\right)+(1-p)Br\left({\Theta }_2\right)\subseteq Br(p{\Theta }_1+(1-p\left){\Theta }_2\right)$$

Now suppose that we are we are mixing hypotheses whose state spaces are disjoint. In this case, there can be no washing away since the state “remembers” which hypothesis it belongs to.

Proposition 2.13: Consider some $\Gamma$, ${\Phi }_1$, ${\Phi }_2$, ${\Theta }_1\in {\square }^c(\Gamma \times {\Phi }_1)$, ${\Theta }_1\in {\square }^c(\Gamma \times {\Phi }_2)$ and $p\in [0,1]$. Regard ${\Phi }_{1,2}$ as subsets of ${\Phi }_1\bigsqcup {\Phi }_2$, so that $p{\Theta }_1+(1-p){\Theta }_2\in {\square }^c(\Gamma \times ({\Phi }_1\bigsqcup {\Phi }_2\left)\right)$. Then,

$$pBr\left({\Theta }_1\right)+(1-p)Br\left({\Theta }_2\right)=Br(p{\Theta }_1+(1-p\left){\Theta }_2\right)$$

In particular, if we consider the hypotheses within a prior as having disjoint state spaces because of using different ontologies, then the total bridge transform (Definition 1.2) is just the ordinary bridge transform, justifying that definition.

Conjunction

Given a hypothesis and a subset of $\Gamma$, we can form a new hypothesis by performing conjunction with the subset. Intuitively, the same conjunction should apply to the manifest facts.

Proposition 2.14: Consider some $\Gamma$, $\Phi$, $\Theta \in {\square }^c(\Gamma \times \Phi )$ and $F\subseteq \Gamma$. Let ${\cap }_F:2^{\Gamma }\to 2^{\Gamma }$ be defined by ${\cap }_F\left(\alpha \right):=F\cap \alpha$. Then,

$${Br}_{\Gamma }(\Theta \cap {\top }_{F\times \Phi })=({id}_{\Gamma }\times {\cap }_F\times {id}_{\Phi }{)}_{\ast }({Br}_{\Gamma }\left(\Theta \right)\cap {\top }_{F\times 2^{\Gamma }\times \Phi })$$

Moreover, let $\iota :F\to \Gamma$ and ${el}^{\iota }:{el}^F\to {el}^{\Gamma }$ be the natural injections. Then,

$${Br}_F\left(\right(\iota \times {id}_{\Phi }{)}^{\ast }(\Theta \cap {\top }_{F\times \Phi }))=({el}^{\iota }\times {id}_{\Phi }{)}^{\ast }{Br}_{\Gamma }(\Theta \cap {\top }_{F\times \Phi })$$

That first identity says that it doesn’t matter whether you update on a fact about math first and apply the bridge transform later, or whether you do the bridge transform first and then “update” (actually it’s slightly more complicated than that since you have to narrow down the sets $\alpha$) the hypothesis afterwards. The second identity shows that, post conjunction with $F$, taking the bridge transform with respect to $F$ is essentially the same as taking it with respect to $\Gamma$. If a batch of math universes is certain not to occur, then dropping them entirely from the domain $\Gamma$ produces the same result.

As a corollary, we can eliminate dependent variables. That is, suppose some programs can be expressed as functions of other programs (using $\Theta$). Intuitively, such programs can be ignored. Throwing in the dependent variables and doing the bridge transform on that augmented space of “math universes” produces the same results as trying to compute the dependent variables after the bridge transform is applied. So, if you know how to generate the dependent variables, you can just snip them off your hypothesis, and compute them as-needed at the end. Formally:

Proposition 2.15: Consider some ${\Gamma }_0$, ${\Gamma }_1$, $r:{\Gamma }_0\to {\Gamma }_1$, $\Phi$, $\Theta \in {\square }^c({\Gamma }_0\times \Phi )$. Let $\iota :{\Gamma }_0\to {\Gamma }_0\times {\Gamma }_1$ be given by $\iota \left(y\right):=(y,r(y\left)\right)$. Then,

$$Br\left(\right(\iota \times {id}_{\Phi }{)}_{\ast }\Theta )=({el}^{\iota }\times {id}_{\Phi }{)}_{\ast }Br\left(\Theta \right)$$

Factoring $\Gamma$

When $\Gamma$ is of the form ${\Gamma }_0\times {\Gamma }_1$, we can interpret the same $\Theta \in \square (\Gamma \times \Phi )$ in two ways: treat ${\Gamma }_1$ as part of the computational universe, or treat it as part of the physical state. We can also “marginalize” over it altogether staying with a HUD on ${\Gamma }_0\times \Phi$. The bridge transforms of these objects satisfy a simple relationship:

Proposition 2.16: Consider some ${\Gamma }_0$, ${\Gamma }_1$, $\Phi$ and $\Theta \in {\square }^c({\Gamma }_0\times {\Gamma }_1\times \Phi )$. Define $\pi :{el}^{{\Gamma }_0\times {\Gamma }_1}\to {el}^{{\Gamma }_0}$ by $\pi (y,z,\alpha ):=(y,\{y^{\prime }\in {\Gamma }_0\mid (y^{\prime },z)\in \alpha \left\}\right)$. Then,

$$(\pi \times {id}_{\Phi }{)}_{\ast }{Br}_{{\Gamma }_0\times {\Gamma }_1}(\Theta )\subseteq {pr}_{{el}^{{\Gamma }_0}\times \Phi }{Br}_{{\Gamma }_0}(\Theta )\subseteq {Br}_{{\Gamma }_0}({pr}_{{\Gamma }_0\times \Phi }\Theta )$$

Intuitively, the least informative result is given by completely ignoring the ${\Gamma }_1$ part of math about how a particular batch of computations turns out. A more informative result is given by treating ${\Gamma }_1$ as an aspect of physics. And the most informative result is given by treating ${\Gamma }_1$ as an aspect of math, and then pruning the resulting $\alpha :2^{{\Gamma }_0\times {\Gamma }_1}$ down to be over $2^{{\Gamma }_0}$ by plugging in the particular ${\Gamma }_1$ value.

Continuity

Proposition 2.17: $Br\left(\Theta \right)$ is a continuous function of $\Theta$.

Lamentably, this only holds when all the involved sets are finite. However, a generalization of continuity, scott-continuity, still appears to hold in the infinite case. Intuitively, the reason why is, if $\Phi$ is a continuous space, like the interval $[0,1]$, you could have a convergent sequence of hypotheses $\Theta$ where the events “program outputs 0” and “program outputs 1″ are believed to result in increasingly similar outputs in physics, and then in the limit, where they produce the exact same physics outcome, the two possible outputs of the program suddenly go from “clearly distinguishable by looking at physics” to “indistinguishable by looking at physics” and there’s a sharp change in whether the program is instantiated. This is still a Scott-continuous operation, though.

3. Constructing loss functions

In section 1 we allowed the loss function $L:{el}^{\Gamma }\to R$ to be arbitrary. However, in light of Proposition 2.4, we might as well require it to be monotonically non-decreasing. Indeed, we have the following:

Proposition 3.1: Let $X$ be a finite poset, $f:X\to R$ and $\Theta \in {\square }^{c}X$ downward closed. Define $f^{max}:X\to R$ by $f^{max}\left(x\right):={max}_{y\le x}f\left(y\right)$. Observe that $f^{max}$ is always non-decreasing. Then, $\Theta \left(f\right)=\Theta \left(f^{max}\right)$.

This essentially means that downwards-closed HUC’s are entirely pinned down by their expectation value on monotone functions, and all their other expectation values are determined by the expectation value of the “nearest monotone function”, the $f^{max}$.

This requirement on $L$ (we will call it the “monotonicity principle”) has strange consequences that we will return to below.

The informal discussion in section 1 hinted at how the computationalist stance allows describing preferences via a loss function of the type ${el}^{\Gamma }\to R$. We will now proceed to make it more formal.

Selfish agents

A selfish agent naturally comes with a cartesian loss function $L:D\to R.$

Definition 3.1: For any $(y,\alpha )\in {el}^{\Gamma }$, the set of experienced histories $X_{y\alpha }$ is the set of all $ha\in H_{y\alpha }\times A$ (see Definition 1.5) s.t. for any $y^{\prime }\in \alpha$, $G^{y^{\prime }}\left(h\right)=a$. The physicalized loss function is

$$L^{\text{phys}}(y,\alpha ):=\underset{ha\in X_{y\alpha }}{min}\underset{d\in D:ha⊑d}{max}L\left(d\right)$$

Notice that $L^{\text{phys}}$ implicitly depends on $┌G┐$ through $X_{y\alpha }$.

Intuitively, if $\alpha$ is a large set, there’s different mathematical universes that result in the agent’s policy being different, but which are empirically indistinguishable (ie, they’re in the same set $\alpha$), so the agent’s policy isn’t actually being computed on those histories where the contents of $\alpha$ disagree on what the agent does. The agent takes the loss of the best history that occurs, but if there’s uncertainty over what happens afterwards/​later computations aren’t instantiated, it assumes the worst-case extension of that history.

This definition has the following desirable properties:

• If $g\in X_{y\alpha }$, $ha\in H\times A$ and $ha⊑g$, then $h\in X_{y\alpha }.$

• $L^{\text{phys}}$ is monotone. If the set of indistinguishable mathematical universes gets larger, fewer histories are instantiated, so the loss is larger because of the initial minimization.

• Fix $(y,\alpha )\in {el}^{\Gamma }$. Suppose there is $d\in D$ s.t. $X_{y\alpha }=\{ha\in H\times A\mid ha⊑d\}$. Then $L^{\text{phys}}(y,\alpha )=L\left(d\right)$. If the set of indistinguishable mathematical universes all agree on what the destiny of the agent is and disagree elsewhere, then the loss is just the loss of that destiny.

We can think of inconsistent histories within $X_{y\alpha }$ as associated with different copies of the agent. If $ha\in X_{y\alpha }$ is s.t. for any $g⊐ha$, $g\notin X_{y\alpha }$, then $h$ represents the history of a copy which reached the end of its life. If there is only one copy and its lifespan is maximal, then $L^{\text{phys}}$ produces the cartesian loss of this copy. If the lifespan is less than maximal, $L^{\text{phys}}$ produces the loss of the worst-case continuation of the history which was experienced. In other words, death (i.e., the future behavior of the agents policy not being a computation that has any effect on the state of the universe) is always the worst-case outcome. If there are multiple copies, the best-off copy determines the loss.

Definition 3.1 is somewhat arbitrary, but the monotonicity principle is not. Hence, for a selfish agent (i.e. if $L$ only depends on $y,\alpha$ through $X_{y\alpha }$) death must be the worst-case outcome (for fixed $y$). Note that this weighs against getting the agent to shut down if it’s selfish. And creating additional copies must always be beneficial. For non-selfish agents, similar observations hold if we replace “death” by “destruction of everything that matters” and “copies of the agent” by “copies of the (important sector of the) world”.

This is an odd state of affairs. Intuitively, it seems coherent to have preferences that contradict the monotonicity principle. Arguably, humans have such preferences, since we can imagine worlds that we would describe as net negative. At present, we’re not sure how to think about it, but here are some possibilities, roughly in order of plausibility:

1. Perfect physicalists must obey the monotonicity principle, but humans are not perfect physicalists. Indeed, why would they be? The ancestral environment didn’t require us to be physicalists, and it took us many centuries to stop thinking of human minds as “metaphysically privileged”. This might mean that we need of theory of “cartesian agents in a physicalist universe”. For example, perhaps from a cartesian perspective, a physicalist hypothesis needs to be augmented by some “theory of epiphenomena” which predicts e.g. which of several copies will “I” experience.

2. Something is simply wrong about the formalism. One cheap way to escape monotonicity is by taking the hull of maximal points of the bridge transform. However, this spoils compatibility with refinements (Propositions 2.9 and 2.10). We expect this to spoil ability to learn in some way.

3. Death is inherently an irreversible event. Every death is the agent’s first death, therefore the agent must always have uncertainty about the outcome. Given a non-dogmatic prior, there will be some probability on outcomes that are not actually death. These potential “afterlives” will cause the agent to rank death as better than the absolute worst-case. So, “non-survivalist” preferences can be reinterpreted as a prior biased towards not believing in death. Similar reasoning applies to non-selfish agents.

4. We should just bite the bullet and prescriptively endorse the odd consequences of the monotonicity principle.

In either case, we believe that continuing the investigation of monotonic physicalist agents is a promising path forward. If a natural extension or modification of the theory exists which admits non-monotonic agents, we might discover it along the way. If we don’t, at least we will have a better lower bound for the desiderata such a theory should satisfy.

Unobservable states

What if the loss depends on an unobservable state from some state space $S$? In order to encode states, let’s assume some $M_S\in N$ and ${dec}_S:{\Sigma }^{M_S}\to S$. That is, just as the action of the agent can be thought of as a tuple of programs returning the bits of the computation of the agent’s action, a state can be thought of as a tuple of programs returning the bits of the computation of what the next state is. Let $B$ be a finite set (“nature’s actions”). Just as observations, in the selfish case, occur based on what computations are instantiated, nature’s choice of action $B$ depends on what computations are instantiated. Denote $H_S:=(A\times B\times O\times S{)}^{<N}$, $D_S:=(A\times B\times O\times S{)}^{N-1}\times A$. Let $┌T┐:S\times A\times B\to R^{M_S}$ be the “ontological”^[8] transition rule program, $E:S\to O$ the observation rule and $L:D_S\to R$ the loss function.

Definition 3.2: For any $(y,\alpha )\in {el}^{\Gamma }$, the set of admissible histories $Z_{y\alpha }$ is the set of all $ha\in H_S\times A$ s.t. the following conditions hold:

1. ${pr}_{H\times A}ha\in X_{y\alpha }$

2. If $g\in H_S\times A\times B$, $o\in O$ and $s\in S$ are s.t.$gos⊑h$ then $o=E\left(s\right)$.

3. If $g\in H_S\times A\times B\times O$, $s_1,s_2\in S$, $a\in A$, $b\in B$ and $o\in O$ are s.t. $g{s}_{1}abo{s}_2⊑h$ and $y^{\prime }\in \alpha$ then $T^{y^{\prime }}(s_1,a,b)=s_2$ (the notation $T^y$ is analogical to Definition 1.3).

Pretty much, condition 1 says that the visible parts of the history are consistent with how math behaves in the same sense as we usually enforce with $H_{y\alpha }$. Condition 2 says that the history should have the observations arising from the states in the way that $E$ demands. And condition 3 says that the history should have the states arising from your action and nature’s action and the previous state in the way the transition program dictates (in that batch of mathematical universes).

The physicalized loss function is

$$L^{\text{s-phys}}(y,\alpha ):=\underset{ha\in Z_{y\alpha }}{min}\underset{g\in D_S:ha⊑g}{max}L\left(g\right)$$

This arises in a nearly identical way to the loss function from Definition 3.1, it’s precisely analogous.

That is, for a history to be admissible we require not only the agent to be realized on all relevant inputs but also the transition rule.

We can also define a version with $┌T┐:S\times B\to R^{M_S}$ in which the agent is not an explicit part of the ontology. In this case, the agent’s influence is solely through the “entanglement” of $G$ with $T$. I.e., the agent’s beliefs over math have entanglements between how the agent’s policy turns out and how the physics transition program behaves, s.t. when conditioning on different policies, the computations corresponding to different histories become instantiated.

Cellular automata

What if our agent maximizes the number of gliders in the game of life? More generally, consider a cellular automaton with cell state space $S$ and dimension $d\in N$. For any $t\in N$, let $N_t\subseteq Z^d$ be the set of cells whose states are necessary to compute the state of the origin cell at timestep $t$. Let $┌T_t┐:S^{N_t}\to R^{M_S}$ be the program which computes the time evolution of the cellular automaton. I.e., it’s mapping the state of a chunk of space to the computation which determines the value of a cell $t$ steps in the future. Denote $D_S^d:=S^{N\times Z^d}$. This is the space of all possible histories, specifying everything that occurs from the start of time. Let $L:D_S^d\to R$ be the loss function (the $N$ coordinate is time).

Definition 3.3: For any $\alpha \in 2^{\Gamma }$, the set of admissible histories $Z_{\alpha }$ is the set of all pairs $(C,h)$ where $C\subset N\times Z^d$ is finite and $h:C\to S$ s.t. the following conditions hold:

1. If $(t_2,c)\in C$, $t_1<t_2$ and $v\in N_{t_2-t_1}$ then $(t_1,c+v)\in C$.

2. If $(t_2,c)\in C$, $t_1<t_2$ and $y\in \alpha$ then $T_{t_2-t_1}^y(\lambda v.h(t_1,c+v\left)\right)=h(t_2,c)$.

Basically, the $C$ is a chunk of spacetime, and condition 1 says that said chunk must be closed under taking the past lightcone, and condition 2 says the state of that spacetime chunk must conform with the behavior of the transition computation.

The physicalized loss function is defined very much as it used to be, except that for what used to be “past histories” now it has states of past-closed chunks of spacetime.

$$L^{\text{c-phys}}(y,\alpha ):=\underset{(C,h)\in Z_{\alpha }}{min}\underset{g\in D_S^d:h=g{|}_C}{max}L\left(g\right)$$

Notice that, since the dynamics of the cellular automaton is fixed, the only variables “nature” can control are the initial conditions. However, it is straightforward to generalize this definition to cellular automata whose time evolution is underspecified (similarly to the $B$ we had before).

Diamond maximizer

What if our agent maximizes the amount of diamond (or some other physical property)? In order to define “diamond” we need a theory of physics. For example, we can start with macroscopic/​classical physics and define diamond by its physical and/​or chemical properties. Or, we can start with nonrelativistic quantum mechanics and define diamond in terms of its molecular structure. These definitions might be equivalent in our universe, but different in other hypothetical universes (this is similar to twin Earth). In any case, there is no single “right” choice, it is simply a matter of definition.

A theory of physics is mathematically quite similar to a cellular automaton. This theory will usually be incomplete, something that we can represent in infra-Bayesianism by Knightian uncertainty. So, the “cellular automaton” has underspecified time evolution. Using this parallel, the same method as in the previous subsection should allow physicalizing the loss function. Roughly speaking, a possible history is “real” when the computation simulating the time evolution of the theory of physics with the corresponding initial condition is realized. However, we are not sure what do if the theory of physics is stochastic. We leave it as an open problem for now.

To summarize, you can just take a hypothesis about how computations are entangled with some theory of physics, some computations simulating the time evolution of the universe (in which the notion of “diamond” is actually well-defined letting the amount of diamond in a history be computed), and a utility function over these states. The bridge transform extracts which computations are relevant for the behavior of physics in the hypothesis (which may or may not include the diamond-relevant computations), and then you just check whether the computations for a particular diamond-containing history are instantiated, take the best-case one, and that gives you your loss.

Human preferences seem not substantially different from maximizing diamonds. Most of the things we value are defined in terms of people and social interactions. So, we can imagine a formal loss function defined in terms of some “theory of people” implicit in our brain.

4. Translating Cartesian to Physicalist

Physicalism allows us to do away with the cartesian boundary between the agent and its environment. But what if there is a cartesian boundary? An important sanity test is demonstrating that our theory can account for purely cartesian hypotheses as a special case.

Ordinary laws

We will say “law” to mean what was previously called “belief function”. We start from examining a causal law $W:A^H\to \square D$. I.e., it maps a policy to an ultradistribution over destinies, in a way that looks like a closed convex set of environments. Let $R:=H$ and $\Sigma =A$ so that $\Gamma$ becomes the set of policies (we identify the history $h\in H$ with the computation $┌G\left(h\right)┐\in R$ given by what the agent’s source does on input $h$.). In particular, for now there is no computer, or the computer can only simulate the agent itself. Let $\Phi :=D$. This is our set of “physics outcomes”, complete action-observation sequences. Then, $W$ induces the physicalist hypothesis ${\Theta }_W\in \square (\Gamma \times \Phi )$ by ${\Theta }_W:={\top }_{\Gamma }⋉W$. ${\Theta }_W$ is what we called the “first-person static view” of the law. Total uncertainty over the choice of policy, interacting with the kernel which maps a policy to the resulting uncertainty over the environment.

We want to find the bridge transform of ${\Theta }_W$ and evaluate the corresponding counterfactuals, in order to show that the resulting $L^{\text{pol}}$ is the same as in the cartesian setting. In order to do this, we will use some new definitions.

Definition 4.1: Let $X,Y$ be finite sets and $Q\subseteq Y\times X$ a relation. A polycontribution on $Q$ is $\varpi :X\to R_{+}$ s.t. for any $y\in Y$:

$$\sum _{x\in Q\left(y\right)}\varpi \left(x\right)\le 1$$

$\varpi$ is a polydistribution when for any $y\in Y$

$$\sum _{x\in Q\left(y\right)}\varpi \left(x\right)=1$$

We denote the space of polycontributions by ${\Delta }_Q^{c}X$ and the space of polydistributions by ${\Delta }_{Q}X$.

Here, $Q\left(y\right)$ stands for $\{x\in X\mid yQx\}$. Polycontributions and polydistributions are essentially a measure on a space, along with a family of subsets indexed by the $y\in Y$, s.t. restricting to any particular subset makes a contribution/​distribution, respectively.

Definition 4.2: Let $X$ and $Y$ be finite sets and $Q\subseteq Y\times X$ a relation. A polycontribution kernel (PoCK) over $Q$ is $\kappa :Y\to {\Delta }^{c}X$ s.t. there is $\varpi \in {\Delta }_Q^{c}X$ s.t. for any $x\in X$ and $y\in Y$

$$\kappa (y;x)={\chi }_{Q\left(y\right)}\left(x\right)\varpi \left(x\right)$$

Or, in other words, a PoCK is the kernel that you get by taking a polycontribution and using the input point $y$ to decide which $Y$-indexed subset to restrict the measure to. Let $Q_0\subseteq A^H\times D$ be defined by $Q_0:=\left\{\right(\pi ,a_0{o}_0\dots a_{N-1})\mid \forall i<N:a_i=\pi (a_0\dots o_{i-1}\left)\right\}$. This is the “destiny is consistent with the policy” relation where the policy produces the actions that the destiny says it does, when placed in that situation. Let $\varphi :A^H\to \Delta D$ be a Bayesian causal law (an environment). Then, $\varphi$ is a PoCK over $Q_0$! Conversely, any $\varphi :A^H\to \Delta D$ which is a PoCK is an environment.

Essentially, for an environment, each choice of policy results in a different probability distribution over outcomes. However, these resulting probability distributions over outcomes are all coherent with each other where the different policies agree on what to do, so an alternate view of an environment is one where there’s one big measure over destinies (where the measure on a particular destiny is the probability of getting that destiny if the policy plays along by picking the appropriate actions) and the initial choice of policy restricts to a particular subset of the space of destinies, yielding a probability distributions over destinies compatible with a given policy.

Clearly, the next step is to generalize this beyond probability distributions.

Definition 4.3: Let $X$ and $Y$ be finite sets $Q\subseteq Y\times X$ a relation. A polyultracontribution kernel (PUCK) over $Q$ is $\Xi :Y\to {\square }^{c}X$ s.t. there is $\Pi \subseteq Y\to {\Delta }^{c}X$ a set of PoCKs s.t. for any $y\in Y$, $\Xi \left(y\right)$ is the closure of the convex hull of ${\bigcup }_{\kappa \in \Pi }\kappa \left(y\right)$.

In particular, $W$ is a PUCK over $Q_0$. Conversely, any $W:A^H\to \square D$ which is a PUCK is a causal law.

Proposition 4.1: Consider some $\Gamma$, $\Phi$, a relation $Q\subseteq \Gamma \times \Phi$ and a PUCK $\Xi$ over $Q$. Let $\Theta :={\top }_{\Gamma }⋉\Xi$. Then,

$$Br\left(\Theta \right)=[{\top }_{\Gamma }⋉({sus}^{\Theta }⋊\Xi ){]}^{\downarrow }=[{\top }_{\Gamma }⋉(Q^{-1}⋊\Xi ){]}^{\downarrow }$$

Here, $Q^{-1}\left(x\right):=\{y\in Y\mid yQx\}$. The first identity says that the inequality of Proposition 2.7 is saturated.

Specifically, as shown in the next proposition, for our case of cartesian causal laws, the computations realized according to $Br\left({\Theta }_W\right)$ are exactly the histories that occur.

Corollary 4.3: Suppose that for any $h\in D$ and $\pi :H\to A$ s.t. $h\in suppW\left(\pi \right)$, it holds that $hC\pi$. That is, the observations $W$ predicts to receive from the computer are consistent with the chosen policy. Let $L:D\to R$ be a cartesian loss function and $\pi :H\to A$ a policy. Then,

$$\left({pr}_{{el}^{\Gamma }}Br\right({\Theta }_W)\cap C_{\text{fair}}^{\pi })\left(L^{\text{phys}}\right)=W(\pi ;L)$$

Notice that the assumption $W$ is causal combined with the assumption $W$ is consistent with $C$ imply that it’s not possible to use the computer to simulate the agent on any history other than factual past histories. For pseudocausal $W$ the bridge transform no longer agrees with the cartesian setting: if a selfish agent is simulated in some counterfactual, then the computationalist stance implies that it can incur the loss for that counterfactual. This artifact should be avoidable for non-selfish agents, but we leave this for future articles.

Turing laws

Now let’s examine a cartesian setting which involves a general computer. Without further assumptions, we again cannot expect the physicalist loss to agree with the cartesian, since the programs running on the computer might be entangled with the agent. If the agent runs simulations of itself then it should expect to experience those simulations (as opposed to the cartesian view). Therefore, we will need a “no entanglement” assumption.

Let $\Gamma :={\Gamma }_1\times A^H$ for some ${\Gamma }_1$ (the computations that don’t involve the agent) and $\Phi$ is still $D$. We want a notion of “joint belief about the environment and ${\Gamma }_1$” in the cartesian setting, which generalizes causal laws. We will call it a “Turing law” or “Turing environment” in the Bayesian case.

For our particular application (though the theorems are more general), $Z$ will be generally taken to be the space of computations that aren’t about what you personally do, $Y$ is the space of computations about what you personally do, and $X$ is the space of destinies/​physics outcomes, if the reader wants to try their hand at interpreting some of the more abstract theorems.

Definition 4.4: Let $X$, $Y$ and $Z$ be finite sets and $Q\subseteq Y\times X$ a relation. A $Z$-polycontribution on $Q$ is $\varpi :X\times Z\to R_{+}$ s.t.

$$\sum _{z\in Z}\underset{y\in Y}{max}\sum _{x\in Q\left(y\right)}\varpi (x,z)\le 1$$

$\varpi$ is a $Z$-polydistribution on $Q$ when it is both a $Z$-polycontribution on $Q$ and a polydistribution on $Q\times Z$. Here, we regard $Q\times Z$ as a relation between $Y$ and $X\times Z$.

We denote the space of $Z$-polycontributions by ${\Delta }_Q^{c}XZ$ and the space of $Z$-polydistributions by ${\Delta }_{Q}XZ$.

Notice that a $Z$-polycontribution on $Q$ is in particular a polycontribution on $Q\times Z$: ${\Delta }_Q^{c}XZ\subseteq {\Delta }_{Q\times Z}^c(X\times Z)$.

Definition 4.5: Let $X$, $Y$, $Z$ be finite sets and $Q\subseteq Y\times X$ a relation. A $Z$-polycontribution kernel ($Z$-PoCK) over $Q$ is $\kappa :Y\to {\Delta }^c(X\times Z)$ s.t. there is $\varpi \in {\Delta }_Q^{c}XZ$ s.t. for any $x\in X$, $y\in Y$ and $z\in Z$

$$\kappa (y;x,z)={\chi }_{Q\left(y\right)}\left(x\right)\varpi (x,z)$$

In the context of the identification we’ll be using where $Z$ is the rest of math, $Y$ is the parts of math about what you personally do, and $X$ is the space of physics outcomes/​destinies, this definition is saying that the Z-PoCK is mapping a policy to the joint distribution over math and destinies you’d get from restricting the measure to “I’m guaranteed to take this policy, regardless of how the rest of math behaves”.

A Turing environment can be formalized as a ${\Gamma }_1$-PoCK $\kappa :A^H\to \Delta (D\times {\Gamma }_1)$ over $Q_0$. Indeed, in this case Definition 4.4 essentially says that the environment can depend on a ${\Gamma }_1$ variable (Remember, ${\Gamma }_1$ is the rest of math), and also we have a belief about this variable that doesn’t depend on the policy. Now, how to transform that into a physicalist hypothesis:

Proposition 4.2: Let $X$, $Y$ and $Z$ be finite sets, $Q\subseteq Y\times X$ a relation, $\kappa :Y\to {\Delta }^c(X\times Z)$ a $Z$-PoCK over $Q$ and $\Theta \in {\square }^{c}Y$. Then, there exist $\mu \in {\Delta }^{c}Z$ and $\varphi :Z\times Y\to {\Delta }^{c}X$ s.t. for all $z\in Z$, $\lambda y.\varphi (z,y)$ is a PoCK over $Q$, and for all $y\in Y$, $\kappa \left(y\right)=(\lambda z.\varphi (z,y\left)\right)⋊\mu$. Moreover, suppose that $({\mu }_1,{\varphi }_1)$ and $({\mu }_2,{\varphi }_2)$ are both as above^[9]. Then,

$${\mu }_1⋉\Theta ⋉{\varphi }_1={\mu }_2⋉\Theta ⋉{\varphi }_2$$

Or, for a more concrete example, this is saying that it’s possible to decompose the function mapping a policy to a distribution over the rest of math and the destiny, into two parts. One part is just a probability distribution over the rest of math, the second part is a function mapping a policy and the behavior of the rest of math to a probability distribution over destinies. Further, this decomposition is essentially unique, in that for any two ways you do it, if you take the probability distribution over the rest of math, have it interact with some arbitrary way of mapping the rest of math to policies, and then have that interact with the way to get a distribution over destinies, that resulting joint distribution could also be produced by any alternate way of doing the decomposition.

In the setting of Proposition 4.2, we define $\Theta \ast \kappa \in {\square }^c(Z\times Y\times X)$ by $\mu ⋉\Theta ⋉\varphi$. The physicalist hypothesis corresponding to Turing environment $\kappa$ turns out be ${\top }_{A^H}\ast \kappa$. The no-entanglement assumption is evident in how the $A^H$ and ${\Gamma }_1$ variables are “independent”. Well, the actual choice of policy can depend on how the rest of math turns out, but the set of available choices of policy doesn’t depend on how the rest of math turns out, and that’s the sense in which there’s independence.

To go from Turing environments to Turing laws, we will need the following generalization to sets of contributions.

Definition 4.6: Let $X$, $Y$ and $Z$ be finite sets $Q\subseteq Y\times X$ a relation. A $Z$-polyultracontribution kernel ($Z$-PUCK) over $Q$ is $\Xi :Y\to {\square }^c(X\times Z)$ s.t. there is $\Pi \subseteq \left({\Delta }^c\right(X\times Z){)}^Y$ a set of $Z$-PoCKs s.t. for any $y\in Y$, $\Xi \left(y\right)$ is the closure of the convex hull of ${\bigcup }_{\kappa \in \Pi }\kappa \left(y\right)$.

A Turing law is a ${\Gamma }_1$-PUCK $W:A^H\to \square (D\times {\Gamma }_1)$ over $Q_0$.

Definition 4.7: Let $X$, $Y$ and $Z$ be finite sets, $Q\subseteq Y\times X$ a relation, $\Xi :Y\to {\square }^c(X\times Z)$ a $Z$-PUCK over $Q$ and $\Theta \in {\square }^{c}Y$. Let $\Pi \subseteq \left({\Delta }^c\right(X\times Z){)}^Y$ be the set of $Z$-PoCKs $\kappa$ s.t. for any $y\in Y$, $\kappa \left(y\right)\in \Xi \left(y\right)$. Then, $\Theta \ast \Xi \in {\square }^c(Z\times Y\times X)$ is the closure of the convex hull of ${\bigcup }_{\kappa \in \Pi }(\Theta \ast \kappa )$.

As a sanity test for the sensibility of Definition 4.7, we have the following proposition. It demonstrates that instead of taking all compatible $Z$-PoCKs, we could take any subset whose convex hull is all of them. If that was not the case, it would undermine our justification for using convex sets (namely, that taking convex hull doesn’t affect anything).

Proposition 4.3: Let $X$, $Y$ and $Z$ be finite sets, $Q\subseteq Y\times X$ a relation, ${\kappa }_1,{\kappa }_2:Y\to {\Delta }^c(X\times Z)$ $Z$-PoCKs over $Q$, $\Theta \in {\square }^{c}Y$ and $p\in [0,1]$. Then, $p{\kappa }_1+(1-p){\kappa }_2$ is also a $Z$-PoCK over $Q$, and

$$\Theta \ast (p{\kappa }_1+(1-p\left){\kappa }_2\right)\subseteq p\Theta \ast {\kappa }_1+(1-p)\Theta \ast {\kappa }_2$$

The physicalist hypothesis correspond to Turing law $W$ is ${\Theta }_W:={\top }_{A^H}\ast W$. Total uncertainty over the available choices of policy interacting with the correspondence between the policy, the rest of math, and the resulting destiny. Now, we need to study its bridge transform with a necessary intermediate result to proceed further, bounding the bridge transform result between an upper and lower bound:

Proposition 4.4: Let $X$, $Y$ and $Z$ be finite sets, $Q\subseteq Y\times X$ a relation and $\Xi :Y\to {\square }^c(X\times Z)$ a $Z$-PUCK over $Q$. Denote $\Theta :={\top }_{Z\times Y}\ast \Xi$. Define ${\beta }_0,{\beta }_1:Z\times Y\times X\to 2^{Z\times Y}$ by ${\beta }_0(z,y,x):=\left\{z\right\}\times Q^{-1}\left(x\right)$, ${\beta }_1(z,y,x):=Z\times Q^{-1}\left(x\right)$. Then,

$$(\Theta ⋉{\beta }_0{)}^{\downarrow }\subseteq Br(\Theta )\subseteq (\Theta ⋉{\beta }_1{)}^{\downarrow }$$

Here, we slightly abuse notation by implicitly changing the order of factors in $Z\times Y\times 2^{Z\times Y}\times X$.

The application of this is that the resulting uncertainty-over-math/​info about what computations are realized (i.e., the $\alpha$ set) attained when you do the bridge transform is bounded between two possible results. One bound is perfect knowledge of what the rest of math besides your policy is (everything’s realized), paired with exactly as much policy uncertainty as is compatible with the destiny of interest. The other bound is no knowledge of the rest of math besides your policy (possibly because the policy was chosen without looking at other computations, and the rest of reality doesn’t depend on those computations either), combined with exactly as much policy uncertainty as is compatible with the destiny of interest.

Corollary 4.5: Suppose that for any $h\in D$, $z\in {\Gamma }_1$ and $\pi :H\to A$ s.t. $(h,z)\in suppW\left(\pi \right)$, it holds that $hCz\pi$. That is, the observations $W$ predicts to receive from the computer are consistent with the chosen policy and $W$’s beliefs about computations. Let $L:D\to R$ be a cartesian loss function and $\pi :H\to A$ a policy. Define $\tilde{L}:D\times {\Gamma }_1\to R$ by $\tilde{L}(h,z):=L\left(h\right)$. Then,

$$\left({pr}_{{el}^{\Gamma }}Br\right({\Theta }_W)\cap C_{fair}^{\pi })\left(L^{\text{phys}}\right)=W(\pi ;\tilde{L})$$

And so, the Turing law case manages to add up to normality in the cartesian setting (with computer access) as well.

5. Discussion

Summary

Let’s recap how our formalism solves the problems stated in section 0:

• Physicalist hypotheses require no bridge rules, since they are formulated in terms of a physical state space (that be chosen to be natural/​impartial) rather than agent’s actions and observations. Therefore, they accrue no associated description complexity penalty, and should allow for a superior learning rate. Intuitively, a physicalist agent knows it is an manifestation of the program $G$ rather than just any object in the universe.

• A physicalist agent doesn’t develop uncertainty after subjectively-simple but objectively-complex events. It is easy to modify a cartesian hypothesis such that the camera going black leads to completely new behavior. However, it would require an extremely contrived (complex) modification of physics, since the camera is not a simple object and has no special fundamental significance. Basically, the simplest way that computations and reality correspond which instantiates the computations of the agent seeing black forever afterward gets a complexity boost by about the complexity of the agent’s own source code.

• A physicalist agent doesn’t regard its existence as axiomatic. A given physicalist hypothesis might contain 0 copies of the agent, 1 copy or multiple copies (more generally, some ultradistribution over the number of copies). Therefore, hypotheses that predict the agent’s existence with relatively high probability gain an advantage (i.e. influence the choice of policy more strongly), as they intuitively should. At the same time, hypotheses that predict too many copies of the agent to have predictive power are typically^[10] also suppressed, since policy selection doesn’t affect their loss by much (whatever you do, everything happens anyway).

• Acausal attacks against physicalist agents are still possible, but the lack of bridge rule penalty means the malign hypotheses are not overwhelmingly likely compared to the true hypothesis. This should allow defending using algorithms of “consensus” type.

What about Solomonoff’s universality?

For selfish agents the physicalist framework can be reinterpreted as a cartesian framework with a peculiar prior. Given that the Solomonoff prior is universal (dominates every computable prior), does it mean AIXI is roughly as good as a selfish physicalist agent? Not really. The specification of that “physicalist prior” would involve the source code of the agent. (And, if we want to handle hypotheses that predict multiple copies, it would also involve the loss function.) For AIXI, there is no such source code. More generally, the intelligence of an agent w.r.t. the Solomonoff prior is upper bounded by its Kolmogorov complexity (see this). So, highly intelligent cartesian agents place small prior probability on physicalism, and it would take such an agent a lot of data to learn it.

Curiously, the converse is also true: the translation of section 4 also involves the source code of the agent, so intelligent physicalist agents place small prior probability on cartesian dualism. We could maybe use description complexity relative to the agent’s own source code, in order to define a prior^[11] which makes dualism and physicalism roughly equivalent (for selfish agents). This might be useful as a way to make cartesian algorithms do physicalist reasoning. But in any case, you still need physicalism for the analysis.

It would be interesting to do a similar analysis for bounded Solomonoff-like priors, which might require understanding the computational complexity of the bridge transform. In the bounded case, the Kolmogorov complexity of intelligent agents need not be high, but a similar penalty might arise from the size of the environment state space required for the physicalist-to-cartesian translation.

Are manifest facts objective?

Physicalism is supposed to be “objective”, i.e. avoid privileging the observer. But, is the computationalist stance truly objective? Is there an objective fact of the matter about which facts about computations are physically manifest /​ which computations are realized?

At first glance it might seem there isn’t. By Proposition 2.7, if the physicalist hypothesis $\Theta \in \square (\Gamma \times \Phi )$ is s.t. $supp\Theta \subseteq F\times \Phi$ for some $F\subseteq \Gamma$, then $suppBr\left(\Theta \right)\subseteq {el}^F\times \Phi$. So, if the hypothesis asserts a fact $F$ then $F$ is always physically manifest. But, isn’t $\Theta$ merely the subjective knowledge of the agent? And if so, doesn’t it follow that physical manifesting is also subjective?

At second glance things get more interesting. Suppose that agent $G$ learned hypothesis $\Theta$. Since $G$ is part of the physical universe, it is now indeed manifest in the state of the physical universe that hypothesis $\Theta$ (and in particular $F$) is true. Looking at it from a different angle, suppose that a different agent $G^{\prime }$ knows that $G$ is a rational physicalist. Then it must agree that if $G$ will learn $\Theta$ then $F$ is manifest. So, possibly there is a sense in which manifesting is objective after all.

It would be interesting to try and build more formal theory around this question. In particular, the ability of the user and the AI to agree on manifesting (and agree in general) seems important for alignment in general (since we want them to be minimizing the same loss) and for ruling out acausal attacks in particular (because we don’t want the AI to believe in a hypothesis s.t. it wouldn’t be rational for the user to believe in it).

Physicalism and alignment

The domain of the physicalist loss function doesn’t depend on the action and observation sets, which is an advantage for alignment since it can make it easier to “transfer” the loss function from the user to the AI when they have different action or observation sets.

On the other hand, the most objectionable feature of physicalism is the monotonicity principle (section 3). The default conclusion is, we need to at least generalize the formalism to allow for alignment. But, suppose we stick with the monotonicity principle. How bad is the resulting misalignment?

Consider an alignment protocol in which the AI somehow learns the loss function of the user. As a toy model, assume the user has a cellular automaton loss function $L_{\text{user}}$, like in section 3. Assume the AI acts according to the loss function $L_{\text{user}}^{\text{c-phys}}$ of Definition 3.3. What will result?

Arguably, in all probable scenarios the result will be pretty close to optimal from the user’s perspective. One type of scenario where the AI makes the wrong choice is, when choosing between destruction of everything and a future which is even worse than destruction of everything. But, if these are the only available options then probably something has gone terribly wrong at a previous point. Another type of scenario where the AI possibly makes the wrong choice is, if it creates multiple disconnected worlds each of which has valence from the perspective of the user, at least one of which is good but some of the which are bad. However, there usually seems to be no incentive to do that^[12].

In addition, the problems might be milder in approaches that don’t require learning the user’s loss function, such as IDA of imitation.

So, the monotonicity principle might not introduce egregious misalignment in practice. However, this requires more careful analysis: there can easily be failure modes we have overlooked.

Future research directions

Here’s a list of possible research directions, not necessarily comprehensive. Some of them were already mentioned in the body of the article.

Directions which are already more or less formalized:

• Generalizing everything to infinite spaces (actually, we already have considerable progress on this and hopefully will publish another article soon). [EDIT: See also profinite field spaces.]

• Generalizing section 4 to certain pseudocausal laws. Here, we can make use of the monotonicity principle to argue that if Omega discontinues all of its simulations, they correspond to copies which are not the best-off and therefore don’t matter in the loss calculus. Alternatively, we can design a non-selfish loss function that only counts the “baseline reality” copies. Moreover, it would be interesting to understand the connection between the pseudocausality condition and the “fairness” of Definition 1.5.

• Define a natural candidate for the Solomonoff prior for physicalist hypotheses.

• Prove that the physicalist intelligence $g$ is truly unbounded, and study its further properties.

Directions which are mostly clear but require more formalization:

• Prove learning-theoretic results (regret bounds). For example, if we assume that there is a low-cost experiment that reveals the true hypothesis, then low regret should be achievable. In particular, such theorems would allow validating Definition 1.5.

• Generalize section 3 to stochastic ontologies.

• Allow uncertainty about the loss function, and/​or about the source code. This would introduce another variable into our physicalist hypothesis, and we need to understand how to bridge transform should act on it. [EDIT: Maybe we don’t need uncertainty about source code, see quining in IBP.]

• Analyze the relationship between cartesian and physicalist agents with bounded simplicity priors.

• For (cartesian) reinforcement learning, MDPs and POMDPs are classical examples of environment for which various theoretical analysis is possible. It would be interesting to come up with analogical examples of physicalist hypotheses and study them. For example, we can consider infra-Markov chains where the transition kernel depends on $\Gamma$.

Directions where it’s not necessarily clear how to approach the problem:

• Understand the significance of the monotonicity principle better, and whether there are interesting ways to avoid it. [EDIT: Looks like there are.]

• Formally study the objectivity of manifesting, perhaps by deriving some kind of agreement theorems for physicalist agents.

• Define and analyze physicalist alignment protocols. [EDIT: one important candidate is Physicalist Superimitation.]

• It is easy to imagine how a physicalist hypothesis can describe a universe with classical physics. But, the real universe is quantum. So, it doesn’t really have a distribution over some state space $\Phi$ but rather a density matrix on some Hilbert space. Is there a natural way to deal with this in the present formalism? If so, that would seem like a good solution to the interpretation of quantum mechanics. [EDIT: Indeed, there is a solution.]

---

1. ↩︎

   If we ignore ideas such as the von Neumann-Wigner interpretation of quantum mechanics.

2. ↩︎

   This other direction also raises issues with counterfactuals. These issues are also naturally resolved in our formalism.

3. ↩︎

   The notation $\Theta \times \Lambda$ is reserved for a different, commutative, operation which we will not use here.

4. ↩︎↩︎↩︎

   A simplifying assumption we are planning to drop in future articles.

5. ↩︎

   We have considered this type of setting before with somewhat different motivation.

6. ↩︎

   The careful reader will observe that programs sometimes don’t halt which means that the “true” computational universe is ill-defined. This turns out not to matter much.

7. ↩︎

   Previously we defined pullback s.t. it can only be applied to particular infra/​ultradistributions. Here we avoid this limitation by using infra/​ultracontributions as the codomain.

8. ↩︎

   Meaning that, this rule is part of the definition of the state rather than a claim about physical reality.

9. ↩︎

   Disintegrating a distribution into a semidirect product yields a unique result, but for contributions that’s no longer true, since it’s possible to move scalars between the two factors.

10. ↩︎

    One can engineer loss functions for which they are not suppressed, for example if the loss only depends on the actions and not on the observations. But such examples seem contrived.

11. ↩︎

    It’s a self-referential definition, but we can probably resolve the self-reference by quining.

12. ↩︎

    One setting in which there is an incentive: Suppose there are multiple users and the AI is trying to find a compromise between their preferences. Then, it might decide to create disconnected worlds optimized for different users. But, this solution might be much worse than the AI thinks, if Alice’s world contains Bob!atrocities.

[Vanessa Kosoy]

The post is still largely up-to-date. In the intervening year, I mostly worked on the theory of regret bounds for infra-Bayesian bandits, and haven’t made much progress on open problems in infra-Bayesian physicalism. On the other hand, I also haven’t found any new problems with the framework.

The strongest objection to this formalism is the apparent contradiction between the monotonicity principle and the sort of preferences humans have. While my thinking about this problem evolved a little, I am still at a spot where every solution I know requires biting a strange philosophical bullet. On the other hand, IBP is still my best guess about naturalized induction, and, more generally, about the conjectured “attractor submanifold” in the space of minds, i.e. the type of mind to which all sufficiently advanced minds eventually converge.

One important development that did happen is my invention of the PreDCA alignment protocol, which critically depends on IBP. I consider PreDCA to be the most promising direction I know at present to solving alignment, and an important (informal) demonstration of the potential of the IBP formalism.

[plex]

fwiw that strange philosophical bullet fits remarkably well with a set of thoughts I had while reading Anthropic Bias about ‘amount of existence’ being the fundamental currency of reality (a bunch of the anthropic paradoxes felt like they were showing that if you traded sufficiently large amounts of “patterns like me exist more” then you could get counterintuitive results like bending the probabilities of the world around you without any causal pathway), and infraBayes requiring it actually updated me a little towards infraBayes being on the right track.

My model of why humans seem to prefer non-existence to existence in some cases is that our ancestors faced situations which could reduce their ability to self-propagate to almost zero, and needed to avoid these really hard. Evolution gave us training signals which can easily generate subagents which are single-mindedly obsessed with avoiding certain kinds of intense suffering. This motivates us to avoid a wide range of realistic things which cost us existence, but as a side-effect of being emphasized so much make it possible to tip into suicidality in cases where, in our history, it was not too costly because things were bad enough anyway that the agent wouldn’t propagate much (suicide when the cues for self-propagation being relatively likely for on-distribution humans should have been weeded out). This strikes me as unintended and a result of a hack which works pretty well on-distribution, and likely not reflectively consistent in the limit. An evolution which could generate brains with unbounded compute would not make agents which ever preferred suicide or non-existence.

Another angle on this is thinking of evolution having set things up for a sign-flipped subagent to be reinforced, which just wants to Not Be. This is not a natural shape for an agent to be, but it’s useful enough that the pattern to generate it is common.

This is all pretty handwave-y and I don’t claim high confidence that it’s correct or useful, but might be interesting babble.

[tailcalled]

Personally I am very very confused about infra-Bayesianism, so I cannot comment on this post. This is despite the fact that I would really like to understand it as I’m pretty interested in the notion of naturalized induction.

I think a big constraint preventing me and others from getting into infra-Bayesianism is that you are immediately hit with a giant wall of math that you have no intuition for. Meanwhile when it comes to ordinary Bayesianism, I have a lot of intuition due to having been exposed to lots of basic examples and such. I think production of extremely beginner-friendly material could get a lot more people discussing this.

(I could probably learn it without extremely beginner-friendly material if I took a lot of time to sit down and work with the hard math. But then it competes for my attention with many other topics that I also really should put effort into, so it is unlikely to happen.)

[Vanessa Kosoy]

Well, Alex is on a working on an infra-Bayesianism textbook, maybe that will help.

[Raemon]

I think the thing I would like most is a post that’s “explain why infrabayes matters like I’m 5, basically without math.” (Admittedly I’m not the main target audience, but, I think this is a valuable thing to exist that will help figure out the pedagogical architecture of an eventual textbook)

I’m somewhat worried, from some past experience hearing people try to learn about infrabayes, that you and Alex keep overshooting in complexity. Curious if for the textbook Alex is talking to people with various degrees of background knowledge and seeing how different explanations land?

[tailcalled]

Personally I feel like I get why it matters from the other posts, and would just like a gentle introduction to the math.

[Daniel Kokotajlo]

Cool! I too was intimidated by all the math, but I want to understand it at some point!

[Jon Garcia]

Could you explain what the monotonicity principle is, without referring to any symbols or operators? I gathered that it is important, that it is problematic, that it is a necessary consequence of physicalism absent from cartesian models, and that it has something to do with the min-(across copies of an agent) max-(across destinies of the agent copy) loss. But I seem to have missed the content and context that makes sense of all of that, or even in what sense and over what space the loss function is being monotonic.

Your discussion section is good. I would like to see more of the same without all the math notation.

If you find that you need to use novel math notation to convey your ideas precisely, I would advise you to explain what every symbol, every operator, and every formula as a whole means every time you reference them. With all the new notation, I forgot what everything meant after the first time they were defined. If I had a year to familiarize myself with all the symbols and operators and their interpretations and applications, I imagine that this post would be much clearer.

That being said, I appreciate all the work you put into this. I can tell there’s important stuff to glean here. I just need some help gleaning it.

[Vanessa Kosoy]

Could you explain what the monotonicity principle is, without referring to any symbols or operators?

The loss function of a physicalist agent depends on which computational facts are physically manifest (roughly speaking, which computations the universe runs), and on the computational reality itself (the outputs of computations). The monotonicity principle requires it to be non-decreasing w.r.t. the manifesting of less facts. Roughly speaking, the more computations the universe runs, the better.

This is odd, because it implies that the total destruction of the universe is always the worst possible outcome. And, the creation of an additional, causally disconnected, world can never be net-negative. For a monotonic agent, there can be no net-negative world^[1]. In particular, for selfish monotonic agents (such that only assign value to their own observations), this means death is the worst possible outcome and the creation of additional copies of the agent can never be net-negative.

With all the new notation, I forgot what everything meant after the first time they were defined.

Well, there are the “notation” and “notation reference” subsections, that might help.

That being said, I appreciate all the work you put into this. I can tell there’s important stuff to glean here.

Thank you!

---

1. ↩︎

   At least, all of this is true if we ignore the dependence of the loss function on the other argument, namely the outputs of computations. But it seems like that doesn’t qualitatively change the picture.

[ViktoriaMalyasova]

Thank you for explaining this! But then how can this framework be used to model humans as agents? People can easily imagine outcomes worse than death or destruction of the universe.

[Vanessa Kosoy]

The short answer is, I don’t know.

The long answer is, here are some possibilities, roughly ordered from “boring” to “weird”:

1. The framework is wrong.

2. The framework is incomplete, there is some extension which gets rid of monotonicity. There are some obvious ways to make such extensions, but they look uglier and without further research it’s hard to say whether they break important things or not.

3. Humans are just not physicalist agents, you’re not supposed to model them using this framework, even if this framework can be useful for AI. This is why humans took so much time coming up with science.

4. Like #3, and also if we thought long enough we would become convinced of some kind of simulation/​deity hypothesis (where the simulator/​deity is a physicalist), and this is normatively correct for us.

5. Because the universe is effectively finite (since it’s asymptotically de Sitter), there are only so many computations that can run. Therefore, even if you only assign positive value to running certain computations, it effectively implies that running other computations is bad. Moreover, the fact the universe is finite is unsurprising since infinite universes tend to have all possible computations running which makes them roughly irrelevant hypotheses for a physicalist.

6. We are just confused about hell being worse than death. For example, maybe people in hell have no qualia. This makes some sense if you endorse the (natural for physicalists) anthropic theory that only the best-off future copy of you matters. You can imagine there always being a “dead copy” of you, so that if something worst-than-death happens to the apparent-you, your subjective experiences go into the “dead copy”.

[Jon Garcia]

The monotonicity principle requires it to be non-decreasing w.r.t. the manifesting of less facts. Roughly speaking, the more computations the universe runs, the better.

I think this is what I was missing. Thanks.

So, then, the monotonicity principle sets a baseline for the agent’s loss function that corresponds to how much less stuff can happen to whatever subset of the universe it cares about, getting worse the fewer opportunities become available, due to death or some other kind of stifling. Then the agent’s particular value function over universe-states gets added/​subtracted on top of that, correct?

[Vanessa Kosoy]

No, it’s not a baseline, it’s just an inequality. Let’s do a simple example. Suppose the agent is selfish and cares only about (i) the experience of being in a red room and (ii) the experience of being in a green room. And, let’s suppose these are the only two possible experiences, it can’t experience going from a room in one color to a room in another color or anything like that (for example, because the agent has no memory). Denote $G$ the program corresponding to “the agent deciding on an action after it sees a green room” and $R$ the program corresponding to “the agent deciding on an action after it sees a red room”. Then, roughly speaking^[1], there are 4 possibilities:

• ${\alpha }_{\varnothing }$: The universe runs neither $R$ nor $G$.

• ${\alpha }_R$: The universe runs $R$ but not $G$.

• ${\alpha }_G$: The universe runs $G$ but not $R$.

• ${\alpha }_{RG}$: The universe runs both $R$ and $G$.

In this case, the monotonicity principle imposes the following inequalities on the loss function $L$:

$$L\left({\alpha }_{\varnothing }\right)\ge L\left({\alpha }_R\right)$$ $$L\left({\alpha }_{\varnothing }\right)\ge L\left({\alpha }_G\right)$$ $$L\left({\alpha }_R\right)\ge L\left({\alpha }_{RG}\right)$$ $$L\left({\alpha }_G\right)\ge L\left({\alpha }_{RG}\right)$$

That is, ${\alpha }_{\varnothing }$ must be the worst case and ${\alpha }_{RG}$ must be the best case.

---

1. ↩︎

   In fact, manifesting of computational facts doesn’t amount to selecting a set of realized programs, because programs can be entangled with each other, but let’s ignore this for simplicity’s sake.

[Jon Garcia]

Okay, so it’s just a constraint on the final shape of the loss function. Would you construct such a loss function by integrating a strictly non-positive computation-value function over all of space and time (or at least over the future light-cones of all its copies, if it focuses just on the effects of its own behavior)?

[Vanessa Kosoy]

Space and time are not really the right parameters here, since these refer to $\Phi$ (physical states), not $\Gamma$ (computational “states”) or $2^{\Gamma }$ (physically manifest facts about computations). In the example above, it doesn’t matter where the (copy of the) agent is when it sees the red room, only the fact the agent does see it. We could construct such a loss function by a sum over programs, but the constructions suggested in section 3 use minimum instead of sum, since this seems like a less “extreme” choice in some sense. Ofc ultimately the loss function is subjective: as long as the monotonicity principle is obeyed, the agent is free to have any loss function.

[bargo]

A theory of physics is mathematically quite similar to a cellular automaton. This theory will usually be incomplete, something that we can represent in infra-Bayesianism by Knightian uncertainty. So, the “cellular automaton” has underspecified time evolution.

What evidence is there that incomplete models with Knightian uncertainty are a way to turn rough models of physics into loss functions? Can the ideas behind it be applied to regular Bayesianism?

[Morpheus]

A physicalist hypothesis is a pair $(\Phi ,\Theta$), where $\Phi$ is a finite^[4:2] set representing the physical states of the universe and $\Theta \in \square (\Gamma \times \Phi )$ represents a joint belief about computations and physics. [...] Our agent will have a prior over such hypotheses, ranging over different $\Phi$.

I am confused what the state space $\Phi$ is adding to your formalism and how it is supposed to solve the ontology identification problem. Based on what I understood, if I want to use this for inference, I have this prior $\xi \in {\square }^c(\Phi ,\Theta )$, and now I can use the bridge transform to project phi out again to evaluate my loss in different counterfactuals. But when looking at your loss function, it seems like most of the hard work is actually done in your relation $C\in e{l}^{\Gamma }$ that determines which universes are consistent, but its definition does not seem to depend on $\Phi$. How is that different from having a prior that is just over $\xi \in {\square }^c\left(\Gamma \right)$ and taking the loss, if $\Phi$ is projected out anyway and thus not involved?

[Vanessa Kosoy]

First, the notation $\xi \in {\square }^c(\Phi ,\Theta )$ makes no sense. The prior is over hypotheses, each of which is an element of $\square (\Gamma \times \Phi )$. $\Theta$ is the notation used to denote a single hypothesis.

Second, having a prior just over $\Gamma$ doesn’t work since both the loss function and the counterfactuals depend on $2^{\Gamma }\times \Gamma$.

Third, the reason we don’t just start with a prior over $2^{\Gamma }\times \Gamma$, is because it’s important which prior we have. Arguably, the correct prior is the image of a simplicity prior over physicalist hypotheses by the bridge transform. But, come to think about it, it might be about the same as having a simplicity prior over $2^{\Gamma }\times \Gamma$, where each hypothesis is constrained to be invariant under the bridge transform (thanks to Proposition 2.8). So, maybe we can reformulate the framework to get rid of $\Phi$ (but not of the bridge transform). Then again, finding the “ultimate prior” for general intelligence is a big open problem, and maybe in the end we will need to specify it with the help of $\Phi$.

Fourth, I wouldn’t say that $\Phi$ is supposed to solve the ontology identification problem. The way IBP solves the ontology identification problem is by asserting that $2^{\Gamma }\times \Gamma$ is the correct ontology. And then there are tricks how to translate between other ontologies and this ontology (which is what section 3 is about).

[David Johnston]

Γ=Σ^R, it’s a function from programs to what result they output. It can be thought of as a computational universe, for it specifies what all the functions do.

Should this say “elements are function… They can be thought of as...?”

Can you make a similar theory/​special case with probability theory, or do you really need infra-bayesianism? If the second, is there a simple explanation of where probability theory fails?

[Vanessa Kosoy]

Should this say “elements are function… They can be thought of as...?”

Yes, the phrasing was confusing, I fixed it, thanks.

Can you make a similar theory/​special case with probability theory, or do you really need infra-bayesianism?

We really need infrabayesianism. On bayesian hypotheses, the bridge transform degenerates: it says that, more or less, all programs are always running. And, the counterfactuals degenerate too, because selecting most policies would produce “Nirvana”.

The idea is, you must have Knightian uncertainty about the result of a program in order to meaningfully speak about whether the universe is running it. (Roughly speaking, if you ask “is the universe running 2+2?” the answer is always yes.) And, you must have Knightian uncertainty about your own future behavior in order for counterfactuals to be meaningful.

It is not surprising that you need infrabayesianism in order to do naturalized induction: if you’re thinking of the agent as part of the universe then you are by definition in the nonrealizable setting, since the agent cannot possibly have a full description of something “larger” than itself.

[noam]

Can someone break Definition 1.1 down for me? I got lost in all the notation and what acts on what, what is projected to where..

[Morpheus]

stands for “support”

I am guessing this refers to this notion of support I found on Wikipedia:

$supp\left(f\right)=\{x\in Domain(f\left)|f\right(x)\ne 0\}$

Edit: fixed Definition.

[Vanessa Kosoy]

It should be $f\left(x\right)\ne 0$. More generally, there is the notion of support from measure theory, which sometimes comes up, although in this post we only work with finite sets so it’s the same.