Vanessa Kosoy

• Vanessa KosoyMay 19, 2025, 12:52 PM

  LW: 4 AF: 3

  0

  AF

  in reply to: Vanessa Kosoy’s comment on: Model­ing ver­sus Implementation

  Btw, what are some ways we can incorporate heuristics into our algorithm while staying on level 1-2?

  1. We don’t know how the prove to required desiderata about the heuristic, but we can still reasonably conjecture them and support the conjectures with empirical tests.

  2. We can’t prove or even conjecture anything useful-in-itself about the heuristic, but the way the heuristic is incorporated into the overall algorithm makes it safe. For example, maybe the heuristic produces suggestions together with formal certificates of their validity. More generally, we can imagine an oracle-machine (where the heuristic is slotted into the oracle) about which we cannot necessarily prove something like a regret bound w.r.t. the optimal policy, but we can prove (or at least conjecture) a regret bound w.r.t. some fixed simple reference policy. That is, the safety guarantee shows that no matter what the oracle does, the overall system is not worse than “doing nothing”. Maybe, modulo weak provable assumptions about the oracle, e.g. that it satisfies a particular computational complexity bound.

  3. [Epistemic status: very fresh idea, quite speculative but intriguing.] We can’t find even a guarantee like a above for a worst-case computationally bounded oracle. However, we can prove (or at least conjecture) some kind of an “average-case” guarantee. For example, maybe we have high probability of safety for a random oracle. However, assuming a uniformly random oracle is quite weak. More optimistically, maybe we can prove safety even for any oracle that is pseudorandom against some complexity class $C_1$ (where we want $C_1$ to be as small as possible). Even better, maybe we can prove safety for any oracle in some complexity class $C_2$ (where we want $C_2$ to be as large as possible) that has access to another oracle which is pseudorandom against $C_1$. If our heuristic is not actually in this category (in particular, $C_2$ is smaller than $P$ and our heuristic doesn’t lie in $C_2$), this doesn’t formally guarantee anything, but it does provide some evidence for the “robustness” of our high-level scheme.

• Vanessa KosoyMay 19, 2025, 11:42 AM

  12 points

  0

  in reply to: Cole Wyeth’s comment on: Model­ing ver­sus Implementation

  Hold my beer ;)

• Vanessa KosoyMay 19, 2025, 7:19 AM

  LW: 21 AF: 7

  7

  AF

  on: Model­ing ver­sus Implementation

  I see modeling vs. implementation as a spectrum more than a dichotomy. Something like:

  1. On the “implementation” extreme you prove theorems about the exact algorithm you implement in your AI, s.t. you can even use formal verification to prove these theorems about the actual code you wrote.

  2. Marginally closer to “modeling”, you prove (or at least conjecture) theorems about some algorithm which is vaguely feasible in theory. Some civilization might have used that exact algorithm to build AI, but in our world it’s impractical, e.g. because it’s uncompetitive with other AI designs. However, your actual code is conceptually very close to the idealized algorithm, and you have good arguments why the differences don’t invalidate the safety properties of the idealized model.

  3. Further along the spectrum, your actual algorithm is about as similar to the idealized algorithm as DQN is similar to vanilla Q-learning. Which is to say, it was “inspired” by the idealized algorithm but there’s a lot of heavy lifting done by heuristics. Nevertheless, there is some reason to hope the heuristic aspects don’t change the safety properties.

  4. On the “modeling” extreme, your idealized model is something like AIXI: completely infeasible and bears little direct resemblance to the actual algorithm in your AI. However, there is still some reason to believe real AIs will have similar properties to the idealized model.

  More precisely, rather than a 1-dimensional spectrum, there are at least two parameters involved:

  • How close is the object you make formal statements about to the actual code of your AI, where “closeness” is measured by the strength of the arguments you have for the analogy, on a scale from “they are literally the same” to solid theoretical and/​or empirical evidence to pure hand-waving/​intuition

  • How much evidence you have for the formal statements, on a scale from “I proved it within some widely accepted mathematical foundation (e.g. PA)” to “I proved vaguely related things, tried very hard but failed to disprove the thing and/​or accumulated some empirical evidence”.

  [EDIT: And a 3rd parameter is, how justified/​testable the assumptions of your model is. Ideally, you want these assumptions to be grounded in science. Some will likely be philosophical assumptions which cannot be tested empirically, but at least they should fit into a coherent holistic philosophical view. At the very least, you want to make sure you’re not assuming away the core parts of the problem.]

  For the purposes of safety, you want to be as close to the implementation end of the spectrum as you can get. However, the model side of the spectrum is still useful as:

  • A backup plan which is better than nothing, more so if there is some combination of theoretical and empirical justification for the analogizing

  • A way to demonstrate threat models, as the OP suggests

  • An intermediate product that helps checking that your theory is heading in the right direction, comparing different research agendas, and maybe even making empirical tests.

• Vanessa KosoyMay 15, 2025, 3:03 AM

  LW: 5 AF: 2

  0

  AF

  in reply to: James Payor’s comment on: Work­ing through a small tiling result

  Sorry, I was wrong. By Lob’s theorem, all versions of ${\text{good}}_{\text{new}}$ are provably equivalent, so they will trust each other.

• Vanessa KosoyMay 14, 2025, 6:33 PM

  LW: 3 AF: 1

  1

  AF

  on: Work­ing through a small tiling result

  IIUC, fixed point equations like that typically have infinitely many solution. So, you defined not one ${\text{good}}_{\text{new}}$ predicate, but an infinite family of them. Therefore, your agent will trust a copy of itself, but usually won’t trust variants of itself with other choices of fixed point. In this sense, this proposal is similar to proposals based on quining (as quining has many fixed points as well).

• Vanessa KosoyApr 30, 2025, 8:16 AM

  6 points

  2

  on: Dat­ing Roundup #4: An App for That

  I don’t know what’s so bad about the “human male” bio. I might have swiped right on that one. (Especially if the profile had additional info that makes him sound interesting.)

• Vanessa KosoyApr 27, 2025, 8:06 AM

  LW: 13 AF: 8

  0

  AF

  in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform

  TLDR: A new proposal for a prescriptive theory of multi-agent rationality, based on generalizing superrationality using local symmetries.

  Hofstadter’s “superrationality” postulates that a rational agent should behave as if other rational agents are similar to itself. In a symmetric game, this implies the selection of a Pareto efficient outcome. However, it’s not clear how to apply this principle to asymmetric games. I propose to solve this by suggesting a notion of “local” symmetry that is much more ubiquitous than global (ordinary) symmetry.

  We will focus here entirely on finite deterministic (pure strategy) games. Generalizing this to mixed strategies and infinite games more generally is an important question left for the future.

  Quotient Games

  Consider a game $\Gamma$ with a set $P$ of players, for each player $i\in P$ a non-empty set $S_i$ or pure strategies and a utility function $u_i:S_{\ast }\to R$, where $S_{\ast }:={\prod }_{j\in P}{S}_j$. What does it mean for the game to be “symmetric”? We propose a “soft” notion of symmetry that is only sensitive to the ordinal ordering between payoffs. While the cardinal value of the payoff will be important in the stochastic context, we plan to treat the latter by explicitly expanding the strategy space.

  Given two games ${\Gamma }^{\left(1\right)}$, ${\Gamma }^{\left(2\right)}$, a premorphism from ${\Gamma }^{\left(1\right)}$ to ${\Gamma }^{\left(2\right)}$ is defined to consist of

  • A mapping $q:P^{\left(1\right)}\to P^{\left(2\right)}$

  • For each $i\in P^{\left(1\right)}$, a mapping ${\overline{q}}_i:S_{q\left(i\right)}^{\left(2\right)}\to S_i^{\left(1\right)}$

  Notice that these mappings induce a mapping $\hat{q}:S_{\ast }^{\left(2\right)}\to S_{\ast }^{\left(1\right)}$ via

  $$\hat{q}(x{)}_i:={\overline{q}}_i\left(x_{q\left(i\right)}\right)$$

  A premorphism is said to be a homomorpism if $q$ and $\overline{q}$ are s.t. that for any $i\in P^{\left(1\right)}$ and $x,y\in S_{\ast }^{\left(2\right)}$, if $u_{q\left(i\right)}\left(x\right)\le u_{q\left(i\right)}\left(y\right)$ then $u_i\left(\hat{q}\right(x\left)\right)\le u_i\left(\hat{q}\right(y\left)\right)$. Homomorphisms makes games into a category in a natural way.

  An automorphism of $\Gamma$ is a homomorphism from $\Gamma$ to $\Gamma$ that has an inverse homomorphism. An automorphism $q$ of $\Gamma$ is said to be flat when for any $i\in P$, if $q\left(i\right)=i$ then ${\overline{q}}_i$ is the identity mapping^[1]. A flat symmetry of $\Gamma$ is a group $G$ together with a group homomorphism $\phi :G\to Aut\left(\Gamma \right)$ s.t.$\phi \left(g\right)$ is flat for all $g\in G$.

  Given a flat symmetry $(G,\phi )$, we can apply the superrationality principle to reduce $\Gamma$ to the “smaller” quotient game $\Gamma /G$. The players are $P/G$ i.e. orbits in the original player set. Given $\alpha \in P/G$, we define the set of strategies for player $\alpha$ in the game $\Gamma /G$ to be

  $$(S/G{)}_{\alpha }:=\{\sigma \in \prod _{i\in \alpha }{S}_{\alpha }|\forall i\in \alpha ,g\in G:\sigma \left(i\right)={\overline{\phi \left(g\right)}}_i\sigma \left(\phi \right(g\left)i\right)\}$$

  This is non-empty thanks to flatness.

  Observe that there is a natural mapping $\gamma :(S/G{)}_{\ast }\to S_{\ast }$ given by

  $$\gamma (x{)}_i:=x_{Gi}(i)$$

  Finally, the utility function of $\alpha$ is defined by

  $$u_{\alpha }\left(x\right):=\frac{1}{\left|\alpha \right|}\sum _{i\in \alpha }{u}_i\left(\gamma \right(x\left)\right)$$

  It is easy to see that the construction of $\Gamma /G$ is functorial w.r.t. the category structure on games that we defined.

  We can generalize this further by replacing game homomorphisms with game quasimorphisms: premorphisms satisfying the following relaxed condition on $q$ and $\overline{q}$:

  • For any $i\in P^{\left(2\right)}$ and $x,y\in S_{\ast }^{\left(1\right)}$, if $u_{q\left(i\right)}\left(x\right)<u_{q\left(i\right)}\left(y\right)$ then $u_i\left(\hat{q}\right(x\left)\right)\le u_i\left(\hat{q}\right(y\left)\right)$.

  This is no longer closed under composition, so no longer defines a category structure^[2]. However, we can still define flat quasisymmetries and the associated quotient games, and this construction is still functorial w.r.t. the original (not “quasi”) category structure. Moreover, there is a canonical homomorphism (not just a quasimorphism) from $\Gamma$ to $\Gamma /G$, even when $G$ is a mere quasisymmetry.

  The significance of quasisymmetries can be understood as follows.

  The set of all games on a fixed set of players $P$ with fixed strategy sets $\{S_i{\}}_{i\in P}$ naturally forms a topological space^[3] $X$. Given a group $G$ acting on the game via invertible premorphisms, the subset of $X$ where $G$ is a symmetry is not closed, in general. However, the subset of $X$ where $G$ is a quasisymmetry is closed. I believe this will be important when generalizing the formalism to infinite games.

  Coarse Graining

  What if a game doesn’t have even quasisymmetries? Then, we can look for a coarse graining of the game which does have them.

  Consider a game $\Gamma$. For each $i\in P$, let $S_i^{\prime }$ be some set and $f_i:S_i\to S_i^{\prime }$ a surjection. Denote $S_{\ast }^{\prime }:={\prod }_{i\in P}{S}_i^{\prime }$. Given any $x\in S_{\ast }^{\prime }$, we have the game ${\Gamma }^x$ in which:

  • The set of players is $P$.

  • For any $i\in P$, their set of strategies is $S_i^x:=f_i^{-1}\left(x_i\right)$. In particular, it implies that the set of outcomes $S_{\ast }^x$ satisfies $S_{\ast }^x\subseteq S_{\ast }$.

  • For any $i\in P$, their utility function is $u_i^x:=u_i{|}_{S_{\ast }^x}$.

  If for any $x\in S_{\ast }^{\prime }$ we choose some ${\alpha }_x\in S_{\ast }^x$ this allows us to define the coarse-grained game ${\Gamma }^{\left(\alpha \right)}$ in which:

  • The set of players is $P$.

  • For any $i\in P$, the set of strategies is $S_i^{\prime }$.

  • For any $i\in P$ and $x\in S_{\ast }^{\prime }$, the utility function is $u_i^{\left(\alpha \right)}\left(x\right):=u_i\left({\alpha }_x\right)$.

  Essentially, we rephrased $\Gamma$ as an extensive form game in which first the game ${\Gamma }^{\left(\alpha \right)}$ is played and then, if the outcome was $x$, the game ${\Gamma }^x$ is played. This, assuming the expected outcome of game ${\Gamma }^x$ is ${\alpha }_x$.

  It is also possible to generalize this construction by allowing multivalued $f$.

  Local Symmetry Solutions

  Given a game $\Gamma$, a local symmetry presolution (LSP) is recursively defined to be one of the following:

  • Assume there is $i\in P$ s.t. for any $j\in P\setminus i$, $|S_j|=1$. Then, any $x^{\ast }\in \arg {max}_{x\in S_{\ast }}{u}_i\left(x\right)$ is an LSP of $\Gamma$. [Effectively single player game]

  • Let $\{f_i:S_i\to S_i^{\prime }{\}}_{i\in P}$ be a coarse-graining of $\Gamma$ and for any $x\in S_{\ast }^{\prime }$, let ${\alpha }_x$ be an LSP of ${\Gamma }^x$. Let $\beta$ be an LSP of ${\Gamma }^{\left(\alpha \right)}$. Define $x^{\ast }\in S_{\ast }$ by $x_i^{\ast }:={\alpha }_{{\beta }_i}$. Then, $x^{\ast }$ is an LSP of $\Gamma$.

  • Let $G$ be a quasisymmetry of $\Gamma$ and let $y^{\ast }$ be an LSP of $\Gamma /G$. Then, $\gamma \left(y^{\ast }\right)$ is an LSP of $\Gamma$.

  It’s easy to see that an LSP always exists, because for any game we can choose a sequence of coarse-grainings whose effect is making the players choose their strategies one by one (with full knowledge of choices by previous players). However, not all LSP are “born equal”. It seems appealing to prefer LSP which use “more” symmetry. This can be formalized as follows.

  The way an LSP is constructed defines the set of coherent outcomes $A\subseteq S_{\ast }$, which is the set of outcomes compatible with the local symmetries^[4]. We define it recursively as follows:

  • For the LSP of an effectively single-player game, we set $A:=S_{\ast }$.

  • For a coarse-graining LSP, for any $x\in S_{\ast }^{\prime }$, let $A_x$ be the set of coherent outcomes of ${\Gamma }_x$ and let $B$ be the set of coherent outcomes of ${\Gamma }^{\left(\alpha \right)}$. We then define $A:=\{x\in S_{\ast }|f(x)\in B\text{and}x\in A_{f\left(x\right)}\}$. Here, $f:S_{\ast }\to S_{\ast }^{\prime }$ is defined by $f(x{)}_i:=f_i(x_i)$.

  • For a quasisymmetry LSP, let $B$ be the set of coherent outcomes of $\Gamma /G$. We then define $A:=\gamma \left(B\right)$.

  We define a local symmetry solution (LSS) to be an LSP whose set of coherent outcomes is minimal w.r.t. set inclusion.

  Examples

  • In the Prisoner’s Dilemma, the only LSS is Cooperate-Cooperate.

  • In Stag Hunt, the only LSS is Stag-Stag.

  • In Chicken, the only LSS is Swerve-Swerve. If we add a finite number of randomized strategies, it opens more possibilities.

  • In Battle of the Sexes, assuming a small payoff from going to your preferred place alone, the only LSS is: each player goes to their own preferred place. If we give each player a “coinflip” strategy, then I think the only LSS is both players using the coinflip.

  • Consider a pure cooperation game where both players have strategies $\{a,b\}$ and the payoff is 1 if they choose the same strategy and 0 if they choose different strategies. Then, any outcome is an LSS. I think that, if we add a “fair coinflip” strategy, any LSS has payoff at least $\frac{1}{4}$. I believe the LSS payoff will approach optimum if we (i) allow randomization (ii) make the game repeated with time long horizon and (iii) add some constraints on the symmetries, but I’ll leave this for a future post.

  • Consider a pure cooperation game where both players have strategies $\{a,b\}$, the payoff is 2 if both choose $a$, 1 if both choose $b$ and 0 if they choose different strategies. Then, the only LSS is $aa$.

  • In any game, an LSS guarantees each player at least their deterministic-maximin payoff. In particular, in a two-player zero-sum game in which a pure Nash equilibrium exists, any LSS is a Nash equilibrium.

  1. ^

     Note that flatness is in general not preserved under composition.

  2. ^

     I believe this can be interpreted as a category enriched over the category of sets and partial functions, with the monoidal structure given by Cartesian product of sets.

  3. ^

     An affine space, even.

  4. ^

     Maybe we can interpret the set of coherent outcomes as a sharp infradistirbution corresponding to the players’ joint belief about the outcome of the game.

• Vanessa KosoyApr 27, 2025, 8:06 AM

  LW: 2 AF: 2

  0

  AF

  on: Vanessa Kosoy’s Shortform

  Master post for multi-agency. See also.

• Vanessa KosoyApr 15, 2025, 7:10 PM

  LW: 11 AF: 5

  0

  AF

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  In (non-monotonic) infra-Bayesian physicalism, there is a vaguely similar asymmetry even though it’s formalized via a loss function. Roughly speaking, the loss function expresses preferences over “which computations are running”. This means that you can have a “positive” preference for a particular computation to run or a “negative” preference for a particular computation not to run^[1].

  1. ^

     There are also more complicated possibilities, such as “if P runs then I want Q to run but if P doesn’t run then I rather that Q also doesn’t run” or even preferences that are only expressible in terms of entanglement between computations.

• Vanessa KosoyApr 13, 2025, 5:41 PM

  4 points

  0

  in reply to: faul_sname’s comment on: New Paper: In­fra-Bayesian De­ci­sion-Es­ti­ma­tion Theory

  It’s roughly on the right track, but here are some inaccuracies in your description that stood out to me:

  • There is no requirement that the “hidden state space” is finite. It is perfectly fine to consider a credal set which is not a polytope (i.e. not a convex hull of a finite set of distributions).

  • The point of how market prices are computed, missing from your description, is that they prevent any bettor from making unbounded earnings (essentially, by making them bet against each other). This is the same principle as Garrabrant induction. In particular, this implies that if any of our models is true then the market predictions will converge to lying inside the corresponding credal set.

  • The market predictions do not somehow assume that “the parts of the universe we don’t observe are out to get us”. Thanks to the pessimistic better, they do satisfy the “not too optimistic condition”, but that’s “not too optimistic” relatively to the true environment.

  • Your entire description only talks about the “estimation” part, not about the “decision” part.

• Vanessa KosoyApr 12, 2025, 3:03 PM

  LW: 4 AF: 3

  0

  AF

  in reply to: Lorxus’s comment on: Some Rules for an Alge­bra of Bayes Nets

  I think that there are two different questions which might be getting mixed up here:

  Question 1: Can we fully classify all rules for which sets of Bayes nets imply other Bayes nets over the same variables? Naturally, this is not a fully rigorous question, since “classify” is not a well-defined notion. One possible operationalization of this question is: What is the computational complexity of determining whether a given Bayes net follows from a set of other Bayes nets? For example, if there is a set of basic moves that generate all such inferences then the problem is probably in NP (at least if the number of required moved can also be bounded).

  Question 2: What if we replace “Bayes nets” by something like “string diagrams in a Markov category”? Then there might be less rules (because maybe some rules hold for Bayes nets but not in the more abstract setting).

• Vanessa KosoyApr 10, 2025, 9:33 AM

  4 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: New Paper: In­fra-Bayesian De­ci­sion-Es­ti­ma­tion Theory

  Thank you <3

  Any chance of more exposition for those of us less cognitively-inclined? =)

  Read the paper! :)

  It might seem long at first glance, but all the results are explained in the first 13 pages, the rest is just proofs. If you don’t care about the examples, you can stop on page 11. Naturally, I welcome any feedback on the exposition there.

New Paper: In­fra-Bayesian De­ci­sion-Es­ti­ma­tion Theory

• Vanessa KosoyApr 9, 2025, 6:40 AM

  LW: 6 AF: 4

  −3

  AF

  in reply to: Cole Wyeth’s comment on: abramdem­ski’s Shortform

  I think that in 2 years we’re unlikely to accomplish anything that leaves a dent in P(DOOM), with any method, but I also think it’s more likely than not that we actually have >15 years.

  As to “completing” the theory of agents, I used the phrase (perhaps perversely) in the same sense that e.g. we “completed” the theory of information: the latter exists and can actually be used for its intended applications (communication systems). Or at least in the sense we “completed” the theory of computational complexity: even though a lot of key conjectures are still unproven, we do have a rigorous understanding of what computational complexity is and know how to determine it for many (even if far from all) problems of interest.

  I probably should have said “create” rather than “complete”.

• Vanessa KosoyApr 8, 2025, 5:49 PM

  LW: 12 AF: 3

  0

  AF

  in reply to: Cole Wyeth’s comment on: abramdem­ski’s Shortform

  (Summoned by @Alexander Gietelink Oldenziel)

  I don’t understand this comment. I usually don’t think of “building a safer LLM agent” as a viable route to aligned AI. My current best guess about how to create aligned AI is Physicalist Superimitation. We can imagine other approaches, e.g. Quantilized Debate, but I am less optimistic there. More importantly, I believe that we need to complete the theory of agents first, before we can have strong confidence about which approaches are more promising.

  As to heuristic implementations of infra-Bayesianism, this is something I don’t want to speculate about in public, it seems exfohazardous.

• Vanessa KosoyApr 8, 2025, 3:16 PM

  LW: 2 AF: 2

  0

  AF

  in reply to: Lorxus’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Somehow being able to derive all relevant string diagram rewriting rules for latential string diagrams, starting with some fixed set of equivalences?

  What are “latential” string diagrams?

  What does it it mean that you can’t derive them all from a “fixed” set? Do you imagine some strong claim e.g. that the set of rewriting rules being undecidable, or something else?

  Two Bayes nets are of the same Markov equivalence class when they have precisely the same set of conditionality relations holding on them (and by extension, precisely the same undirected skeleton).

  Okay, so this is not what you care about? Maybe you are saying the following: Given two diagrams X,Y, we want to ask whether any distribution compatible with X is compatible with Y. We don’t ask whether the converse also holds. This is a certain asymmetric relation, rather than an equivalence.

• Vanessa KosoyApr 8, 2025, 11:45 AM

  LW: 2 AF: 2

  0

  AF

  in reply to: Lorxus’s comment on: Some Rules for an Alge­bra of Bayes Nets

  I found the above comment difficult to parse.

  that’s not a thing that really happens?

  What is the thing that doesn’t happen? Reading the rest of the paragraph only left me more confused.

  we don’t quite care about Markov equivalence class

  What do you mean by “Markov equivalence class”?

• Vanessa KosoyMar 25, 2025, 8:23 AM

  LW: 2 AF: 2

  0

  AF

  in reply to: Cole Wyeth’s comment on: Vanessa Kosoy’s Shortform

  Thanks for this!

  What I was saying up there is not a justification of Hurwicz’ decision rule. Rather, it is that if you already accept the Hurwicz rule, it can be reduced to maximin, and for a simplicity prior the reduction is “cheap” (produces another simplicity prior).

  Why accept the Hurwicz’ decision rule? Well, at least you can’t be accused of a pessimism bias there. But if you truly want to dig deeper, we can start instead from an agent making decisions according to an ambidistribution, which is a fairly general (assumption-light) way of making decisions. I believe that a similar argument (easiest to see in the LF-dual cramble set representation) would allow reducing that to maximin on infradistributions (credal sets).

  To make such an approach even more satisfactory, it would be good to add a justification for a simplicity ambi/​infra-prior. I think this should be possible by arguing from “opinionated agents”: the ordinary Solomonoff prior is the unique semicomputable one that dominates all semicomputable measures, which decision-theoretically corresponds to something like “having preferences about as many possible worlds as we can”. Possibly, the latter principle formalized can be formalized into something which ends up picking out an infra-Solomonoff prior (and, replacing “computability” by a stronger condition, some other kind of simplicity infra-prior).

• Vanessa KosoyMar 24, 2025, 9:03 AM

  LW: 2 AF: 2

  0

  AF

  in reply to: Chris van Merwijk’s comment on: Com­po­si­tional lan­guage for hy­pothe­ses about computations

  You now understand correctly. The reason I switch to colored operads is to add even more generality. My key use case is when the operad consists of terms-with-holes in a programming language, in which case the colors are the types of the terms/​holes.

• Vanessa KosoyMar 23, 2025, 5:53 PM

  LW: 14 AF: 5

  0

  AF

  in reply to: Vanessa Kosoy’s comment on: Vanessa Kosoy’s Shortform

  The following are my thoughts on the definition of learning in infra-Bayesian physicalism (IBP), which is also a candidate for the ultimate prescriptive agent desideratum.

  In general, learning of hypotheses about the physical universe is not possible because of traps. On the other hand, learning of hypotheses about computable mathematics is possible in the limit of ample computing resources, as long as we can ignore side effects of computations. Moreover, learning computable mathematics implies approximating Bayesian planning w.r.t the prior about the physical universe. Hence, we focus on this sort of learning.

  We consider an agent comprised of three modules, that we call Simulator, Learner and Controller. The agent’s history consists of two phases. In the Training phase, the Learner interacts with the Simulator, and in the end produces a program for the Controller. In the Deployment phase, the Controller runs the program.

  Roughly speaking:

  • The Simulator is a universal computer whose function is performing computational experiments, which we can think of as “thought experiments” or coarse-grained simulations of potential plans. It receives commands from the Learner (which computations to run /​ threads to start/​stop) and reports to the Learner the results. We denote the Simulator’s input alphabet by $I_S$ and output alphabet by $O_S$.

  • The Learner is the machine learning (training) module. The algorithm whose desideratum we’re specifying resides here.

  • The Controller (as in “control theory”) is a universal computer connected to the agent’s external interface (environmental actions $A$ and observations $O$). It’s responsible for real-time interaction with the environment, and we can think of it as the learned policy. It is programmed by the Learner, for which purpose it has input alphabet $I_C$.

  We will refer to this as the SiLC architecture.

  Let $H\subseteq \square \Gamma$ be our hypothesis class about computable mathematics. Let $\xi :\Gamma \to \square 2^{\Gamma }$ be our prior about the physical universe^[1]. These have to satisfy the coherence conditions

  $$\forall y\in \Gamma ,\alpha \in supp\xi \left(y\right):y\in \alpha$$$$\forall y,y^{\prime }\in \Gamma ,\theta \in \xi \left(y\right):{\chi }_{y^{\prime }\in \alpha }\theta \in \xi \left(y^{\prime }\right)$$$$\forall \Theta \in H,\theta \in \Theta ,s:\Gamma \to \Gamma ,\varphi :\Gamma \to {\Delta }^c\Gamma :\varphi ⪯\xi ⟹{pr}_{\Gamma }{\chi }_{{el}^{\Gamma }}(s\times {id}_{2^{\Gamma }}{)}_{\ast }(\theta ⋉\varphi )\in \Theta$$

  Here, $\varphi ⪯\xi$ means that $\forall y\in \Gamma :\varphi \left(y\right)\in \xi \left(y\right)$.

  Together, these ensure that $\Theta ⋉\xi$ is a coherent IBP hypothesis. Notice that for any ${\xi }_0:\Gamma \to \square 2^{\Gamma }$ satisfying the first condition^[2], there is a unique minimal coherent $\xi :\Gamma \to \square 2^{\Gamma }$ s.t. $\forall y\in \Gamma :{\xi }_0\left(y\right)\subseteq \xi \left(y\right)$. Moreover, given a coherent $\xi$ and any ${\Theta }_0\in \square \Gamma$, there is a unique minimal coherent $\Theta \in \square \Gamma$ s.t. ${\Theta }_0\subseteq \Theta$.

  The duration of the Training phase will be denoted by $\tau \in N$^[3]. We can think of it as “computational time”.

  Let the source codes of the Learner (obtained by quining), the Simulator and the Controller respectively be denoted by

  $$D_L:\Gamma \times N\times (I_S\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$$$$D_S:\Gamma \times I_S^{+}\to O_S$$$$D_C:\Gamma \times I_C^{\ast }\times O^{\ast }\to A$$

  Here, the $N$ argument of $D_L$ corresponds to $\tau$ and ${\Delta }_Q$ is a probability distribution in which all probabilities are rational numbers^[4].

  We assume that the simulator can indeed run any computation, and that any given halting computation would run fast for $\tau \gg 0$. These are assumptions on $D_S$ (or, on some combination of (i) $D_S$, (ii) the definition of $\Gamma$, and (iii) the support of all $\Theta \in H$) that we will not spell out here.

  We will say that a policy is a mapping of type $(I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$ and a metapolicy is a mapping of type $N\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$.

  Given any $D_L^{\prime }:\Gamma \times N\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$, we can compose it with $D_S$ and $D_C$ in the obvious way^[5] to yield

  $$D_S\otimes D_L^{\prime }\otimes D_C:\Gamma \times N\times (A\times O{)}^{\ast }\to {\Delta }_{Q}A$$

  In particular, we can take $D_L^{\prime }=\mu$ for some metapolicy $\mu$ by postulating no dependence on the $\Gamma$ argument.

  Denote by $P$ the set of all policies. Given metapolicy $\mu$ and $\pi \in P$, we define ${\mu }^{\pi }:[\tau +1]\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$ by

  $${\mu }^{\pi }(k,pq):=\{\begin{array}{c}\mu (k,pq)\text{if}k<\tau \\ \pi \left(pq\right)\text{if}k=\tau \end{array}$$

  Given any $\nu :[\tau +1]\times (I_S\times I_C\times O_S{)}^{\ast }\to {\Delta }_Q(I_S\times I_C)$, we say that $y\in \Gamma$ is a ${\nu }^{\tau }$-consistent counterpossible when the following conditions hold:

  • For all $k<\tau$ and $h\in (I_S\times I_C\times O_S{)}^{<k}$, $D_L(y,k,pq)=\nu (k,pq)$

  • For all $k\le \tau$ and $h\in (A\times O{)}^{\ast }$, $(D_S\otimes D_L\otimes D_C)(y,k,pq)=(D_S\otimes \nu \otimes D_C)(y,k,pq)$

  We denote by $C_{\nu }^{\tau }\subseteq \Gamma$ the set of ${\nu }^{\tau }$-consistent counterpossibles.

  A (deterministic) copolicy is a mapping of signature $I_S^{+}\to O_S$. We denote the set of copolicies by $C$. Given a policy $\pi$ and a copolicy $\nu$, we define $\pi \hat{\otimes }\nu \in \Delta (I_S\times I_C\times O_S{)}^{\ast }$ in the obvious way. Given policies ${\pi }_1,{\pi }_2$, we define their total variation distance^[6] to be

  $$d_{\text{TV}}({\pi }_1,{\pi }_2):=\underset{\nu \in C}{max}{d}_{\text{TV}}({\pi }_1\hat{\otimes }\nu ,{\pi }_2\hat{\otimes }\nu )$$

  Given $\Xi \in \square (\Gamma \times 2^{\Gamma })$, $f:\Gamma \times 2^{\Gamma }\to [0,\infty )$, $\tau \in N$ and metapolicy $\mu$, we will use the notation

  $$E_{\Xi }[f\mid \mid {\mu }^{\tau }]:=\underset{\pi \in P}{min}(E_{\Xi }[f\cdot {\chi }_{C_{{\mu }^{\pi }}^{\tau }}]+d_{\text{TV}}(\pi ,\mu (\tau \left)\right))$$

  Intuitively, $E_{\Xi }[f\mid \mid {\mu }^{\tau }]$ should be thought as the counterfactual expectation of loss function $f$ assuming metapolicy $\mu$, while adding a “bonus” to account for “fair” treatment of randomization by the agent. More on that below.

  Given a metapolicy $\mu$ and $\tau \in N$, we define $E_{\mu }^{\tau }\subseteq 2^{\Gamma }$ by

  $$E_{\mu }^{\tau }:=\{\alpha \in 2^{\Gamma }\mid \exists \pi \in P:C_{{\mu }^{\pi }}^{\tau }\cap \alpha =\varnothing \}$$

  Intuitively, $E_{\mu }^{\tau t}$ is the set of universe states for which at least one copy of the agent exists which followed the metapolicy $\mu$ until computational time $\tau$.

  Given a loss function $l:N\times 2^{\Gamma }\to [0,1]$^[7] (which we allow to explicitly depend on computational time for greater generality), the learning condition on a metapolicy ${\mu }^{\ast }$ and hypothesis $\Theta \in H$ is

  $$\sum _{\tau <N}{E}_{{\xi }_{\ast }\Theta }\left[l\right(\tau )\mid \mid {\mu }^{\ast \tau }]\le \sum _{\tau <N}(\underset{\pi \in P}{min}{E}_{{\xi }_{\ast }\Theta }\left[l\right(\tau )\cdot {\chi }_{C_{{\mu }^{\ast \pi }}^{\tau }}]+E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]ϵ(\Theta ,N\left)\right)$$

  Here, $ϵ$ is the “regret bound” function which should vanish in the $N\to \infty$ limit.

  Some remarks on the particulars of this definition:

  • There are several reasons we impose $(D_S\otimes D_L\otimes D_C)(y,k,pq)=(D_S\otimes \nu \otimes D_C)(y,k,pq)$ rather than $D_L(y,k,pq)=\nu (k,pq)$:

  • First, we want to ignore the side effects of running computations on the Simulator (both causal side effects and “mindcrime”, i.e. the direct contribution of those computations to $l$). Because, taking side effects into account is usually inconsistent with the unlimited experimentation needed for learning.

  • Second, learning requires trusting the reports of the Simulator, which means we should only impose the policy on copies of $D_L$ that are actually connected to $D_S$.

  • Third, we should also be able to trust the Controller, because otherwise we lose the semantic grounding of the agent’s external interface. (Even though this is not necessary for learning per se.).

  • On the other hand, we impose $D_L(y,k,pq)=\nu (k,pq)$ in the computational past because that’s valid additional information that doesn’t interfere with the learning or decision theory.

  • The learning criterion treats computational time myopically, so that we won’t have to worry about traps in computational time.

  • The reason we need randomization is, it’s often necessary for efficient learning. In the simplest non-trivial examples, we learn by IID sampling a distribution over computations (e.g. we simulate the interaction between a particular policy and our physical prior $\xi$). If we sampled deterministically instead, Murphy would be able to fool us by changing behavior precisely at the sampled instances.

  • The reason we need $E_{\Xi }[f\mid \mid {\mu }^{\tau t}]$ is, randomization only helps if low probability events can be ignored. However, if sufficiently many copies of the agents are instantiated, even a low probability even would be detectable. Hence, we use a “forgiving” metric that assigns low loss even to distributions that technically have high loss but are close to a different distribution with low loss.

  • We can consider Newcombian problems where Omega makes decisions based on the agent’s action probabilities. I suspect that if Omega’s policy is Lipschitz in the agent policy, the behavior advised by the $E_{\Xi }[f\mid \mid {\mu }^{\tau t}]$ counterfactual will converge to FDT-optimal in the limit of sufficiently many iterations.

  • Both in the case of ignoring side effects of computations and in the case of the treatment of randomization, we can be accused of departing from priorism (“updatelessness”). However, I think that this departure is justified. In the original TDT paper, Yudkowsky addressed the “Omega rewards irrationality” objection by postulating that, a decision problem is “fair” when it only depends on the agent’s decisions rather than on how the agent makes those decisions. Here, we use the same principle: the agent should not be judged based on its internal thought process (side effects), and it should in some sense be judged based on its decisions rather than the probabilities assigned to those decisions.

  • Also about priorism, this kind of agents will not endorse iterated-in-computational-time “logical” counterfactual mugging when the same coin is reused indefinitely, but will endorse it when a new coin is used every time, for an appropriate definition of “new” (or e.g. we switch to a new coin every $k$ rounds). Arguably, this solves the tension between priorism and learning observed by Demski. Formulating the precise criterion when Learning-IBP behavior converges to priorist /​ FDT-optimal is left for further research.

  • The dependence of $ϵ(\Theta ,N)$ on $\Theta$ should ultimately involve some kind of description complexity. However, it will also involve something in the vein of “what are the computational resource bounds, according to the belief $\Theta$, for running certain computations, selected for their importance in testing $\Theta$”. In particular, we won’t require the agent to learning anything about non-halting computations. Indeed, any hypothesis about such computations will either assert a time bound on running the non-halting computations (in which case it is false) or will fail to assert any such bound, in which case its learning complexity is known to be infinite.

  • We could make do without the $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$ factor but that would make the learning criterion weaker. The presence of this factor means that, roughly speaking, regret should be low even conditional on the agent existing, which seems like a reasonable demand.

  • Given an AI designed along these principles, we might worry about the impact of the side effects that are ignored. Maybe these can produce non-Cartesian daemons. However, during the Training phase, the algorithm has no access to external observation, which arguably makes it unlikely anything inside it can learn how to cyberattack. Moreover, during the Deployment phase, any reason for concern would be mediated by the particular algorithm the Controller runs (rather than the particulars of how it’s implemented), which is what we do take into account in our optimization. Finally, the agent might be able to self-modify to make itself safer: we can even intentionally give it the means to do so (as part of its action space $A$). This probably requires careful prior-shaping to work well.

  1. ^

     This framework assumes all our hypotheses are disintegrable w.r.t. the factorization into $\Gamma$ and $2^{\Gamma }$. It is an interesting question to understand whether we should or can relax this assumption.

  2. ^

     For example, we can imagine ${\xi }_0$ to be a Solomonoff-like prior along the following lines. Every hypothesis comprising ${\xi }_0$ is defined by a Turing machine $M$ with access to two oracles representing $y,y^{\prime }\in \Gamma$ and two tapes of random and “ambiguous” bits respectively.${\xi }_0\left(y\right)$ is defined by running $M$ with one oracle fielding queries about $y$ (i.e. we given a program $P$ we can request to know its counterpossible output $P^y$) and the other oracle fielding queries about some $y^{\prime }$ s.t. we want to decide whether $y^{\prime }\in \alpha$ for $\alpha \sim \theta \in {\xi }_0\left(y\right)$. $M$ is only allowed to return NO if there was at least one query to which the two oracles gave different answers.

  3. ^

     We use the “duration” interpretation for simplicity, but more generally $\tau$ can be some parameter controlling the computing resources available in the Training phase, and we can also allow the computing resources of the Controller to scale with $\tau$.

  4. ^

     The reason we restrict to rational numbers is because we need a notion of computing the distribution. It is in principle possible to generalize further to computable numbers. On the other hand, it might be more realistic to constrain even further to e.g. dyadic rationals (which can be implemented via fair coinflips). We stick to $Q$ for simplicity.

  5. ^

     We let the Learner interact with the Simulator for $\tau$ timesteps, producing some output $g\in I_C^{\tau }$, and then run the Controller with $g$ as an input.

  6. ^

     This is not technically a distance since it is possible to have $d_{\text{TV}}({\pi }_1,{\pi }_2)=0$ if ${\pi }_1\ne {\pi }_2$ so long as ${\pi }_1$ and ${\pi }_2$ only disagree on histories that are inconsistent with these policies. Such ${\pi }_1$ and ${\pi }_2$ are morally equivalent.

  7. ^

     We could also allow $l$ to have a $\Gamma$ argument, but then we would have to remove the $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$ factor from the learning condition, because the choice of policy would matter intrinsically even if the agent doesn’t exist. Alternatively, we could modify the definition of $C_{\mu }^{\tau }$ to avoid that. Or perhaps use some normalization factor more complicated than $E_{{\xi }_{\ast }\Theta }\left[{\chi }_{E_{{\mu }^{\ast }}^{\tau }}\right]$.

  What links here?

  • I “in­vented” semimea­sure the­ory and all I got was im­pre­cise prob­a­bil­ity theory by Cole Wyeth (Apr 3, 2025, 4:33 PM; 14 points)