StrivingForLegibility

A Bench­mark for De­ci­sion Theories

Us­ing Threats to Achieve So­cially Op­ti­mal Outcomes

Best-Re­spond­ing Is Not Always the Best Response

Safety Data Sheets for Op­ti­miza­tion Processes

The Gears of Argmax

• StrivingForLegibilityJan 2, 2024, 3:52 AM

  1 point

  0

  in reply to: osten’s comment on: Op­ti­miza­tion Markets

  It seems straightforward! Kaggle is the closest example I’ve been able to think of. But yes that’s totally the sort of thing that I think would constitute an optimization market!

When Can Op­ti­miza­tion Be Done Safely?

Op­ti­miza­tion Markets

So­cial Choice The­ory and Log­i­cal Handshakes

Distributed Strate­gic Epistemology

Build­ing Trust in Strate­gic Settings

An On­tol­ogy for Strate­gic Epistemology

• StrivingForLegibilityDec 26, 2023, 4:53 PM

  7 points

  2

  in reply to: mishka’s comment on: How Emer­gency Medicine Solves the Align­ment Problem

  Absolutely! I have less experience on the “figuring out what interventions are appropriate” side of the medical system, but I know of several safety measures they employ that we can adapt for AI safety.

  For example, no actor is unilaterally permitted to think up a novel intervention and start implementing it. They need to convince a institutional review board that the intervention has merit, and that a clinical trial can be performed safely and ethically. Then the intervention needs to be approved by a bunch of bureaucracies like the FDA. And then medical directors can start incorporating that intervention into their protocols.

  The AI design paradigm that I’m currently most in favor of, and that I think is compatible with the EMS Agenda 2050, is Drexler’s Comprehensive AI Services (CAIS). Where a bunch of narrow AI systems are safely employed to do specific, bounded tasks. A superintelligent system might come up with amazing novel interventions, and collaborate with humans and other superintelligent systems to design a clinical trial for testing them. Every party along the path from invention to deployment can benefit from AI systems helping them perform their roles more safely and effectively.

How Emer­gency Medicine Solves the Align­ment Problem

• StrivingForLegibilityDec 22, 2023, 5:41 AM

  4 points

  0

  in reply to: Cole Wyeth’s comment on: A De­ci­sion The­ory Can Be Ra­tional or Com­putable, but Not Both

  This is a much more nuanced take! At the beginning of Chapter 6, Jan proposes restricting our attention to agents which are limit computable.

  Our agents are useless if they cannot be approximated in practice, i.e., by a regular Turing machine. Therefore we posit that any ideal for a ‘perfect agent’ needs to be limit computable (${\Delta }_2^0$).

  This seems like a very reasonable restriction! Any implementation needs to be computable, but it makes sense to look for theoretic ideals which can be approximated.

• StrivingForLegibilityDec 22, 2023, 4:47 AM

  1 point

  0

  in reply to: RogerDearnaley’s comment on: A De­ci­sion The­ory Can Be Ra­tional or Com­putable, but Not Both

  Yes! I’m a fan of Yudkowsky’s view that the sensation of free will is the sensation of “couldness” among multiple actions. When it feels like I could do one thing or another, it feels like I have free will. When it feels like I could have chosen differently, it feels like I chose freely.

  I suspect that an important ingredient of the One True Decision Theory is being shaped in such a way that other agents, modelling how you’ll respond to different policies they might implement, find it in their interest to implement policies which treat you fairly.

A De­ci­sion The­ory Can Be Ra­tional or Com­putable, but Not Both

• StrivingForLegibilityNov 25, 2023, 5:16 AM

  1 point

  0

  in reply to: Cleo Nardo’s comment on: Game The­ory with­out Argmax [Part 2]

  Got it, I misunderstood the semantics of what $B\left(x\right)$ was supposed to capture. I thought the elements needed to be mutual best-responses. Thank you for the clarification, I’ve updated my implementation accordingly!

• StrivingForLegibilityNov 24, 2023, 2:59 AM

  2 points

  −6

  in reply to: rotatingpaguro’s comment on: Game The­ory with­out Argmax [Part 2]

  Edit: Cleo Nardo has confirmed that they intended $\prod _{i\in I}$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B\left(x\right)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is based on my initial (wrong) interpretation of $B\left(x\right)$.

  I write the original expression, and your expression rewritten using the OP’s notation:

  Original: $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$

  Yours: $\begin{array}{cc}B\left(x\right)& =\bigcap _{i\in I}\left\{x\right(i\mapsto \alpha ):\alpha \in {\psi }_i(g\circ U_i(x\left)\right)\}\\ & =\bigcap _{i\in I}{U}_i\left(x\right)\left({\psi }_i\right(g\circ U_i\left(x\right)\left)\right)\end{array}$

  (I’m using the notation that a function applied to a set is the image of that set.)

  This is a totally clear and valid rewriting using that notation! My background is in programming and I spent a couple minutes trying to figure out how mathematicians write “apply this function to this set.”

  I believe the way that $B\left(x\right)$ is being used is to find Nash equilibria, using Cleo’s definition 6.5:

  Like before, the $\Psi$-nash equilibria of $g$ is the set of option-profiles $x\in X$ such that $x\in B\left(x\right)$.

  These are going to be option-profiles where “not deviating” is considered optimal by every player simultaneously. I agree with your conclusion that this leads $B\left(x\right)$ to take on values that are either $\left\{\right\}$ or $\left\{x\right\}$. When $B\left(x\right)=\left\{\right\}$, this indicates that $x$ is not a Nash equilibrium. When $B\left(x\right)=\left\{x\right\}$, we know that $x$ is a Nash equilibrium.

• StrivingForLegibilityNov 23, 2023, 7:49 AM

  2 points

  −3

  in reply to: rotatingpaguro’s comment on: Game The­ory with­out Argmax [Part 2]

  Edit: Cleo Nardo has confirmed that they intended $\prod _{i\in I}$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B\left(x\right)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is my initial (wrong) interpretation of $B\left(x\right)$.

  I needed to go back to one of the papers cited in Part 1 to understand what that $\prod _{i\in I}$ was doing in that expression. I found the answer in A Generalization of Nash’s Theorem with Higher-Order Functionals. I’m going to do my best to paraphrase Hedges’ notation into Cleo’s notation, to avoid confusion.

  TLDR: $B\left(x\right)$ is picking out the set of option-profiles $P\left(X\right)$ that are simultaneously best-responses by all players to that option-profile $x$. It does this by considering all of the option-profiles that can result by each player best-responding, then takes the intersection of those sets.

  On page 6, Hedges defines the best response correspondence $B\in X\to P\left(X\right)$

  $B\left(x\right)=\bigcap _{i\in I}{B}_i\left(x\right)$

  Where

  $B_i\in X\to P\left(X\right)$

  Hedges builds up the idea of Nash Equilibria using quantifiers rather than optimizers, (like $\text{max}$ rather than $\text{argmax}$), but I believe the approaches are equivalent. Unpacking $B:x\mapsto \prod _{i\in I}{\psi }_i(g\circ U_i(x\left)\right)$ from the inside out:

  $U_i\left(x\right)\in X_i\to X$

  $g\circ U_i\left(x\right)\in X_i\to R$

  That makes $g\circ U_i(x$) a ${\psi }_i$-task. Since ${\psi }_i\in (X_i\to R)\to P\left(X_i\right)$, we know that ${\psi }_i(g\circ U_i(x\left)\right)\in P\left(X_i\right)$.

  This is where I had to go looking through papers. What sort of product takes a set of best-responses from each player, relative to a given option-profile, and returns a set of option-profiles that are simultaneously regarded by each player as a best-response? I thought about just taking the Cartesian product of the sets, but that wouldn’t get us only the mutual best-responses.

  Let’s call the way that each player maps option-profiles to best-responses $b_i\in X\to P\left(X_i\right)$. This is exactly the sets we want to take the product of:

  $b_i\left(x\right)={\psi }_i(g\circ U_i(x\left)\right)$

  Hedges introduces notation on page 3 to handle the operation of taking an option-profile, varying one player’s option, and leaving the rest the same. Paraphrasing, Hedges defines $x(i\mapsto \alpha )\in \prod _{j\in I}{X}_j$ by

  $x(i\mapsto \alpha {)}_j=\{\begin{array}{cc}\alpha & \text{if}i=j\\ x_j& otherwise\end{array}$

  You can read $x(i\mapsto \alpha )$ as “give me a new copy of $x$, where the $i$th entry has been set to the value $\alpha$.” Hedges uses this to define the deviation maps equivalently to the way Cleo did. $U_i:X\to (X_i\to X)$

  $U_i\left(x\right)\left(\alpha \right)=x(i\mapsto \alpha )$

  The correspondences $B_i\in X\to P\left(X\right)$ take as input an option profile, and returns the set of option-profiles which are player $i$’s optimal unilateral deviations from that option profile. To construct $B_i$ from $b_i$, we want to map $b_i\left(x\right)\in P\left(X_i\right)$ to the option-profiles which deviate from $x$ in those exact ways.

  $B_i\left(x\right)=\left\{x\right(i\mapsto \alpha ):\alpha \in b_i(x\left)\right\}$

  We can then use Hedges’ $B\left(x\right)=\bigcap _{i\in I}{B}_i\left(x\right)$ to get the best-response correspondence! We can unpack this to get a definition of $B$ using objects that Cleo defined, using that deviation notation from Hedges:

  $B\left(x\right)=\bigcap _{i\in I}\left\{x\right(i\mapsto \alpha ):\alpha \in {\psi }_i(g\circ U_i(x\left)\right)\}$

  Thank you Cleo for writing this article! This was my first introduction to Higher-Order Game Theory, and I wrote up an implementation in TypeScript to help me understand how all of the pieces fit together!