StrivingForLegibility

Karma: 376

StrivingForLegibility Jan 22, 2024, 6:00 PM
1 point
0
in reply to: Richard_Kennaway’s comment on: Incorporating Justice Theory into Decision Theory
Oops, when I heard about it I’d gotten the impression that this had been adopted by at least one AI firm, even a minor one, but I also can’t find anything suggesting that’s the case. Thank you!
It looks like OpenAI has split into a nonprofit organization and a “capped-profit” company.
The fundamental idea of OpenAI LP is that investors and employees can get a capped return if we succeed at our mission, which allows us to raise investment capital and attract employees with startup-like equity. But any returns beyond that amount—and if we are successful, we expect to generate orders of magnitude more value than we’d owe to people who invest in or work at OpenAI LP—are owned by the original OpenAI Nonprofit entity.
OpenAI Nonprofit could act like the Future of Life Instutute’s proposed Windfall Trust, and a binding commitment to do so would be a Windfall Clause. They could also do something else prosocial with those profits, consistent with their nonprofit status.

StrivingForLegibility Jan 22, 2024, 4:37 PM
5 points
0
in reply to: Richard_Kennaway’s comment on: Incorporating Justice Theory into Decision Theory
I think we agree that in cases where competition is leading to good results, no change to the dynamics is called for.
We probably also agree on a lot of background value judgements like “when businesses become more competitive by spending less on things no one wants, like waste or pollution, that’s great!” And “when businesses become more competitive by spending less on things people want, like fair wages or adequate safety, that’s not great and intervention is called for.”
One case where we might literally want to distribute resources from the makers of a valuable product, to their competitors and society at large, is the development of Artificial General Intelligence (AGI). One of the big causes for concern here is that the natural dynamics might be winner-take-all, leading to an arms race that sacrifices spending on safety in favor of spending on increased capabilities or an earlier launch date.
If instead all AGI developers believed that the gains from AGI development would be spread out much more evenly, this might help to divert spending away from increasing capabilities and deploying as soon as possible, and towards making sure that deployment is done safely. ~~Many AI firms have already voluntarily signed~~ ~~Windfall Clauses, committing to share significant fractions of the wealth generated by successful AGI development.~~
EDIT: At the time of writing, it looks like Windfall Clauses have been advocated for but not adopted. Thank you Richard_Kennaway for the correction!

StrivingForLegibility Jan 22, 2024, 3:51 PM
5 points
0
in reply to: Dagon’s comment on: Incorporating Justice Theory into Decision Theory
For games without these mechanisms, the rational outcomes don’t end up that pleasant. Except sometimes, with players who have extra-rational motives.
I think we agree that if a selfish agent needs to be forced to not treat others poorly, in the absence of such enforcement they will treat others poorly.
It also seems like in many cases, selfish agents have an incentive to create exactly those mechanisms ensuring good outcomes for everyone, because it leads to good outcomes for them in particular. A nation-state comprised entirely of very selfish people would look a lot different from any modern country, but they face the same instrumental reasons to pool resources to enforce laws. The more inclined their populace is towards mistreatment in the absence of enforcement, the higher those enforcement costs need to be in order to achieve the same level of good treatment.
I also think “fairness” is a Schelling point that even selfish agents can agree to coordinate around, in a way that they could never be aligned on “maximizing Zaire’s utility in particular.” They don’t need to value fairness directly to agree that “an equal split of resources is the only compromise we’re all going to agree on during this negotiation.”
So I think my optimism comes from at least two places:
- Even utterly selfish agents still have an incentive to create mechanisms enforcing good outcomes for everyone.
- People have at least some altruism, and are willing to pay costs to prevent mistreatment of others in many cases.

StrivingForLegibility Jan 22, 2024, 12:03 AM
1 point
1
in reply to: Richard_Kennaway’s comment on: Incorporating Justice Theory into Decision Theory
That sounds reasonable to me! This could be another negative externality that we judge to be acceptable, and that we don’t want to internalize. Something like “if you break any of these rules, (e.g. worker safety, corporate espionage, etc.) then you owe the affected parties compensation. But as long as you follow the rules, there is no consensus-recognized debt.”

StrivingForLegibility Jan 2, 2024, 3:52 AM
1 point
0
in reply to: osten’s comment on: Optimization Markets
It seems straightforward! Kaggle is the closest example I’ve been able to think of. But yes that’s totally the sort of thing that I think would constitute an optimization market!

StrivingForLegibility Dec 26, 2023, 4:53 PM
7 points
2
in reply to: mishka’s comment on: How Emergency Medicine Solves the Alignment Problem
Absolutely! I have less experience on the “figuring out what interventions are appropriate” side of the medical system, but I know of several safety measures they employ that we can adapt for AI safety.
For example, no actor is unilaterally permitted to think up a novel intervention and start implementing it. They need to convince a institutional review board that the intervention has merit, and that a clinical trial can be performed safely and ethically. Then the intervention needs to be approved by a bunch of bureaucracies like the FDA. And then medical directors can start incorporating that intervention into their protocols.
The AI design paradigm that I’m currently most in favor of, and that I think is compatible with the EMS Agenda 2050, is Drexler’s Comprehensive AI Services (CAIS). Where a bunch of narrow AI systems are safely employed to do specific, bounded tasks. A superintelligent system might come up with amazing novel interventions, and collaborate with humans and other superintelligent systems to design a clinical trial for testing them. Every party along the path from invention to deployment can benefit from AI systems helping them perform their roles more safely and effectively.

StrivingForLegibility Dec 22, 2023, 5:41 AM
4 points
0
in reply to: Cole Wyeth’s comment on: A Decision Theory Can Be Rational or Computable, but Not Both
This is a much more nuanced take! At the beginning of Chapter 6, Jan proposes restricting our attention to agents which are limit computable.
Our agents are useless if they cannot be approximated in practice, i.e., by a regular Turing machine. Therefore we posit that any ideal for a ‘perfect agent’ needs to be limit computable ( $Δ_{2}^{0}$ ).
This seems like a very reasonable restriction! Any implementation needs to be computable, but it makes sense to look for theoretic ideals which can be approximated.

StrivingForLegibility Dec 22, 2023, 4:47 AM
1 point
0
in reply to: RogerDearnaley’s comment on: A Decision Theory Can Be Rational or Computable, but Not Both
Yes! I’m a fan of Yudkowsky’s view that the sensation of free will is the sensation of “couldness” among multiple actions. When it feels like I could do one thing or another, it feels like I have free will. When it feels like I could have chosen differently, it feels like I chose freely.
I suspect that an important ingredient of the One True Decision Theory is being shaped in such a way that other agents, modelling how you’ll respond to different policies they might implement, find it in their interest to implement policies which treat you fairly.

StrivingForLegibility Nov 25, 2023, 5:16 AM
1 point
0
in reply to: Cleo Nardo’s comment on: Game Theory without Argmax [Part 2]
Got it, I misunderstood the semantics of what $B (x)$ was supposed to capture. I thought the elements needed to be mutual best-responses. Thank you for the clarification, I’ve updated my implementation accordingly!

StrivingForLegibility Nov 24, 2023, 2:59 AM
2 points
−6
in reply to: rotatingpaguro’s comment on: Game Theory without Argmax [Part 2]
Edit: Cleo Nardo has confirmed that they intended $\prod i \in I$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B (x)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is based on my initial (wrong) interpretation of $B (x)$ .
I write the original expression, and your expression rewritten using the OP’s notation:
Original: $B : x \mapsto \prod i \in I ψ_{i} (g \circ U_{i} (x))$
Yours: $\begin{matrix} B (x) & = ⋂ i \in I {x (i \mapsto α) : α \in ψ_{i} (g \circ U_{i} (x))} = ⋂ i \in I U_{i} (x) (ψ_{i} (g \circ U_{i} (x))) \end{matrix}$
(I’m using the notation that a function applied to a set is the image of that set.)
This is a totally clear and valid rewriting using that notation! My background is in programming and I spent a couple minutes trying to figure out how mathematicians write “apply this function to this set.”
I believe the way that $B (x)$ is being used is to find Nash equilibria, using Cleo’s definition 6.5:
Like before, the $Ψ$ -nash equilibria of $g$ is the set of option-profiles $x \in X$ such that $x \in B (x)$ .
These are going to be option-profiles where “not deviating” is considered optimal by every player simultaneously. I agree with your conclusion that this leads $B (x)$ to take on values that are either ${}$ or ${x}$ . When $B (x) = {}$ , this indicates that $x$ is not a Nash equilibrium. When $B (x) = {x}$ , we know that $x$ is a Nash equilibrium.

StrivingForLegibility Nov 23, 2023, 7:49 AM
2 points
−3
in reply to: rotatingpaguro’s comment on: Game Theory without Argmax [Part 2]
Edit: Cleo Nardo has confirmed that they intended $\prod i \in I$ to mean the cartesian product of sets, the ordinary thing for that symbol to mean in that context. I misunderstood the semantics of what $B (x)$ was intended to represent. I’ve updated my implementation to use the intended cartesian product when calculating the best response function, the rest of this comment is my initial (wrong) interpretation of $B (x)$ .
I needed to go back to one of the papers cited in Part 1 to understand what that $\prod i \in I$ was doing in that expression. I found the answer in A Generalization of Nash’s Theorem with Higher-Order Functionals. I’m going to do my best to paraphrase Hedges’ notation into Cleo’s notation, to avoid confusion.
TLDR: $B (x)$ is picking out the set of option-profiles $P (X)$ that are simultaneously best-responses by all players to that option-profile $x$ . It does this by considering all of the option-profiles that can result by each player best-responding, then takes the intersection of those sets.
On page 6, Hedges defines the best response correspondence $B \in X \to P (X)$
$B (x) = ⋂ i \in I B_{i} (x)$
Where
$B_{i} \in X \to P (X)$
Hedges builds up the idea of Nash Equilibria using quantifiers rather than optimizers, (like $max$ rather than $argmax$ ), but I believe the approaches are equivalent. Unpacking $B : x \mapsto \prod i \in I ψ_{i} (g \circ U_{i} (x))$ from the inside out:
$U_{i} (x) \in X_{i} \to X$
$g \circ U_{i} (x) \in X_{i} \to R$
That makes $g \circ U_{i} (x$ ) a $ψ_{i}$ -task. Since $ψ_{i} \in (X_{i} \to R) \to P (X_{i})$ , we know that $ψ_{i} (g \circ U_{i} (x)) \in P (X_{i})$ .
This is where I had to go looking through papers. What sort of product takes a set of best-responses from each player, relative to a given option-profile, and returns a set of option-profiles that are simultaneously regarded by each player as a best-response? I thought about just taking the Cartesian product of the sets, but that wouldn’t get us only the mutual best-responses.
Let’s call the way that each player maps option-profiles to best-responses $b_{i} \in X \to P (X_{i})$ . This is exactly the sets we want to take the product of:
$b_{i} (x) = ψ_{i} (g \circ U_{i} (x))$
Hedges introduces notation on page 3 to handle the operation of taking an option-profile, varying one player’s option, and leaving the rest the same. Paraphrasing, Hedges defines $x (i \mapsto α) \in \prod j \in I X_{j}$ by
$x (i \mapsto α)_{j} = {\begin{matrix} α & if i = j x_{j} & o t h e r w i s e \end{matrix}$
You can read $x (i \mapsto α)$ as “give me a new copy of $x$ , where the $i$ th entry has been set to the value $α$ .” Hedges uses this to define the deviation maps equivalently to the way Cleo did. $U_{i} : X \to (X_{i} \to X)$
$U_{i} (x) (α) = x (i \mapsto α)$
The correspondences $B_{i} \in X \to P (X)$ take as input an option profile, and returns the set of option-profiles which are player $i$ ’s optimal unilateral deviations from that option profile. To construct $B_{i}$ from $b_{i}$ , we want to map $b_{i} (x) \in P (X_{i})$ to the option-profiles which deviate from $x$ in those exact ways.
$B_{i} (x) = {x (i \mapsto α) : α \in b_{i} (x)}$
We can then use Hedges’ $B (x) = ⋂ i \in I B_{i} (x)$ to get the best-response correspondence! We can unpack this to get a definition of $B$ using objects that Cleo defined, using that deviation notation from Hedges:
$B (x) = ⋂ i \in I {x (i \mapsto α) : α \in ψ_{i} (g \circ U_{i} (x))}$
Thank you Cleo for writing this article! This was my first introduction to Higher-Order Game Theory, and I wrote up an implementation in TypeScript to help me understand how all of the pieces fit together!