SCP

• SCPOct 18, 2023, 11:34 PM

  LW: 3 AF: 3

  0

  AF

  in reply to: johnswentworth’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  On John’s-simplified-model-of-Thornley’s-proposal, we have complete preference orderings over trajectories-in-which-the-button-isn’t-pressed and trajectories-in-which-the-button-is-pressed, separately, but no preference between any button-pressed and button-not-pressed trajectory pair.

  For the purposes of this discussion, this is right. I don’t think the differences between this description and the actual proposal matter in this case.

  Represented as subagents, those incomplete preferences require two subagents:

  • One subagent always prefers button pressed to unpressed, is indifferent between unpressed trajectories, and has the original complete order on pressed trajectories.

  • The other subagent always prefers button unpressed to pressed, is indifferent between pressed trajectories, and has the original complete order on unpressed trajectories.

  I don’t think this representation is quite right, although not for a reason I expect to matter for this discussion. It’s a technicality but I’ll mention it for completeness. If we’re using Bradley’s representation theorem from section 2.1., the set of subagents must include every coherent completion of the agent’s preferences. E.g., suppose there are three possible trajectories. Let $p$ denote a pressed trajectory and $u_1,u_2$ two unpressed trajectories, where $u_1$ gets you strictly more coins than $u_2$. Then there’ll be five (ordinal) subagents, described in order of preference: $⟨u_1,u_2,p⟩$, $⟨u_1,u_{2}p⟩$, $⟨u_1,p,u_2⟩$ , $⟨u_{1}p,u_2⟩$, and $⟨p,u_1,u_2⟩$.

  But you and Thornley want to interpret “no preference” as “just choose arbitrarily/​randomly” rather than “don’t trade either for the other”, so… this agent just chooses arbitrarily/​randomly between the best-available pressed-option and the best-available unpressed-option? But that would imply that the agent is choosing (albeit arbitrarily/​randomly) between button-pressed and button-unpressed, which is not what we want, so presumably you’re imagining something else?

  Indeed, this wouldn’t be good, and isn’t what Thornley’s proposal does. The agent doesn’t choose arbitrarily between the best pressed vs unpressed options. Thornley’s proposal adds more requirements on the agent to ensure this. My use of ‘arbitrary’ in the post is a bit misleading in that context. I’m only using it to identify when the agent has multiple permissible options available, which is what we’re after to get TND. If no other requirements are added to the agent, and it’s acting under certainty, this could well lead it to actually choose arbitrarily. But it doesn’t have to in general, and under uncertainty and together with the rest of Thornley’s requirements, it doesn’t. (The requirements are described in his proposal.)

  With that flagged, on to the trammelling issue. The potential problem is that the two subagents might want to trade, so that the system sometimes tries to make the button be pressed (in cases which get relatively high pressed-utility), and sometimes tries to make the button be unpressed (in cases which get relatively high unpressed-utility). (E.g. the pink arrows in the diagram.) And… locking in a trajectory at the start doesn’t seem to help that problem at all? Like, the subagents do that trading in logical time (i.e. time zero), that adds preferences, and then sometimes they lock in a plan which involves manipulating the button.

  I’ll first flag that the results don’t rely on subagents. Creating a group agent out of multiple subagents is possibly an interesting way to create an agent representable as having incomplete preferences, but this isn’t the same as creating a single agent whose single preference relation happens not to satisfy completeness.

  That said, I will spend some more time thinking about the subagent idea, and I do think collusion between them seems like the major initial hurdle for this approach to creating an agent with preferential gaps.

• SCPOct 18, 2023, 10:25 PM

  LW: 3 AF: 3

  0

  AF

  in reply to: johnswentworth’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  Why does wanting to maintain indifference to shifting probability mass between (some) trajectories, imply that we care about ex-ante permissibility?

  The ex-ante permissible trajectories are the trajectories that the agent lacks any strict preference between. Suppose the permissible trajectories are {A,B,C}. Then, from the agent’s perspective, A isn’t better than B, B isn’t better than A, and so on. The agent considers them all equally choiceworthy. So, the agent doesn’t mind picking any one of them over any other, nor therefore switching from one lottery over them with some distribution to another lottery with some other distribution. The agent doesn’t care whether it gets A versus B, versus an even chance of A or B, versus a one-third chance of A, B, or C.^[1]

  Suppose we didn’t have multiple permissible options ex-ante. For example, if only A was permissible, then the agent would dislike shifting probability mass away from A and towards B or C—because B and C aren’t among the best options.^[2] So that’s why we want multiple ex-ante permissible trajectories: it’s the only way to maintain indifference to shifting probability mass between (those) trajectories.

  [I’ll respond to the stuff in your second paragraph under your longer comment.]

  1. ^

     The analogous case with complete preferences is clearer: if there are multiple permissible options, the agent must be indifferent between them all (or else the agent would be fine picking a strictly dominated option). So if $n$ options are permissible, then $u\left(x_i\right)=u\left(x_j\right)\forall i,j\in N_n$. Assuming expected utility theory, we’ll then of course have ${\sum }_{i=1}^{n}u\left(x_i\right)p\left(x_i\right)={\sum }_{i=1}^{n}u\left(x_i\right)p^{\prime }\left(x_i\right)$ for any probability functions $p,p^{\prime }$. This means the agent is indifferent to shifting probability mass between the permissible options.

  2. ^

     This is a bit simplified but it should get the point across.

• SCPOct 16, 2023, 9:35 PM

  LW: 7 AF: 5

  4

  AF

  in reply to: johnswentworth’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  This is a tricky topic to think about because it’s not obvious how trammelling could be a worry for Thornley’s Incomplete Preference Proposal. I think the most important thing to clarify is why care about ex-ante permissibility. I’ll try to describe that first (this should help with my responses to downstream concerns).
  

  Big picture

  Getting terminology out of the way: words like “permissibility” and “mandatory” are shorthand for rankings of prospects. A prospect is permissible iff it’s in a choice set, e.g. by satisfying DSM. It’s mandatory iff it’s the sole element of a choice set.

  To see why ex-ante permissibility matters, note that it’s essentially a test to see which prospects the agent is either indifferent between or has a preferential gap between (and are not ranked below anything else). When you can improve a permissible prospect along some dimension and yet retain the same set of permissible prospects, for example, you necessarily have a preferential gap between those remaining prospects. In short, ex-ante permissibility tells you which prospects the agent doesn’t mind picking between.

  The part of the Incomplete Preference Proposal that carries much of the weight is the Timestep Near-Dominance (TND) principle for choice under uncertainty. One thing it does, roughly, is require that the agent does not mind shifting probability mass between trajectories in which the shutdown time differs. And this is where incompleteness comes in. You need preferential gaps between trajectories that differ in shutdown time for this to hold in general. If the agent had complete preferences over trajectories, it would have strict preferences between at least some trajectories that differ in shutdown time, giving it reason to shift probability mass by manipulating the button.

  Why TND helps get you shutdownability is described in Thornley’s proposal, so I’ll refer to his description and take that as a given here. So, roughly, we’re using TND to get shutdownability, and we’re using incompleteness to get TND. The reason incompleteness helps is that we want to maintain indifference to shifting probability mass between certain trajectories. And that is why we care about ex-ante permissibility. We need the agent, when contemplating manipulating the button, not to want to shift probability mass in that direction. That’ll help give us TND. The rest of Thornley’s proposal includes further conditions on the agent such that it will in fact, ex-post, not manipulate the button. But the reason for the focus on ex-ante permissibility here is TND.
  

  Miscellany

  For purposes of e.g. the shutdown problem, or corrigibility more generally, I don’t think I care about the difference between “mandatory” vs “actually chosen”?

  The description above should help clear up why we care about multiple options being permissible and none mandatory: to help satisfy TND. What’s “actually chosen” in my framework doesn’t neatly connect to the Thornley proposal since he adds extra scaffolding to the agent to determine how it should act. But that’s a separate issue.

  The rough mental model I have of DSM is: at time zero, the agent somehow picks between a bunch of different candidate plans (all of which are “permissible”, whatever that means), and from then on it will behave-as-though it has complete preferences consistent with that plan.
  ...
  it sounds like the proposal in the post just frontloads all the trammelling—i.e. it happens immediately at timestep zero.

  The notion of trammelling I’m using refers to the set of permissible options shrinking as a result of repeated choice. And I argued that there’s no trammelling under certainty or uncertainty, and that trammelling under unawareness is bounded. Here’s why I don’t think you can see it as the agent behaving as if its preferences were complete.

  Consider the case of static choice. It’s meaningful to say that an agent has incomplete preferences. (I don’t think you disagree with that but just for the sake of completeness, I’ll give an example.) Suppose the agent has preferential gaps between all different-letter prospects. From {A,A+,B} the agent will pick either A+ or B. Suppose it picks B. That doesn’t imply, say, that the agent can be thought of as having a strict preference for B over A+. After all, if you offered it {A,A+,B} once again, it might just pick A+, a contradiction. And you can set up something similar with transitivity to get a contradiction from inferring indifference between A+ and B.

  Onto dynamic choice. As you write, it’s reasonable to think of various dynamic choice principles as immediately, statically, choosing a trajectory at timestep zero. Suppose we do that. Then by the argument just above, it’s still not appropriate to model the agent as having complete preferences at the time of choosing. We’re not frontloading any trammelling; the set of ex-ante permissible prospects hasn’t changed. And that’s what we care about for TND.

  I can kinda vaguely pattern-match the setup in this post to the problem: I want to have one “permissible” choice which involves the shutdown button not being pressed, and another “permissible” choice which involves the button being pressed, and I want these two choices to be incomparable to the agent. Now (my mental model of) the DSM rule says: when the agent is turned on, it somehow chooses between (two plans leading to) those two options, and from then on out acts as though it has complete preferences consistent with the choice—i.e. it either (follows a plan which) makes sure the button is pressed, or (follows a plan which) makes sure the button is not pressed, and actively prevents operators from changing it. Which sounds like not-at-all what I wanted for the shutdown problem!

  Agreed! The ex-ante permissibility of various options is not sufficient for shutdownability. The rest of Thornley’s proposal outlines how the agent has to pick (lotteries over) trajectories, which involves more than TND.

• SCPSep 30, 2023, 1:09 PM

  LW: 2 AF: 1

  0

  AF

  in reply to: SMK’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  Thanks Sylvester; fixed!

• SCPSep 9, 2023, 9:45 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Algon’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  Thanks for saying!

  This is an interesting topic. Regarding the discussion you mention, I think my results might help illustrate Elliott Thornley’s point. John Wentworth wrote:

  That makes me think that the small decision trees implicitly contain a lot of assumptions that various trades have zero probability of happening, which is load-bearing for your counterexamples. In a larger world, with a lot more opportunities to trade between various things, I’d expect that sort of issue to be much less relevant.

  My results made no assumptions about the size or complexity of the decision trees, so I don’t think this itself is a reason to doubt my conclusion. More generally, if there exists some Bayesian decision tree that faithfully represents an agent’s decision problem, and the agent uses the appropriate decision principles with respect to that tree, then my results apply. The existence of such a representation is not hindered by the number of choices, the number of options, or the subjective probability distributions involved.

  I think my results under unawareness (section 3) are particularly likely to be applicable to complex real-world decision problems. The agent can be entirely wrong about their actual decision tree—e.g., falsely assigning probability zero to events that will occur—and yet appropriate opportunism remains and trammelling is bounded. This is because any suboptimal decision by an agent in these kinds of cases is a product of its epistemic state; not its preferences. Whether the agent’s preferences are complete or not, it will make wrong turns in the same class of situations. The globally-DSM choice function will guarantee that the agent couldn’t have done better given its knowledge and values, even if the agent’s model of the world is wrong.

• SCPSep 6, 2023, 10:54 PM

  LW: 9 AF: 4

  0

  AF

  in reply to: Algon’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  Good question. They implicitly assume a dynamic choice principle and a choice function that leaves the agent non-opportunistic.

  • Their dynamic choice principle is something like myopia: the agent only looks at their node’s immediate successors and, if a successor is yet another choice node, the agent represents it as some ‘default’ prospect.

  • Their choice rule is something like this: the agent assigns some natural ‘default’ prospect and deviates from it iff it prefers some other prospect. (So if some prospect is incomparable to the default, it’s never chosen.)

  These aren’t the only approaches an agent can employ, and that’s where it fails. It’s wrong to conclude that “non-dominated strategy implies utility maximization” since we know from section 2 that we can achieve non-domination without completeness—by using a different dynamic choice principle and choice function.

• SCPSep 6, 2023, 10:28 PM

  LW: 1 AF: 1

  0

  AF

  in reply to: Algon’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  I take certainty to be a special case of uncertainty. Regarding proof, the relevant bit is here:

  This argument does not apply when the agent is unaware of the structure of its decision tree, so I provide some formal results for these cases which bound the extent to which preferences can de facto be completed. … These results apply naturally to cases in which agents are unaware of the state space, but readers sceptical of the earlier conceptual argument can re-purpose them to make analogous claims in standard cases of certainty and uncertainty.

• SCPSep 6, 2023, 10:08 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: DanielFilan’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  No, the codomain of gamma is the set of (distributions over) consequences.

  Hammond’s notation is inspired by the Savage framework in which states and consequences are distinct. Savage thinks of a consequence as the result of behaviour or action in some state, though this isn’t so intuitively applicable in the case of decision trees. I included it for completeness but I don’t use the gamma function explicitly anywhere.

• SCPSep 6, 2023, 9:45 PM

  LW: 2 AF: 1

  0

  AF

  in reply to: DanielFilan’s comment on: In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

  It’s the set of elementary states.

  So $S\in P\left(S\right)$ is an event (a subset of elementary states, $S\subseteq S$).

  E.g., we could have $S$ be all the possible worlds; $S=\{S_1,...,S_n\}$ be the possible worlds in which featherless bipeds evolved; and $S_7\in S$ be our actual world.

In­vuln­er­a­ble In­com­plete Prefer­ences: A For­mal Statement

• SCPJul 5, 2023, 10:43 AM

  1 point

  0

  in reply to: DirectedEvolution’s comment on: Ra­tional Unilat­er­al­ists Aren’t So Cursed

  doesn’t Bostrom’s model of “naive unilateralists” by definition preclude updating on the behavior of other group members?

  Yeah, this is right; it’s what I tried to clarify in the second paragraph.

  isn’t updating on the beliefs of others (as signaled by their behavior) an example of adopting a version of the “principle of conformity” that he endorses as a solution to the curse? If so, it seems like you are framing a proof of Bostrom’s point as a rebuttal to it.

  The introduction of the post tries to explain how this post relates to Bostrom et al’s paper (e.g., I’m not rebutting Bostrom et al). But I’ll say some more here.

  You’re broadly right on the principle of conformity. The paper suggests a few ways to implement it, one of which is being rational. But they don’t go so far as to endorse this because they consider it mostly unrealistic. I tried to point to some reasons it might not be. Bostrom et al are sceptical because (i) identical priors are assumed and (ii) it would be surprising for humans to be this thoughtful anyway. The derivation above should help motivate why identical priors are sufficient but not necessary for the main upshot, and what I included in the conclusion suggests that many humans—or at least some firms—actually do the rational thing by default.

  But the main point of the post is to do what I explained in the introduction: correct misconceptions and clarify. My experience of informal discussions of the curse suggests people think of it as a flaw of collective action that applies to agents simpliciter, and I wanted to flesh out this mistake. I think the formal framework I used is better at capturing the relevant intuition than the one used in Bostrom et al.

Ra­tional Unilat­er­al­ists Aren’t So Cursed

• SCPApr 15, 2023, 12:27 PM

  4 points

  1

  in reply to: quanticle’s comment on: What’s this prob­a­bil­ity you’re re­port­ing?

  probabilities should correspond to expected observations and expected observations only

  FWIW I think this is wrong. There’s a perfectly coherent framework—subjective expected utility theory (Jeffrey, Joyce, etc)—in which probabilities can correspond to many other things. Probabilities as credences can correspond to confidence in propositions unrelated to future observations, e.g., philosophical beliefs or practically-unobservable facts. You can unambiguously assign probabilities to ‘cosmopsychism’ and ‘Everett’s many-worlds interpretation’ without expecting to ever observe their truth or falsity.

  However, there is another source of uncertainty: observational uncertainty. The other person might be uncertain whether they have all the facts that feed into their model, or whether their observations are correct.

  This is reasonable. If a deterministic model has three free parameters, two of which you have specificied, you could just use your prior over the third parameter to create a distribution of model outcomes. This kind of situation should be pretty easy to clarify though, by saying something like “my model predicts event E iff parameter A is above A*” and “my prior P(A>A*) is 50% which implies P(E)=50%.”

  But generically, the distribution is not coming from a model. It just looks like your all things considered credence that A>A*. I’d be hesitant calling a probability based on it your “inside view/​model” probability.

What’s this prob­a­bil­ity you’re re­port­ing?

• SCPMar 1, 2023, 5:54 PM

  7 points

  2

  on: Some Var­i­ants of Sleep­ing Beauty

  These are great. Though Sleeping Mary can tell that she’s colourblind on any account of consciousness. Whether or not she learns a phenomenal fact when going from ‘colourblind scientist’ to ‘scientist who sees colour’, she does learn the propositional fact that she isn’t colourblind.
  
  So, if she sees no colour, she ought to believe that the outcome of the coin toss is Tails. If she does see colour, both SSA and SIA say P(Heads)=1/​2.