Benya_Fallenstein

• Benya_Fallenstein 14 Jul 2015 4:02 UTC

  0 points

  AF

  in reply to: Benya_Fallenstein’s comment on: Water­fall Truth Predicates

  We should be more careful, though, about what we mean by saying that $\phi \left(x\right)$ only depends on ${Tr}_m$ for $m>n$, though, since this cannot be a purely syntactic criterion if we allow quantification over the subscript (as I did here). I’m pretty sure that something can be worked out, but I’ll leave it for the moment.

• Benya_Fallenstein 14 Jul 2015 3:57 UTC

  LW: 2 AF: 1

  AF

  on: Water­fall Truth Predicates

  I would suggest changing this system by defining $\psi \left(n\right)$ to mean that no $m\le n$ is the Gödel number of a proof of an inconsistency in ZFC (instead of just asserting that $n$ isn’t). The purpose of this is to make it so that if ZFC were inconsistent, then we only end up talking about a finite number of levels of truth predicate. More specifically, I’d define $T_n$ to be PA plus the axiom schema

  $$\forall m\ge n.\psi \left(m\right)\to \forall x.{Tr}_m(┌\phi (\overline{x})┐)\leftrightarrow \phi \left(x\right).$$

  Then, it seems that Jacob Hilton’s proof that the waterfalls are consistent goes through for this waterfall:

  Work in ZFC and assume that ZFC is inconsistent. Let $n$ be the lowest Gödel number of a proof of an inconsistency. Let $M$ be the following model of our language: Start with the standard model of PA; it remains to give interpretations of the truth predicates. If $m\ge n$, then ${Tr}_m\left(k\right)$ is false for all $k$. If $m<n$, then ${Tr}_m\left(k\right)$ is true iff $k$ is the Gödel number of a true formula involving only ${Tr}_{m^{\prime }}$ for $m^{\prime }>m$. Then, it’s clear that $T_0$, and hence all $T_m$ (since $T_0$ is the strongest of the systems) is sound on $M$, and therefore consistent.

  Thus, we have proven in ZFC that if ZFC is inconsistent, then $T_0$ is consistent; or equivalently, that if $T_0$ is inconsistent, then ZFC is consistent. Stepping out of ZFC, we can see that if $T_0$ is inconsistent, then ZFC proves this, and therefore in this case ZFC proves its own consistency, implying that it is inconsistent. Hence, if ZFC is consistent, then so is $T_0$.

  (Moreover, we can formalize this reasoning in ZFC. Hence, we can prove in ZFC (i) that if ZFC is inconsistent, then $T_0$ is consistent, and (ii) that if ZFC is consistent, then $T_0$ is consistent. By the law of the excluded middle, ZFC proves that $T_0$ is consistent.)

• Benya_Fallenstein 3 Jun 2015 18:29 UTC

  LW: 1 AF: 1

  AF

  in reply to: János Kramár’s comment on: No Good Log­i­cal Con­di­tional Probability

  Hm; we could add an uninterpreted predicate symbol $Q\left(n\right)$ to the language of arithmetic, and let $s\equiv Q\left(0\right)$ and $r_n\equiv Q\left(\overline{n+1}\right)$. Then, it seems like the only barrier to recursive enumerability of $T_{\infty }$ is that $P$’s opinions about $Q(\cdot )$ aren’t computable; this seems worrying in practice, since it seems certain that we would like logical uncertainty to be able to reason about the values of computations that use more resources than we use to compute our own probability estimates. But on the other hand, all of this makes this sound like an issue of self-reference, which is its own can of worms (once we have a computable process assigning probabilities to the value of computations, we can consider the sentence saying “I’m assigned probability $<\frac{1}{2}$” etc.).

• Benya_Fallenstein 2 Jun 2015 20:03 UTC

  LW: 2 AF: 1

  AF

  on: Agents that can pre­dict their New­comb predictor

  The other direction follows from the fact that the algorithm is bounded, and PA can simply show the execution trace of $A$ in $\le CN$ steps.

  Unimportant technical point: I think the length of the PA proof grows faster than this. (More precisely, the length in symbols, rather than the length in number of statements; we’re almost always interested in the former, since it determines how quickly a proof can be checked or be found by exhaustive search.) The obvious way of showing $A\left(\right)=1$ in PA is to successively show for higher and higher $t$ that “after $t$ ticks, the Turing machine computing $A\left(\right)$ is in the following state, and its tapes have the following content”. But even to write out the contents of the tape of a Turing machine that has run for $t$ ticks will in general take $t$ symbols. So the obvious way of doing the proof takes at least $O\left(t^2\right)$ symbols. Moreover, I’m not sure whether we can get from “at time $t$, the configuration is such-and-such” to “at time $t+1$, the configuration is such-and-such” in a constant number of proof steps. I suspect we don’t need more than $O\left(t^3\right)$ symbols overall, but I’m not extremely confident. (I’d be quite surprised if the time turned out not to be polynomial, though!)

• Benya_Fallenstein 16 May 2015 16:38 UTC

  LW: 2 AF: 1

  AF

  in reply to: SamEisenstat’s comment on: An In­for­mal Con­jec­ture on Proof Length and Log­i­cal Counterfactuals

  Next, we consider the case that PA is consistent and work through the agent’s decision. PA can’t prove $A\left(\right)\ne 1$, since we used the chicken rule, so since the sentence $A\left(\right)=1\to U\left(\right)=5$ is easily provable, the sentence $A\left(\right)=1\to U\left(\right)=10$ (ie. the first sentence that the agents checks for proofs of) must be unprovable.

  It seems like this argument needs soundness of PA, not just consistency of PA. Do you see a way to prove in PA that if $PA⊢A\left(\right)\ne 1$, then PA is inconsistent?

  [edited to add:] However, your idea reminds me of my post on the odd counterfactuals of playing chicken, and I think the example I gave there makes your idea go through:

  The scenario is that you get 10 if you take action 1 and it’s not provable that you don’t take action 1; you get 5 if you take action 2; and you get 0 if you take action 1 and it’s provable that you don’t. Clearly you should take action 1, but I prove that modal UDT actually takes action 2. To do so, I show that PA proves $A\left(\right)=1\to \neg \square ┌A\left(\right)=1┐$. (Then from the outside, $A\left(\right)=2$ follows from the outside by soundness of PA.)

  This seems to make your argument go through if we can also show that PA doesn’t show $A\left(\right)\ne 1$. But if it did, then modal UDT would take action 1 because this comes first in its proof search, contradiction.

  Thus, PA proves $A\left(\right)=1\to U\left(\right)=0$ (because this follows from $A\left(\right)=1\to \neg \square ┌A\left(\right)=1┐$), and also PA doesn’t prove $A\left(\right)=1\to U\left(\right)=10$. As in your argument, then, the trolljecture implies that we should think “if the agent takes action 1, it gets utility 0” is a good counterfactual, and we don’t think that’s true.

  Still interested in whether you can make your argument go through in your case as well, especially if you can use the chicken step in a way I’m not seeing yet. Like Patrick, I’d encourage you to develop this into a post.

  What links here?

  • Proof Length and Log­i­cal Coun­ter­fac­tu­als Revisited by orthonormal (10 Feb 2016 18:56 UTC; 3 points)

• Benya_Fallenstein 6 May 2015 2:41 UTC

  LW: 1 AF: 1

  AF

  in reply to: Diffractor’s comment on: Reflec­tive prob­a­bil­is­tic logic can­not as­sign pos­i­tive prob­a­bil­ity to its own co­her­ence and an in­ner re­flec­tion principle

  If you replace the inner “$=1$” by “$>1-ϵ$”, then the literal thing you wrote follows from the reflection principle: Suppose that the outer probability is $<1$. Then

  $$P\left[\right(a<P\left[\phi \right]<b)\wedge P[a-ϵ<P\left[\phi \right]<b+\epsilon ]\le 1-ϵ]>0.$$

  Now, $P[a<P[\phi ]<b]>0$ implies $P[a\le P[\phi ]\le b]>0$, which by the converse of the outer reflection principle yields $a\le P\left[\phi \right]\le b$, whence $a-ϵ<P\left[\phi \right]<b+ϵ$. Now, by the forward direction of the outer reflection principle, we have

  $$P[a-ϵ<P[\phi ]<b+ϵ]=1>1-ϵ,$$

  which, by the outer reflection principle again, implies

  $$P\left[P\right[a-ϵ<P\left[\phi \right]<b+ϵ]>1-ϵ]=1,$$

  a contradiction to the assumption that $\cdots \le 1-ϵ$ had outer probability $>0$.

  However, what we’d really like is an inner reflection principle that assigns probability one to the statement *quantified over all $a$, $b$, and Gödel numbers $┌\phi ┐$. I think Paul Christiano has a proof that this is impossible for small enough $ϵ$, but I don’t remember how the details worked.

• Benya_Fallenstein 28 Apr 2015 0:40 UTC

  LW: 2 AF: 1

  AF

  on: Para­con­sis­tent Tiling Agents (Very Early Draft)

  This is very interesting!

  My main question, which I didn’t see an answer to on my first read, is: If the agent proves that action $a$ leads to the goal $G$ being achieved, are there any conditions under which this implies that the goal actually is achieved? The procrastination paradox shows that this isn’t true in general. Is there a class of sentences (e.g., all ${\Pi }_1$ sentences, though I don’t expect that to be true in this case) such that if $PA\star ⊢\phi$ then $N\models \phi$? In other words, do we have some guarantee that we do better at actually achieving $G$ than an agent which uses the system $BAD:=PA+Con\left(BAD\right)$?

  A more technical point: Given that you claim that Theorem 3.2 (reflectively coherent quantified belief) follows “trivially” from Theorem 2.4, you presumably mean Theorem 2.4 to state that $PA\star ⊢\forall x.\square ┌\phi \left(\overline{x}\right)┐\to \phi \left(x\right)$, rather than $PA\star ⊢\forall x.\square ┌\phi ┐\to \phi$. (Even if you consider $\phi$ to implicitly contain an occurrence of $x$, it is essential that in $\square ┌\phi \left(\overline{x}\right)┐$ the free variable is replaced by the numeral corresponding to the outer value of $x$.)

• Benya_Fallenstein 24 Mar 2015 17:42 UTC

  LW: 2 AF: 1

  AF

  in reply to: Stuart_Armstrong’s comment on: Fo­rum Digest: Cor­rigi­bil­ity, util­ity in­differ­ence, & re­lated con­trol ideas

  Categorization is hard! :-) I wanted to break it up because long lists are annoying to read, but there was certainly some arbitrariness in dividing it up. I’ve moved “resource gathering agent” to the odds & ends.

Fo­rum Digest: Cor­rigi­bil­ity, util­ity in­differ­ence, & re­lated con­trol ideas

• Benya_Fallenstein 22 Mar 2015 1:33 UTC

  LW: 4 AF: 1

  AF

  in reply to: So8res’s comment on: Iden­tity and quin­ing in UDT

  Want to echo Nate’s points!

  One particular thing that I wanted to emphasize is that I think you can see as a thread on this forum (in particular, the modal UDT work is relevant) is that it’s useful to make formal toy models where the math is fully specified, so that you can prove theorems about what exactly an agent would do (or, sometimes, write a program that figures it out for you). When you write out things that explicitly, then, for example, it becomes clearer that you need to assume that a decision problem is “fair” (extensional) to get certain results, as Nate points out (or if you don’t assume it, someone else can look at your result and point out that it’s not true as stated). In your post, you’re using “logical expectations” that condition on something being true, without defining exactly what all of this means, and as a result you can argue about what these agents will do, but not actually prove it; that’s certainly a reasonable part of the research process, but I’d like to encourage you to turn your work into models that are fully specified, so that you can actually prove theorems about them.

Sin­gle-bit re­flec­tive or­a­cles are enough

• Benya_Fallenstein 17 Mar 2015 21:06 UTC

  LW: 2 AF: 1

  AF

  on: Meta- the goals of this forum

  Some additions about how the initial system is going to work:

  • Non-member contributions (comments and links) are going to become publicly visible when they have received 2 likes (from members—only members can like things).

  • However, members will be able to reply to a contribution as soon as it has received 1 like. This means that if you think someone’s made a useful contribution, and you want to reply to them about it, you don’t have to wait for a second person to Like it before you can write that reply. (It won’t be publicly visible until the contribution has two Likes, though.)

An im­ple­men­ta­tion of modal UDT

Gen­er­al­iz­ing the Cor­rigi­bil­ity pa­per’s im­pos­si­bil­ity re­sult?

• Benya_Fallenstein 3 Feb 2015 20:56 UTC

  0 points

  AF

  in reply to: orthonormal’s comment on: Third-per­son counterfactuals

  Yeah, that sounds good! Of course, by the Kripke levels argument, it’s sufficient to consider $\varphi$’s of the form $\neg {\square }^n\perp$. And we might want to have a separate notion of ”$i$ leads to $j$ at level $n$”, which we can actually implement in a finite modal formula. This seems to suggest a version of modal UDT that tries to prove things in PA, then if that has ambiguous counterfactuals (i.e., it can’t prove $A_i\to U_j$ for any $j$) we try PA+1 and so on up to some finite $n$; then we can hope these versions of UDT approximate optimality according to your revised version of ”$i$ leads to $j$” as $n\to \infty$. Worth working out!

On no­ta­tion for modal UDT

From halt­ing or­a­cles to modal logic

• Benya_Fallenstein 3 Feb 2015 1:25 UTC

  LW: 2 AF: 1

  AF

  in reply to: orthonormal’s comment on: Third-per­son counterfactuals

  Fixed—thanks, Patrick!

  Regarding the example, I earlier defined “action $i$ leads to outcome $j$” to mean the conjunction of $GL⊢A_i\to U_j$ and $GL⊬\neg A_i$; i.e., we check for spurious counterfactuals before believing that $GL⊢A_i\to U_j$ tells us something about what action $i$ leads to, and we only consider ourselves “fully informed” in this sense if we have non-spurious information for each $i$. (Of course, my follow-up post is about how that’s still unsatisfactory; the reason to define this notion of “fully informative” so explicitly was really to be able to say more clearly in which sense we intuitively have a problem even when we’ve ruled out the problems of ambiguous counterfactuals and not enough counterfactuals.)

Third-per­son counterfactuals

The odd coun­ter­fac­tu­als of play­ing chicken