I don’t think it’s any different. You could have a Q in the box, and include a person that types it in a calculator as part of the scanning device. Does your variant evoke different intuitions about observational knowledge? It looks similar in all relevant respects to me.
Does your variant evoke different intuitions about observational knowledge?
No. Our intuitions agree here. When I wrote the comment, I didn’t understand what point you were making by having the problem be about a mathematical fact. I wanted to be sure that you weren’t saying that the math version was different from the coin version.
I’m still not certain that I understand the point you’re making. I think you’re pointing out that, e.g., a UDT1.1 agent doesn’t worry about the probability that it has computed the correct value for the expected utility EU(f) of an input-output map f. In contrast, such an agent does involve probabilities when considering a statement like “Q evaluates to an even number”. I’m not sure whether you would agree, but I would say moreover that the agent would involve probabilities when considering the statement “the digit 2, which I am considering as an object of thought in my own mind, denotes an even number.”
Is that a correct interpretation of your point?
The distinction between the way that the agent treats “EU(f)” and “Q” seems to me to be this: The agent doesn’t think about the expression “EU(f)” as an object of thought. The agent doesn’t look at “EU(f)” and wonder whether it evaluates to greater than or less than some other value EU(f’). The agent just runs through a sequence of states that can be seen, from the outside, as instantiating a procedure that maximizes the function EU. But for the agent to think this way would be like having the agent worry about whether it’s doing what it was programmed to do. From the outside, we can worry about whether the agent is in fact programmed to do what we intended to program it to do. But that won’t be the agent’s concern. The agent will just do what it does. Along the way, it might wonder about whether Q denotes an even number. But the agent won’t wonder whether EU(f) > EU(f’), although its ultimate action might certify that fact.
FWIW, here is my UDT1.1 analysis of the problem in the OP. In UDT terms, the way I think of it is to suppose that there are 99 world programs in which the calculator is correct, and 1 world program in which the calculator is incorrect. The utility of a given sequence of execution histories equals the number of world programs in which the answer on the test sheet is correct.
Ignoring obviously-wrong alternatives, there are two possible input-output maps. These are, respectively,
f_1: Try to write your calculator’s answer on your test sheet. If Omega appears to you, make the answer(s) on the test sheet(s) in the opposite-calculator world(s) be identical to the answer on your test sheet.
f_2: Try to write your calculator’s answer on your test sheet. If Omega appears to you, make the answer(s) on the test sheet(s) in the opposite-calculator world(s) be the opposite of the answer on your test sheet.
I assume that, according to the agent’s mathematical intuition, Omega is just as likely to offer the decision in a correct-calculator world as in the incorrect-calculator world. From this, it follows that the expected utilities of the two input-output maps are, respectively,
f_1: p(Omega offers the decision in a correct-calculator world)*100 + p(Omega offers the decision in an incorrect-calculator world)*0 = 50;
f_2: p(Omega offers the decision in a correct-calculator world)*99 + p(Omega offers the decision in an incorrect-calculator world)*99 = 99.
So, the agent will make the test sheet in each world agree with the calculator in that world.
(In particular, anthropic consideration don’t come into play, because UDT1.1 already dictates the non-centered quantity that the agent is maximizing.)
I think considerably more than two things have to go well for your interpretation to succeed in describing this post… I don’t necessarily disagree with what you wrote, in that I don’t see clear enough statements that I disagree with, and some things seem correct, but I don’t understand it well.
Also, calculator is correct 99% of the time, so you’ve probably labeled things in a confusing way that could lead to incorrect solution, although the actual resulting numbers seem fine for whatever reason.
The reason I used a logical statement instead of a coin, was to compare logical and observational knowledge, since logical knowledge, in its usual understanding, applies mostly to logical statements, and doesn’t care what you reason about using it. This can allow extending the thought experiment, for example, in this way.
The reason I used a logical statement instead of a coin, was to compare logical and observational knowledge, since logical knowledge, in its usual understanding, applies mostly to logical statements, and doesn’t care what you reason about using it. This can allow extending the thought experiment, for example, in this way.
I’m not seeing why that extended thought experiment couldn’t have used a coin and two scanners of different reliability.
I’m not seeing why that extended thought experiment couldn’t have used a coin and two scanners of different reliability.
The point is in showing that having a magical kind of knowledge certified by proofs doesn’t help (presumably) in that thought experiment, and hopefully reducing events of possible worlds to logical statements. So I want to use as many logical kinds of building blocks as possible, in order to see the rest in their terms.
The point is in showing that having a magical kind of knowledge certified by proofs doesn’t help (presumably) in that thought experiment, and hopefully reducing events of possible worlds to logical statements. So I want to use as many logical kinds of building blocks as possible, in order to see the rest in their terms.
Fair enough. To me it seems more illuminating to see logical facts (like the parity of Q) as physical facts (in this case, a statement about what certain kinds of physical mechanisms would do under certain circumstances.) But, at any rate, we seem to agree that these two kinds of facts ought to be thought of in the same way.
I think considerably more than two things have to go well for your interpretation to succeed in describing this post.
Indeed. That is because you needed more than two things to go right for your post to succeed in communicating your point ;).
My confusion is over this sentence from your post:
This thought experiment contrasts “logical knowledge” (the usual kind) and “observational knowledge” (what you get when you look at a calculator display).
My difficulty is that everything that I would call knowledge is like what you get when you look at a calculator display. Suppose that the test had asked you whether “2+2” reduced to an even number. Then you would perform certain mental operations on this expression, and you would answer in accordance with how those operations concluded. (For example, you might picture two sets of two dots, one set next to the other, and see whether you can pair off elements in one set with elements in the other. Or you might visualize a proof in Peano arithmetic in your mind, and check whether each line follows from the previous line in accordance with the rules of inference.) At any rate, whatever you do, it amounts to relying on the imperfect wetware calculator that is your brain. If a counterfactual version of you got a different answer with his brain, you would still want his test sheet to match his answer.
So, what is the residue left over, after we set aside observational knowledge? What is this “logical knowledge”? Calling it “the usual kind” is not sufficing to pick out what you mean for me.
My guess was that your “logical knowledge” includes (in your terminology) the “moral arguments” that “the agent can prove” in the “theory it uses”. The analogous role in Wie Dai’s “brute-force” UDT is served by the agent’s computation of an expected utility EU(f) for an input-output map f.
Is this a correct interpretation of what you meant by “logical knowledge”? (I know that I may need more than two things to go right to have interpreted you correctly. That is why I am giving you my interpretation of what you said. If I got it right, great. But my main motivation arises in the case where I am wrong. My hope is that you will then restate your claim, this time calibrating for the way that I am evidently primed to misinterpret you. If I were highly confident that I had understood you correctly, I wouldn’t bother echoing what you said back at you.)
Also, calculator is correct 99% of the time, so you’ve probably labeled things in a confusing way that could lead to incorrect solution, although the actual resulting numbers seem fine for whatever reason.
Could you spell out how exactly the 99% correctness rate means that I’ve probably labeled things confusingly? What is the first probably-confusing label that I used?
My difficulty is that everything that I would call knowledge is like what you get when you look at a calculator display.
In some sense, sure. But you still have to use certain specific reasoning procedure to think about imperfection of knowledge-acquisition methods. That level where you just perform the algorithm is where logic resides. It’s not clear to me how to merge these considerations seamlessly.
My guess was that your “logical knowledge” includes (in your terminology) the “moral arguments” that “the agent can prove” in the “theory it uses”.
Yes. This theory can include tools for reasoning about observational and logical uncertainty, where logical uncertainty refers to inability to reach the conclusions (explore long enough proofs) rather than uncertainty about whether the reasoning apparatus would do something unintended.
Could you spell out how exactly the 99% correctness rate means that I’ve probably labeled things confusingly?
I referred to this statement you made:
I assume that, according to the agent’s mathematical intuition, Omega is just as likely to offer the decision in a correct-calculator world as in the incorrect-calculator world.
It’s not clear what the “Omega offers the decision in a correct-calculator world” event is, since we already know that Omega offers the decision in “even” worlds, in some of which “even” is correct, and in some of which it’s not (as far as you know), and 99% of “even” worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event.
It’s not clear what the “Omega offers the decision in a correct-calculator world” event is, since we already know that Omega offers the decision in “even” worlds, in some of which “even” is correct, and in some of which it’s not (as far as you know), and 99% of “even” worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event.
When you speak of “worlds” here, do you mean the “world-programs” in the UDT1.1 formalism? If that is what you mean, then one of us is confused about how UDT1.1 formalizes probabilities. I’m not sure how to resolve this except to repeat my request that you give your own formalization of your problem in UDT1.1.
For my part, I am going to say some stuff on which I think that we agree. But, at some point, I will slide into saying stuff on which we disagree. Where is the point at which you start to disagree with the following?
UDT1.1 formalizes two different kinds of probability in two very different ways:
One kind of probability is applied to predicates of world-programs, especially predicates that might be satisfied by some of the world-programs while not being satisfied by the others. The probability (in the present sense) of such a predicate R is formalized as the measure of the set of world-programs satisfying R. (In particular, R is supposed to be a predicate such that whether a world-program satisfies R does not depend on the agent’s decisions.)
The other kind of probability comes from the probability M(f, E) that the agent’s mathematical intuition M assigns to the proposition that the sequence E of execution histories would occur if the agent were to implement input-output map f. This gives us probability measures P_f over sequences of execution histories: Given a predicate T of execution-history sequences, P_f(T) is the sum of the values M(f, E) as E ranges over the execution-history sequences satisfying predicate T.
I took the calculator’s 99% correctness rate to be a probability of the first kind. There is a correct calculator in 99% of the world-programs (the “correct-calculator worlds”) and an incorrect calculator in the remaining 1%.*
However, I took the probability of 1⁄2 that Q is even to be a probability of the second kind. It’s not as though Q is even in some of the execution histories, while that same Q is odd in some others. Either Q is even in all of the execution histories, or Q is odd in all of the execution histories.* But the agent’s mathematical intuition has no idea which is the case, so the induced probability distributions give P_f(even) = 1⁄2 (for all f), where even is the predicate such that, for all execution-history sequences E*,
even(E) = “The parity of Q is ultimately revealed by the grader to be even in all of the execution histories in E”
Likewise, I was referring to the second kind of probability when I wrote that, “according to the agent’s mathematical intuition, Omega is just as likely to offer the decision in a correct-calculator world as in the incorrect-calculator world”. The truth or falsity of “Omega offers the decision in a correct-calculator world” is a property of an entire execution-history sequence. This proposition is either true with respect to all the execution histories in the sequence, or false with respect to all of them.
The upshot is that, when you write “99% of ‘even’ worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event”, you are talking about two very different kinds of probabilities.
* Alternatively, this weighting can be incorporated into how the utility function over execution-history sequences responds to an event occurring in one world-program vs. another. If I had used this approach in my UDT1.1 formalization of your problem, I would have had just two world-programs: a correct-calculator world and an incorrect-calculator world. Then, having the correct parity on the answer sheet in the correct-calculator world would have been worth 99 times as much as having the correct parity in the incorrect-calculator world. But this would not have changed my computations. I don’t think that this issue is the locus of our present disagreement.
* You must be disagreeing with me by this point, because I have contradicted your claim that “Omega offers the decision in ‘even’ worlds, in some of which ‘even’ is correct, and in some of which it’s not*”. (Emphasis added.)
You must be disagreeing with me by this point, because I have contradicted your claim that “Omega offers the decision in ‘even’ worlds, in some of which ‘even’ is correct, and in some of which it’s not”. (Emphasis added.)
World-programs are a bad model for possible worlds. For all you know, there could be just one world-program (indeed you can consider an equivalent variant of the theory where it’s so: just have that single world program enumerate all outputs of all possible programs). The element of UDT analogous to possible worlds is execution histories. And some execution histories easily indicate that 2+2=5 (if we take execution histories to be enumerations of logical theories, with world-programs axiomatic definitions of theories). Observations, other background facts, and your actions are all elements that specify (sets/events of) execution histories. Utility function is defined on execution histories (and it’s usually defined on possible worlds). Probability given by mathematical intuition can be read as naming probability that given execution history (possible world) is an actual one.
It’s not clear what the “Omega offers the decision in a correct-calculator world” event is, since we already know that Omega offers the decision in “even” worlds, in some of which “even” is correct, and in some of which it’s not (as far as you know), and 99% of “even” worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event.
So, you intended that the equivalence
“Omega offers the decision” <==> “the calculator says ‘even’ ”
be known to the agent’s mathematical intuition? I didn’t realize that, but my solution still applies without change. It just means that, as far as the agent’s mathematical intuition is concerned, we have the following equivalences between predicates over sequences of execution histories:
“Omega offers the decision in a correct-calculator world”
is equivalent to
“The calculator says ‘even’ in the 99 correct-calculator worlds”,
while
“Omega offers the decision in an incorrect-calculator world”
is equivalent to
“The calculator says ‘even’ in the one incorrect-calculator world”.
Below, I give my guess at your UDT1.1 approach to the problem in the OP. If I’m right, then we use the UDT1.1 concepts differently, but the math amounts to just a rearrangement of terms. I see merits in each conceptual approach over the other. I haven’t decided which one I like best.
At any rate, here is my guess at your formalization: We have one world-program. We consider the following one-place predicates over possible execution histories for this program: Given any execution history E,
CalculatorIsCorrect(E) asserts that, in E, the calculator gives the correct parity for Q.
“even”(E) asserts that, in E, the calculator says “even”. Omega then appears to the agent and asks it what Omega should have written on the test sheet in an execution history in which (1) Omega blocks the agent from writing on the answer sheet and (2) the calculator says “odd”.
“odd”(E) asserts that, in E, the calculator says “odd”. Omega then (1) blocks the agent from writing on the test sheet and (2) computes what the agent would have said to Omega in an execution history F such that “even”(F). Omega then writes what the agent would say in F on the answer sheet in E.
Borrowing notation from my last comment, we make the following assumptions about the probability measures P_f. For all input-output maps f,
P_f(CalculatorIsCorrect) = 0.99,
P_f(“even”) = P_f(“odd”) = 1⁄2,
“even” and “odd” are uncorrelated with CalculatorIsCorrect under P_f.
The input-output maps to consider are
g: On seeing “even”, write “even” and tell Omega, “Write ‘even’.”
h: On seeing “even”, write “even” and tell Omega, “Write ‘odd’.”
The utility U(E) of an execution history E is 1 if the answer on the sheet in E is the true parity of Q. Otherwise, U(E) = 0.
The expected payoffs of g and h are then, respectively,
It’s not clear what the “Omega offers the decision in a correct-calculator world” event is, since we already know that Omega offers the decision in “even” worlds, in some of which “even” is correct, and in some of which it’s not (as far as you know), and 99% of “even” worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event.
I “weakly” argue for the 50% probability as well. My argument follows the Pearl-type of counterfactual (Drescher calls it “choice-friendly”) -- when you counterfactually set a variable, you cut directed arrows that lead to it, but not directed arrows that lead out or undirected arrows (which in another comment I mistakenly called bi-directed). My intuition is that the “causing” node might possibly be logically established before the “caused” node thus possibly leading to contradiction in the counterfactual, while the opposite direction is not possible (the “caused” node cannot be logically established earlier than the “causing” node). Directly logically establishing the counterfactual node is harmless in that it invalidates the counterfactual straight away, the argument “fears” of the “gap” where we possibly operate by using a contradictory counterfactual.
Pearl’s counterfactuals (or even causal diagrams) are unhelpful, as they ignore the finer points of logical control that are possibly relevant here. For example, that definitions (facts) are independent should refer to the absence of logical correlation between them, that is inability to infer (facts about) one from the other. But this, too, is shaky in the context of this puzzle, where the nature of logical knowledge is called into question.
Is it a trivial remark regarding the probability theory behind Pearl’s “causality”, or an intuition with regard to future theories that resemble Pearl’s approach?
It is a statement following from my investigation of logical/ambient control and reality-as-normative-anticipation thesis which I haven’t written much about, but this all is regardless called in question as adequate foundation in light of the thought experiment.
I don’t think it’s any different. You could have a Q in the box, and include a person that types it in a calculator as part of the scanning device. Does your variant evoke different intuitions about observational knowledge? It looks similar in all relevant respects to me.
No. Our intuitions agree here. When I wrote the comment, I didn’t understand what point you were making by having the problem be about a mathematical fact. I wanted to be sure that you weren’t saying that the math version was different from the coin version.
I’m still not certain that I understand the point you’re making. I think you’re pointing out that, e.g., a UDT1.1 agent doesn’t worry about the probability that it has computed the correct value for the expected utility EU(f) of an input-output map f. In contrast, such an agent does involve probabilities when considering a statement like “Q evaluates to an even number”. I’m not sure whether you would agree, but I would say moreover that the agent would involve probabilities when considering the statement “the digit 2, which I am considering as an object of thought in my own mind, denotes an even number.”
Is that a correct interpretation of your point?
The distinction between the way that the agent treats “EU(f)” and “Q” seems to me to be this: The agent doesn’t think about the expression “EU(f)” as an object of thought. The agent doesn’t look at “EU(f)” and wonder whether it evaluates to greater than or less than some other value EU(f’). The agent just runs through a sequence of states that can be seen, from the outside, as instantiating a procedure that maximizes the function EU. But for the agent to think this way would be like having the agent worry about whether it’s doing what it was programmed to do. From the outside, we can worry about whether the agent is in fact programmed to do what we intended to program it to do. But that won’t be the agent’s concern. The agent will just do what it does. Along the way, it might wonder about whether Q denotes an even number. But the agent won’t wonder whether EU(f) > EU(f’), although its ultimate action might certify that fact.
FWIW, here is my UDT1.1 analysis of the problem in the OP. In UDT terms, the way I think of it is to suppose that there are 99 world programs in which the calculator is correct, and 1 world program in which the calculator is incorrect. The utility of a given sequence of execution histories equals the number of world programs in which the answer on the test sheet is correct.
Ignoring obviously-wrong alternatives, there are two possible input-output maps. These are, respectively,
f_1: Try to write your calculator’s answer on your test sheet. If Omega appears to you, make the answer(s) on the test sheet(s) in the opposite-calculator world(s) be identical to the answer on your test sheet.
f_2: Try to write your calculator’s answer on your test sheet. If Omega appears to you, make the answer(s) on the test sheet(s) in the opposite-calculator world(s) be the opposite of the answer on your test sheet.
I assume that, according to the agent’s mathematical intuition, Omega is just as likely to offer the decision in a correct-calculator world as in the incorrect-calculator world. From this, it follows that the expected utilities of the two input-output maps are, respectively,
f_1: p(Omega offers the decision in a correct-calculator world)*100 + p(Omega offers the decision in an incorrect-calculator world)*0 = 50;
f_2: p(Omega offers the decision in a correct-calculator world)*99 + p(Omega offers the decision in an incorrect-calculator world)*99 = 99.
So, the agent will make the test sheet in each world agree with the calculator in that world.
(In particular, anthropic consideration don’t come into play, because UDT1.1 already dictates the non-centered quantity that the agent is maximizing.)
I think considerably more than two things have to go well for your interpretation to succeed in describing this post… I don’t necessarily disagree with what you wrote, in that I don’t see clear enough statements that I disagree with, and some things seem correct, but I don’t understand it well.
Also, calculator is correct 99% of the time, so you’ve probably labeled things in a confusing way that could lead to incorrect solution, although the actual resulting numbers seem fine for whatever reason.
The reason I used a logical statement instead of a coin, was to compare logical and observational knowledge, since logical knowledge, in its usual understanding, applies mostly to logical statements, and doesn’t care what you reason about using it. This can allow extending the thought experiment, for example, in this way.
I’m not seeing why that extended thought experiment couldn’t have used a coin and two scanners of different reliability.
The point is in showing that having a magical kind of knowledge certified by proofs doesn’t help (presumably) in that thought experiment, and hopefully reducing events of possible worlds to logical statements. So I want to use as many logical kinds of building blocks as possible, in order to see the rest in their terms.
Fair enough. To me it seems more illuminating to see logical facts (like the parity of Q) as physical facts (in this case, a statement about what certain kinds of physical mechanisms would do under certain circumstances.) But, at any rate, we seem to agree that these two kinds of facts ought to be thought of in the same way.
Indeed. That is because you needed more than two things to go right for your post to succeed in communicating your point ;).
My confusion is over this sentence from your post:
My difficulty is that everything that I would call knowledge is like what you get when you look at a calculator display. Suppose that the test had asked you whether “2+2” reduced to an even number. Then you would perform certain mental operations on this expression, and you would answer in accordance with how those operations concluded. (For example, you might picture two sets of two dots, one set next to the other, and see whether you can pair off elements in one set with elements in the other. Or you might visualize a proof in Peano arithmetic in your mind, and check whether each line follows from the previous line in accordance with the rules of inference.) At any rate, whatever you do, it amounts to relying on the imperfect wetware calculator that is your brain. If a counterfactual version of you got a different answer with his brain, you would still want his test sheet to match his answer.
So, what is the residue left over, after we set aside observational knowledge? What is this “logical knowledge”? Calling it “the usual kind” is not sufficing to pick out what you mean for me.
My guess was that your “logical knowledge” includes (in your terminology) the “moral arguments” that “the agent can prove” in the “theory it uses”. The analogous role in Wie Dai’s “brute-force” UDT is served by the agent’s computation of an expected utility EU(f) for an input-output map f.
Is this a correct interpretation of what you meant by “logical knowledge”? (I know that I may need more than two things to go right to have interpreted you correctly. That is why I am giving you my interpretation of what you said. If I got it right, great. But my main motivation arises in the case where I am wrong. My hope is that you will then restate your claim, this time calibrating for the way that I am evidently primed to misinterpret you. If I were highly confident that I had understood you correctly, I wouldn’t bother echoing what you said back at you.)
Could you spell out how exactly the 99% correctness rate means that I’ve probably labeled things confusingly? What is the first probably-confusing label that I used?
What I gave looks to me to be the by-the-book way to state and solve your problem within the UDT1.1 formalism. How would you set up the problem within the UDT1.1 formalism? In particular, what would be your set of possible sequences of execution histories for the world-programs?
In some sense, sure. But you still have to use certain specific reasoning procedure to think about imperfection of knowledge-acquisition methods. That level where you just perform the algorithm is where logic resides. It’s not clear to me how to merge these considerations seamlessly.
Yes. This theory can include tools for reasoning about observational and logical uncertainty, where logical uncertainty refers to inability to reach the conclusions (explore long enough proofs) rather than uncertainty about whether the reasoning apparatus would do something unintended.
I referred to this statement you made:
It’s not clear what the “Omega offers the decision in a correct-calculator world” event is, since we already know that Omega offers the decision in “even” worlds, in some of which “even” is correct, and in some of which it’s not (as far as you know), and 99% of “even” worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event.
When you speak of “worlds” here, do you mean the “world-programs” in the UDT1.1 formalism? If that is what you mean, then one of us is confused about how UDT1.1 formalizes probabilities. I’m not sure how to resolve this except to repeat my request that you give your own formalization of your problem in UDT1.1.
For my part, I am going to say some stuff on which I think that we agree. But, at some point, I will slide into saying stuff on which we disagree. Where is the point at which you start to disagree with the following?
(I follow the notation in my write-up of UDT1.1 (pdf).)
UDT1.1 formalizes two different kinds of probability in two very different ways:
One kind of probability is applied to predicates of world-programs, especially predicates that might be satisfied by some of the world-programs while not being satisfied by the others. The probability (in the present sense) of such a predicate R is formalized as the measure of the set of world-programs satisfying R. (In particular, R is supposed to be a predicate such that whether a world-program satisfies R does not depend on the agent’s decisions.)
The other kind of probability comes from the probability M(f, E) that the agent’s mathematical intuition M assigns to the proposition that the sequence E of execution histories would occur if the agent were to implement input-output map f. This gives us probability measures P_f over sequences of execution histories: Given a predicate T of execution-history sequences, P_f(T) is the sum of the values M(f, E) as E ranges over the execution-history sequences satisfying predicate T.
I took the calculator’s 99% correctness rate to be a probability of the first kind. There is a correct calculator in 99% of the world-programs (the “correct-calculator worlds”) and an incorrect calculator in the remaining 1%.*
However, I took the probability of 1⁄2 that Q is even to be a probability of the second kind. It’s not as though Q is even in some of the execution histories, while that same Q is odd in some others. Either Q is even in all of the execution histories, or Q is odd in all of the execution histories.* But the agent’s mathematical intuition has no idea which is the case, so the induced probability distributions give P_f(even) = 1⁄2 (for all f), where even is the predicate such that, for all execution-history sequences E*,
even(E) = “The parity of Q is ultimately revealed by the grader to be even in all of the execution histories in E”
Likewise, I was referring to the second kind of probability when I wrote that, “according to the agent’s mathematical intuition, Omega is just as likely to offer the decision in a correct-calculator world as in the incorrect-calculator world”. The truth or falsity of “Omega offers the decision in a correct-calculator world” is a property of an entire execution-history sequence. This proposition is either true with respect to all the execution histories in the sequence, or false with respect to all of them.
The upshot is that, when you write “99% of ‘even’ worlds are the ones where calculator is correct, while you clearly assign 50% as probability of your event”, you are talking about two very different kinds of probabilities.
* Alternatively, this weighting can be incorporated into how the utility function over execution-history sequences responds to an event occurring in one world-program vs. another. If I had used this approach in my UDT1.1 formalization of your problem, I would have had just two world-programs: a correct-calculator world and an incorrect-calculator world. Then, having the correct parity on the answer sheet in the correct-calculator world would have been worth 99 times as much as having the correct parity in the incorrect-calculator world. But this would not have changed my computations. I don’t think that this issue is the locus of our present disagreement.
* You must be disagreeing with me by this point, because I have contradicted your claim that “Omega offers the decision in ‘even’ worlds, in some of which ‘even’ is correct, and in some of which it’s not*”. (Emphasis added.)
World-programs are a bad model for possible worlds. For all you know, there could be just one world-program (indeed you can consider an equivalent variant of the theory where it’s so: just have that single world program enumerate all outputs of all possible programs). The element of UDT analogous to possible worlds is execution histories. And some execution histories easily indicate that 2+2=5 (if we take execution histories to be enumerations of logical theories, with world-programs axiomatic definitions of theories). Observations, other background facts, and your actions are all elements that specify (sets/events of) execution histories. Utility function is defined on execution histories (and it’s usually defined on possible worlds). Probability given by mathematical intuition can be read as naming probability that given execution history (possible world) is an actual one.
So, you intended that the equivalence
“Omega offers the decision” <==> “the calculator says ‘even’ ”
be known to the agent’s mathematical intuition? I didn’t realize that, but my solution still applies without change. It just means that, as far as the agent’s mathematical intuition is concerned, we have the following equivalences between predicates over sequences of execution histories:
“Omega offers the decision in a correct-calculator world”
is equivalent to
“The calculator says ‘even’ in the 99 correct-calculator worlds”,
while
“Omega offers the decision in an incorrect-calculator world”
is equivalent to
“The calculator says ‘even’ in the one incorrect-calculator world”.
Below, I give my guess at your UDT1.1 approach to the problem in the OP. If I’m right, then we use the UDT1.1 concepts differently, but the math amounts to just a rearrangement of terms. I see merits in each conceptual approach over the other. I haven’t decided which one I like best.
At any rate, here is my guess at your formalization: We have one world-program. We consider the following one-place predicates over possible execution histories for this program: Given any execution history E,
CalculatorIsCorrect(E) asserts that, in E, the calculator gives the correct parity for Q.
“even”(E) asserts that, in E, the calculator says “even”. Omega then appears to the agent and asks it what Omega should have written on the test sheet in an execution history in which (1) Omega blocks the agent from writing on the answer sheet and (2) the calculator says “odd”.
“odd”(E) asserts that, in E, the calculator says “odd”. Omega then (1) blocks the agent from writing on the test sheet and (2) computes what the agent would have said to Omega in an execution history F such that “even”(F). Omega then writes what the agent would say in F on the answer sheet in E.
Borrowing notation from my last comment, we make the following assumptions about the probability measures P_f. For all input-output maps f,
P_f(CalculatorIsCorrect) = 0.99,
P_f(“even”) = P_f(“odd”) = 1⁄2,
“even” and “odd” are uncorrelated with CalculatorIsCorrect under P_f.
The input-output maps to consider are
g: On seeing “even”, write “even” and tell Omega, “Write ‘even’.”
h: On seeing “even”, write “even” and tell Omega, “Write ‘odd’.”
The utility U(E) of an execution history E is 1 if the answer on the sheet in E is the true parity of Q. Otherwise, U(E) = 0.
The expected payoffs of g and h are then, respectively,
EU(g) = P_g(“even” & CalculatorIsCorrect) 1 + P_g(“even” & ~CalculatorIsCorrect) 0 + P_g(“odd” & CalculatorIsCorrect) 0 + P_g(“odd” & ~CalculatorIsCorrect) 1 = 1⁄2 0.99 1 + 1⁄2 0.01 1 = 0.50.
EU(h) = P_h(“even” & CalculatorIsCorrect) 1 + P_h(“even” & ~CalculatorIsCorrect) 0 + P_h(“odd” & CalculatorIsCorrect) 1 + P_h(“odd” & ~CalculatorIsCorrect) 0 = 1⁄2 0.99 1 + 1⁄2 0.99 1 = 0.99.
I “weakly” argue for the 50% probability as well. My argument follows the Pearl-type of counterfactual (Drescher calls it “choice-friendly”) -- when you counterfactually set a variable, you cut directed arrows that lead to it, but not directed arrows that lead out or undirected arrows (which in another comment I mistakenly called bi-directed). My intuition is that the “causing” node might possibly be logically established before the “caused” node thus possibly leading to contradiction in the counterfactual, while the opposite direction is not possible (the “caused” node cannot be logically established earlier than the “causing” node). Directly logically establishing the counterfactual node is harmless in that it invalidates the counterfactual straight away, the argument “fears” of the “gap” where we possibly operate by using a contradictory counterfactual.
Pearl’s counterfactuals (or even causal diagrams) are unhelpful, as they ignore the finer points of logical control that are possibly relevant here. For example, that definitions (facts) are independent should refer to the absence of logical correlation between them, that is inability to infer (facts about) one from the other. But this, too, is shaky in the context of this puzzle, where the nature of logical knowledge is called into question.
Is it a trivial remark regarding the probability theory behind Pearl’s “causality”, or an intuition with regard to future theories that resemble Pearl’s approach?
It is a statement following from my investigation of logical/ambient control and reality-as-normative-anticipation thesis which I haven’t written much about, but this all is regardless called in question as adequate foundation in light of the thought experiment.