Note that world() essentially doesn’t talk about what can happen, instead it talks about how to compute utility, and computation of utility is a single fixed program without parameters (not even depending on the agent), that the agent “controls” from the inside.
To clarify, in the Newcomb example, the preference (world) program could be:
while the agent itself knows its code in the form agent3(). If it can prove both agent1() and agent2() to be equivalent to agent3(), it can decide in exactly the same way, but now the preference program doesn’t contain the agent even once, there is no explicit dependency of the world program (utility of the outcome) on the agent’s actions. Any dependency is logically inferred.
And we now have a prototype of the notion of preference: a fixed program that computes utility.
Yep—thanks. This is an important idea that I didn’t want to burden the post with. It would have brought the argument closer to UDT or ADT research, and I’d much prefer if you and Wei Dai wrote posts on these topics, not me. In matters of ethics and priority, one cannot be too careful. Besides, I want to read your posts because you have different perspectives from mine on these matters.
Thanks, I think I’ll write up this notion of preference remark (a little more generally, as a theory not program, with program as a special case, and lack of explicit dependence in the discussion of why the program/theory is “fixed”).
And we now have a prototype of the notion of preference: a fixed program that computes utility.
One possible problem: There is a difference between a program and the function it computes. The notion of preference is perhaps captured best by a function, not a program. Many programs could be written, all computing the same function. This wouldn’t matter much if we were only going to call or invoke the program, since all versions of the program compute the same result. But, this is not what cousin_it suggests here. He wants us to examine the source of the program, to prove theorems about the program. Given finite resources, the things we can prove about one program may not match the things we can prove about another program, even if the two programs would compute the same result in all cases if agent() returned the same result.
There is a difference between a program and the function it computes. The notion of preference is perhaps captured best by a function, not a program.
No, a program. What the program defines is a single constant value, not a function (remember: if we are talking about a program, it’s a constant program, taking no parameters!). And of course it matters how that constant is defined, and not at all what that constant is. More generally, we can define that constant not by a program, but by a logical theory, which will be the topic of my next post.
By a “logical theory” do you mean what logicians usually mean by a “theory”? A deductively closed set of sentences?
Wow! Well, that eliminates a lot of the arbitrary character I was objecting to in using programs to represent the world/decision problem. But there still are a lot of deductive systems to choose among. I await your next post with interest.
Note that world() essentially doesn’t talk about what can happen, instead it talks about how to compute utility, and computation of utility is a single fixed program without parameters (not even depending on the agent), that the agent “controls” from the inside.
To clarify, in the Newcomb example, the preference (world) program could be:
while the agent itself knows its code in the form agent3(). If it can prove both agent1() and agent2() to be equivalent to agent3(), it can decide in exactly the same way, but now the preference program doesn’t contain the agent even once, there is no explicit dependency of the world program (utility of the outcome) on the agent’s actions. Any dependency is logically inferred.
And we now have a prototype of the notion of preference: a fixed program that computes utility.
Yep—thanks. This is an important idea that I didn’t want to burden the post with. It would have brought the argument closer to UDT or ADT research, and I’d much prefer if you and Wei Dai wrote posts on these topics, not me. In matters of ethics and priority, one cannot be too careful. Besides, I want to read your posts because you have different perspectives from mine on these matters.
Thanks, I think I’ll write up this notion of preference remark (a little more generally, as a theory not program, with program as a special case, and lack of explicit dependence in the discussion of why the program/theory is “fixed”).
One possible problem: There is a difference between a program and the function it computes. The notion of preference is perhaps captured best by a function, not a program. Many programs could be written, all computing the same function. This wouldn’t matter much if we were only going to call or invoke the program, since all versions of the program compute the same result. But, this is not what cousin_it suggests here. He wants us to examine the source of the program, to prove theorems about the program. Given finite resources, the things we can prove about one program may not match the things we can prove about another program, even if the two programs would compute the same result in all cases if agent() returned the same result.
No, a program. What the program defines is a single constant value, not a function (remember: if we are talking about a program, it’s a constant program, taking no parameters!). And of course it matters how that constant is defined, and not at all what that constant is. More generally, we can define that constant not by a program, but by a logical theory, which will be the topic of my next post.
By a “logical theory” do you mean what logicians usually mean by a “theory”? A deductively closed set of sentences?
Wow! Well, that eliminates a lot of the arbitrary character I was objecting to in using programs to represent the world/decision problem. But there still are a lot of deductive systems to choose among. I await your next post with interest.
I won’t settle the choice, only point out the generality of notion and how it applies, direction to look for further refinements.
Yep, in my original email world() was called utility().