I show that, for this particular game (the one in your “explicit optimization” post), the winning mathematical intuitions are the only ones that meet certain reasonable criteria.
(1) Which one of them will actually be given? (2) If there is no sense in which some of these “reasonable” conclusions are better than each other, why do you single them out, rather than mathematical intuitions expressing uncertainty about the outcomes that would express the lack of priority of some of these outcomes over others?
I don’t find the certainty of conclusions a reasonable assumption, in particular because, as you can see, you can’t unambiguously decide which of the conclusions is the right one, and so can’t the agent.
I claim to be giving, at best, a subset of “reasonable criteria” for mathematical intuition functions. Any UDT1-builder who uses a superset of these criteria, and who has enough decision criteria to decide which UDT1 agent to write, will write an agent who wins Wei’s game. In this case, it would suffice to have the criteria I mentioned plus a lexicographic tie-breaker (as in UDT1.1). I’m not optimistic that that will hold in general.
(I also wouldn’t be surprised to see an example showing that my “counterfactual accuracy” condition, as stated, rules out all winning UDT1 algorithms in some other game. I find it pretty unlikely that it suffices to deal with mathematical counterfactuals in such a simple way, even given the binary certainty and accuracy conditions.)
My point was only that the criteria above already suffice to narrow the field of options for the builder down to winning options. Hence, whatever superset of these criteria the builder uses, this superset doesn’t need to include any knowledge about which possible UDT1 agent would win.
(2) If there is no sense in which some of these “reasonable” conclusions are better than each other, why do you single them out, rather than mathematical intuitions expressing uncertainty about the outcomes that would express the lack of priority of some of these outcomes over others?
I don’t follow. Are you suggesting that I could just as reasonably have made it a condition of any acceptable mathematical intuition function that M(1, A, E) = 0.5 ?
I don’t find the certainty of conclusions a reasonable assumption, in particular because, as you can see, you can’t unambiguously decide which of the conclusions is the right one, and so can’t the agent.
If I (the builder/writer) really couldn’t decide which mathematical intuition function to use, then the agent won’t come to exist in the first place. If I can’t choose among the two options that remain after I apply the described criteria, then I will be frozen in indecision, and no agent will get built or written. I take it that this is your point.
But if I do have enough additional criteria to decide (which in this case could be just a lexicographic tie-breaker), then I don’t see what is unreasonable about the “certainty of conclusions” assumption for this game.
If I (the builder/writer) really couldn’t decide which mathematical intuition function to use, then the agent won’t come to exist in the first place.
You don’t pick the output of mathematical intuition in a particular case, mathematical intuition is a general algorithm that works based on world programs, outcomes, and your proposed decisions. It’s computationally intensive, its results are not specified in advance based on intuition, on the contrary the algorithm is what stands for intuition. With more resources, this algorithm will produce different probabilities, as it comes to understand the problem better. And you just pick the algorithm. What you can say about its outcome is a matter of understanding the requirements for such general algorithm, and predicting what it must therefore compute. Absolute certainty of the algorithm, for example, would imply that the algorithm managed to logically infer that the outcome would be so and so, and I don’t see how it’s possible to do that, given the problem statement. If it’s unclear how to infer what will happen, then mathematical intuition should be uncertain (but it can know something to tilt the balance one way a little bit, perhaps enough to decide the coordination problem!)
There is a single ideal mathematical intuition, which, given a particular amount of resources, and a particular game, determines a unique function M mapping {inputs} x {outputs} x {execution histories} --> [0,1] for a UDT1 agent in that game. This ideal mathematical intuition (IMI) is defined by the very nature of logical or mathematical inference under computational limitation. So, in particular, it’s not something that you can talk about choosing using some arbitrary tie-breaker like lexicographic order.
Now, maybe the IMI requires that the function M be binary in some particular game with some particular amount of resources. Or maybe the IMI requires a non-binary function M for all amounts of computational resources in that game. Unless you can explain exactly why the IMI requires a binary function M for this particular game, you haven’t really made progress on the kinds of questions that we’re interested in.
More or less. Of course there is no point in going for a “single” mathematical intuition, but the criteria for choosing one shouldn’t be specific to a particular game. Mathematical intuition primarily works with the world program, trying to estimate how plausible it is that this world program will be equivalent to a given history definition, under the condition that the agent produces given output.
(1) Which one of them will actually be given? (2) If there is no sense in which some of these “reasonable” conclusions are better than each other, why do you single them out, rather than mathematical intuitions expressing uncertainty about the outcomes that would express the lack of priority of some of these outcomes over others?
I don’t find the certainty of conclusions a reasonable assumption, in particular because, as you can see, you can’t unambiguously decide which of the conclusions is the right one, and so can’t the agent.
I claim to be giving, at best, a subset of “reasonable criteria” for mathematical intuition functions. Any UDT1-builder who uses a superset of these criteria, and who has enough decision criteria to decide which UDT1 agent to write, will write an agent who wins Wei’s game. In this case, it would suffice to have the criteria I mentioned plus a lexicographic tie-breaker (as in UDT1.1). I’m not optimistic that that will hold in general.
(I also wouldn’t be surprised to see an example showing that my “counterfactual accuracy” condition, as stated, rules out all winning UDT1 algorithms in some other game. I find it pretty unlikely that it suffices to deal with mathematical counterfactuals in such a simple way, even given the binary certainty and accuracy conditions.)
My point was only that the criteria above already suffice to narrow the field of options for the builder down to winning options. Hence, whatever superset of these criteria the builder uses, this superset doesn’t need to include any knowledge about which possible UDT1 agent would win.
I don’t follow. Are you suggesting that I could just as reasonably have made it a condition of any acceptable mathematical intuition function that M(1, A, E) = 0.5 ?
If I (the builder/writer) really couldn’t decide which mathematical intuition function to use, then the agent won’t come to exist in the first place. If I can’t choose among the two options that remain after I apply the described criteria, then I will be frozen in indecision, and no agent will get built or written. I take it that this is your point.
But if I do have enough additional criteria to decide (which in this case could be just a lexicographic tie-breaker), then I don’t see what is unreasonable about the “certainty of conclusions” assumption for this game.
You don’t pick the output of mathematical intuition in a particular case, mathematical intuition is a general algorithm that works based on world programs, outcomes, and your proposed decisions. It’s computationally intensive, its results are not specified in advance based on intuition, on the contrary the algorithm is what stands for intuition. With more resources, this algorithm will produce different probabilities, as it comes to understand the problem better. And you just pick the algorithm. What you can say about its outcome is a matter of understanding the requirements for such general algorithm, and predicting what it must therefore compute. Absolute certainty of the algorithm, for example, would imply that the algorithm managed to logically infer that the outcome would be so and so, and I don’t see how it’s possible to do that, given the problem statement. If it’s unclear how to infer what will happen, then mathematical intuition should be uncertain (but it can know something to tilt the balance one way a little bit, perhaps enough to decide the coordination problem!)
Okay, I understand you to be saying this:
There is a single ideal mathematical intuition, which, given a particular amount of resources, and a particular game, determines a unique function M mapping {inputs} x {outputs} x {execution histories} --> [0,1] for a UDT1 agent in that game. This ideal mathematical intuition (IMI) is defined by the very nature of logical or mathematical inference under computational limitation. So, in particular, it’s not something that you can talk about choosing using some arbitrary tie-breaker like lexicographic order.
Now, maybe the IMI requires that the function M be binary in some particular game with some particular amount of resources. Or maybe the IMI requires a non-binary function M for all amounts of computational resources in that game. Unless you can explain exactly why the IMI requires a binary function M for this particular game, you haven’t really made progress on the kinds of questions that we’re interested in.
Is that right?
More or less. Of course there is no point in going for a “single” mathematical intuition, but the criteria for choosing one shouldn’t be specific to a particular game. Mathematical intuition primarily works with the world program, trying to estimate how plausible it is that this world program will be equivalent to a given history definition, under the condition that the agent produces given output.