But I’m not sure it deserves one; would CDT really be a probable output anywhere besides a verbal theory advocated by human philosophers in our own Everett branch? Maybe, now that I think about it, but even so, does it matter?
Yes, for reasons of game theory and of practical singularity strategy.
Game theory, because things in Everett branches that are ‘closest’ to us might be the ones it’s most important to be able to interact with, since they’re easier to simulate and their preferences are more likely to have interesting overlap with ours. Knowing very roughly what to expect from our neighbors is useful.
And singularity strategy, because if you can show that architectures like AIXI-tl have some non-negligible chance of converging to whatever an FAI would have converged to, as far as actual policies go, then that is a very important thing to know; especially if a non-uFAI existential risk starts to look imminent (but the game theory in that case is crazy). It is not probable but there’s a hell of a lot of structural uncertainty and Omohundro’s AI drives are still pretty informal. I am still not absolutely sure I know how a self-modifying superintelligence would interpret or reflect on its utility function or terms therein (or how it would reflect on its implicit policy for interpreting or reflecting on utility functions or terms therein). The apparent rigidity of Goedel machines might constitute a disproof in theory (though I’m not sure about that), but when some of the terms are sequences of letters like “makeHumansHappy” or formally manipulable correlated markers of human happiness, then I don’t know how the syntax gets turned into semantics (or fails entirely to get turned into semantics, as they case may well be).
But it will calculate that expected value using CDT!expectation, meaning that it won’t see how self-modifying to be a timeless decision theorist could possibly affect what’s already in the box, etcetera.
This implies that the actually-implemented-CDT agent has a single level of abstraction/granularity at like the naive realist physical level at which it’s proving things about causal relationships. Like, it can’t/shouldn’t prove causal relationships at the level of string theory, and yet it’s still confident that its actions are causing things despite that structural uncertainty, and yet despite the symmetry it for some reason cannot possibly see how switching a few transistors or changing its decision policy might affect things via relationships that are ultimately causal but currently unknown for reasons of boundedness and not speculative metaphysics. It’s plausible, but I think letting a universal hypothesis space or maybe even just Goedelian limitations enter the decision calculus at any point is going to make such rigidity unlikely. (This is related to how a non-hypercomputation-driven decision theory in general might reason about the possibility of hypercomputation, or the risk of self-diagonalization, I think.)
Yes, for reasons of game theory and of practical singularity strategy.
Game theory, because things in Everett branches that are ‘closest’ to us might be the ones it’s most important to be able to interact with, since they’re easier to simulate and their preferences are more likely to have interesting overlap with ours. Knowing very roughly what to expect from our neighbors is useful.
And singularity strategy, because if you can show that architectures like AIXI-tl have some non-negligible chance of converging to whatever an FAI would have converged to, as far as actual policies go, then that is a very important thing to know; especially if a non-uFAI existential risk starts to look imminent (but the game theory in that case is crazy). It is not probable but there’s a hell of a lot of structural uncertainty and Omohundro’s AI drives are still pretty informal. I am still not absolutely sure I know how a self-modifying superintelligence would interpret or reflect on its utility function or terms therein (or how it would reflect on its implicit policy for interpreting or reflecting on utility functions or terms therein). The apparent rigidity of Goedel machines might constitute a disproof in theory (though I’m not sure about that), but when some of the terms are sequences of letters like “makeHumansHappy” or formally manipulable correlated markers of human happiness, then I don’t know how the syntax gets turned into semantics (or fails entirely to get turned into semantics, as they case may well be).
This implies that the actually-implemented-CDT agent has a single level of abstraction/granularity at like the naive realist physical level at which it’s proving things about causal relationships. Like, it can’t/shouldn’t prove causal relationships at the level of string theory, and yet it’s still confident that its actions are causing things despite that structural uncertainty, and yet despite the symmetry it for some reason cannot possibly see how switching a few transistors or changing its decision policy might affect things via relationships that are ultimately causal but currently unknown for reasons of boundedness and not speculative metaphysics. It’s plausible, but I think letting a universal hypothesis space or maybe even just Goedelian limitations enter the decision calculus at any point is going to make such rigidity unlikely. (This is related to how a non-hypercomputation-driven decision theory in general might reason about the possibility of hypercomputation, or the risk of self-diagonalization, I think.)