Robin Hanson’s ‘far mode’ (his take on construal level theory) is a plausible match to this ‘something’. Hanson points out that far mode is about general categories and creative metaphors. This is a match to something from AGI research...categorization and analogical inference. This can be linked to Bayesian inference by considering analogical inference as a natural way of reasoning about ‘priors’.
...and, for some extraordinary and unexplained reason, this ability causes an agent to have a goal G.
A plausible explanation is that analogical inference is associated with sentience (subjective experience), as suggested by Douglas Hofstadter (who has stated he thinks ‘analogies’ are the core of conscious cognition). Since sentience is closely associated with moral reasoning, it’s at least plausible that this ability could indeed give rise to converge on a particular G.
Where goal G is similarly undefined.
Here is a way G can be defined:
Analogical inference is concerned with Knowledge Representation (KR), so we could redefine ethics based on ‘representations of values’ (‘narratives’, which as Daniel Dennett has pointed out,indeed seem to be closely linked to subjective experience) rather than external consequences. At this point we can bring in the ideas of Schmidhuber and recall a powerful point made by Hanson (see below).
For maximum efficiency, all AGIs with the aforementioned ‘philosophical ability’ (analogical inference and production of narratives) would try to minimize the complexity of the cognitive processes generating its internal narratives. This could place universal contraints of what these values are. For example, Schmidhuber pointed out that data compression could be used to get a precise definition of ‘beauty’.
Lets now recall a powerful point Hanson made a while back on OB: the brain/mind can be totally defined in terms of a ‘signal processor’. Given this perspective, we could then view the correct G as the ‘signal’ and moral errors as ‘noise’. Algorithmic information theory could then be used to define a complexity metric that would precisely define this G.
Schmidthuber’s definition of beauty is wrong. He says, roughly, that you’re most pleased when after great effort you find a way to compress what was seemingly incompressible. If that were so, I could please you again and again by making up new AES keys with the first k bits random and the rest zero, and using them to generate and give you a few terabytes of random data. You’d have to brute force the key, at which point you’ll have compressed down from terabytes to kilobytes. What beauty! Let’s play the exact game again, with the exact same cipher but a different key, forever.
Right. That said, wireheading, aka the grounding problem, is a huge unsolved philosophical problem, so I’m not sure Schmidhuber is obligated to answer wireheading objections to his theory.
But the theory fails because this fits it but isn’t wireheading, right? It wouldn’t actually be pleasing to play that game.
I think you are right.
The two are errors that practically, with respect to hedonistic extremism, operate in opposing directions. They are similar in form in as much as they fit the abstract notion “undesirable outcomes due to lost purposes when choosing to optimize what turns out to be a poor metric for approximating actual preferences”.
Meh, yeah, maybe? Still seems like other, more substantive objections could be made.
Relatedly, I’m not entirely sure I buy Steve’s logic. PRNGs might not be nearly as interesting as short mathematical descriptions of complex things, like Chaitin’s omega. Arguably collecting as many bits of Chaitin’s omega as possible, or developing similar maths, would in fact be interesting in a human sense. But at that point our models really break down for many reasons, so meh whatever.
Right. That said, wireheading, aka the grounding problem, is a huge unsolved philosophical problem, so I’m not sure Schmidhuber is obligated to answer wireheading objections to his theory.
Unsolved philsophical problem? Huh? No additional philosophical breakthroughs are required for wireheading to not be a problem.
If I want (all things considered, etc) to wirehead, I’ll wirehead. If I don’t want to wirehead I will not wirehead. Wireheading introduces no special additional problems and is handled the same way all other preferences about future states of the universe can be handled.
(Note: It is likely that you have some more specific point regarding in what sense you consider wireheading ‘unsolved’. I welcome explanations or sources.)
Unsolved in the sense that we don’t know how to give computer intelligences intentional states in a way that everyone would be all like “wow that AI clearly has original intentionality and isn’t just coasting off of humans sitting at the end of the chain interpreting their otherwise entirely meaningless symbols”. Maybe this problem is just stupid and will solve itself but we don’t know that yet, hence e.g. Peter’s (unpublished?) paper on goal stability under ontological shifts. (ETA: I likely don’t understand how you’re thinking about the problem.)
Unsolved in the sense that we don’t know how to give computer intelligences intentional states in a way that everyone would be all like “wow that AI clearly has original intentionality and isn’t just coasting off of humans sitting at the end of the chain interpreting their otherwise entirely meaningless symbols”.
Being able to do this would also be a step towards the related goal of trying to give computer intelligences intelligence that we cannot construe as ‘intentionality’ in any morally salient sense, so as to satisfy any “house-elf-like” qualms that we may have.
e.g. Peter’s (unpublished?) paper on goal stability under ontological shifts.
Robin Hanson’s ‘far mode’ (his take on construal level theory) is a plausible match to this ‘something’. Hanson points out that far mode is about general categories and creative metaphors. This is a match to something from AGI research...categorization and analogical inference. This can be linked to Bayesian inference by considering analogical inference as a natural way of reasoning about ‘priors’.
A plausible explanation is that analogical inference is associated with sentience (subjective experience), as suggested by Douglas Hofstadter (who has stated he thinks ‘analogies’ are the core of conscious cognition). Since sentience is closely associated with moral reasoning, it’s at least plausible that this ability could indeed give rise to converge on a particular G.
Here is a way G can be defined:
Analogical inference is concerned with Knowledge Representation (KR), so we could redefine ethics based on ‘representations of values’ (‘narratives’, which as Daniel Dennett has pointed out,indeed seem to be closely linked to subjective experience) rather than external consequences. At this point we can bring in the ideas of Schmidhuber and recall a powerful point made by Hanson (see below).
For maximum efficiency, all AGIs with the aforementioned ‘philosophical ability’ (analogical inference and production of narratives) would try to minimize the complexity of the cognitive processes generating its internal narratives. This could place universal contraints of what these values are. For example, Schmidhuber pointed out that data compression could be used to get a precise definition of ‘beauty’.
Lets now recall a powerful point Hanson made a while back on OB: the brain/mind can be totally defined in terms of a ‘signal processor’. Given this perspective, we could then view the correct G as the ‘signal’ and moral errors as ‘noise’. Algorithmic information theory could then be used to define a complexity metric that would precisely define this G.
Schmidthuber’s definition of beauty is wrong. He says, roughly, that you’re most pleased when after great effort you find a way to compress what was seemingly incompressible. If that were so, I could please you again and again by making up new AES keys with the first k bits random and the rest zero, and using them to generate and give you a few terabytes of random data. You’d have to brute force the key, at which point you’ll have compressed down from terabytes to kilobytes. What beauty! Let’s play the exact game again, with the exact same cipher but a different key, forever.
Right. That said, wireheading, aka the grounding problem, is a huge unsolved philosophical problem, so I’m not sure Schmidhuber is obligated to answer wireheading objections to his theory.
But the theory fails because this fits it but isn’t wireheading, right? It wouldn’t actually be pleasing to play that game.
I think you are right.
The two are errors that practically, with respect to hedonistic extremism, operate in opposing directions. They are similar in form in as much as they fit the abstract notion “undesirable outcomes due to lost purposes when choosing to optimize what turns out to be a poor metric for approximating actual preferences”.
Meh, yeah, maybe? Still seems like other, more substantive objections could be made.
Relatedly, I’m not entirely sure I buy Steve’s logic. PRNGs might not be nearly as interesting as short mathematical descriptions of complex things, like Chaitin’s omega. Arguably collecting as many bits of Chaitin’s omega as possible, or developing similar maths, would in fact be interesting in a human sense. But at that point our models really break down for many reasons, so meh whatever.
Unsolved philsophical problem? Huh? No additional philosophical breakthroughs are required for wireheading to not be a problem.
If I want (all things considered, etc) to wirehead, I’ll wirehead. If I don’t want to wirehead I will not wirehead. Wireheading introduces no special additional problems and is handled the same way all other preferences about future states of the universe can be handled.
(Note: It is likely that you have some more specific point regarding in what sense you consider wireheading ‘unsolved’. I welcome explanations or sources.)
Unsolved in the sense that we don’t know how to give computer intelligences intentional states in a way that everyone would be all like “wow that AI clearly has original intentionality and isn’t just coasting off of humans sitting at the end of the chain interpreting their otherwise entirely meaningless symbols”. Maybe this problem is just stupid and will solve itself but we don’t know that yet, hence e.g. Peter’s (unpublished?) paper on goal stability under ontological shifts. (ETA: I likely don’t understand how you’re thinking about the problem.)
Being able to do this would also be a step towards the related goal of trying to give computer intelligences intelligence that we cannot construe as ‘intentionality’ in any morally salient sense, so as to satisfy any “house-elf-like” qualms that we may have.
I assume you mean Ontological Crises in Artificial Agents’ Value Systems? I just finished republishing that one. Originally published form. New SingInst style form. A good read.