I once read an interesting book by Kurt Vonnegut called Player Piano that focused on an increasingly automated society (where one was not allowed to have a ‘creative’ job, for example, unless the punch card that assessed your general intelligence by way of standardized test happened to have the ‘creative’ slot punched; and where out of work (displaced by robots) auto mechanics pined over the good ole days when they could ply their trade for a living rather than seeing their vocation-which-they-also-had-creative-passion-for become the work of robots.)
In particular, a character (a discontented engineer who goes Luddite and becomes a farmer) has this to say in a climactic letter in the story:
“You perhaps disagree with the antique and vain notion of Man’s being a creation of God. … But I find it a far more defensible belief than the one implicit in intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence.”
I found this quite interesting and simply wonder what LWers think about this premise. I disagree with the idea that religious faith in God is somehow more defensible than persistent technological progress for the sake of making myself more durable and efficient (and presumably we can extend the quote to include ‘rational’). However, I don’t necessarily disagree outright that “intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence”—is not strongly defensible. That phrase at the end … “eliminate any justification for his own continued existence” is quite resonant.
Might we not be able to define intelligence in this way? Intelligence is a property such that once I have it, then I am personally less on the hook for expending my own resources to sustain my own existence. In a slight sense, intelligence renders its owner less and less necessary for the actual work that goes into self preservation.
The sense in which I think this applies is as follows: suppose my utility function heavily values my perception of sustained existence. But at the same time, at least some component of my utility is derived from perceiving a “sense of meaning” out of life (a concrete definition of that could be debated endlessly, so please lets just go with a basic natural language understanding of that for right now). If I feel that increase in intelligence improves probability of survival but diminishes personal usefulness and “meaning”, then it is at least possible that there is some finite largest amount of intelligence (quantified in terms of efficient optimization power if you so choose) such that beyond that intelligence horizon, my perceived quality of life actually decreases because my perceived personal meaning drops low enough to offset the marginal increased assurance of longer(/more comfortable) life.
I think this is a non-trivial theory about intelligence. This “intelligence horizon” may not be something that humans could even begin to encounter, as most of us seem perfectly capable of extracting “meaning” out of life despite technological progress. But this isn’t the same as knowing a fundamental reason why all intelligences would always value a unit improvement in longevity over a unit improvement in “personal meaning”.
For example, suppose that many billions of years into the future there have taken place several great battles between Bayesian superintelligences forced to hostility over scarcity of resource so severe that even their combined superintelligent efforts at collaboration could not solve the problems. One triumphant superintelligence remains and is confident that the probability it is the last remaining life within its light cone is near 1. Suppose it goes on to self colonize territory and resources until it is confident that the probability that there is additional resource to consume is near zero (here I mean that it believes with a high degree of confidence that it has located all available resource that it can physically access).
Now what does it do? What is the psychology of such a being? I don’t even pretend to know a good answer to this sort of question, but I am sure other LWers will have good things to say. But you can clearly see that as this last Bayesian intelligence completes tasks, it directly loses purpose. Unless it does not weigh a sense of purpose into its utility function, this would be problematic. Maybe it would start to intentionally solve problems very slowly so that it required a very long time to finish. Maybe its slowness would increase as time goes on to ensure that it never actually does finish the task?
It’d be cool if, based on knowing something about your light cone, the problems needing solved during your lifetime, and the imbued method by which your utility function assigns weight to a sense of purpose, you could compute an optimum life span… I mean, suppose the universe was the interval [1,N] for some very large N and my utility function under the scenario where I set myself to just optimize indefinitely and make myself more durable and efficient starts looking like 1/x^2 after a long time, say from integer M to N where M is also very large. In order that we can even have an expected utility (which I think is a reasonable assumption) then any utility I choose ought to be rectifiable on [1,N], so no matter what the integral works out to be, I could find utility functions that actually hit zero (assuming non-negative utility, or hitting -max{utility} for bounded but possibly negative utility, or the debate is over if death → -\infty utility but this doesn’t seem plausible given the occurrence of suicide) at a time before N (corresponding to death before N) but for which the total experienced utility is higher.
[Note: this is not an attempt to pin down every rigorous detail about utility functions… just to illustrate the naive, simplistic view that even simple utility scenarios can lead to counter-intuitive decisions. The opportunity to have these sorts of situations would only increase as utility functions become more realistic and are based upon more realistic models of the universe. Imagine if you had access to your own utility function and a Taylor series approximation of the evolving quantum states that will “affect you”. If you can compute any kind of meaningful cohesive extrapolated volition, why couldn’t you predict your reaction to the predicament of being in a situation like that of Sisyphus and whether or not dying earlier would be better if the overall effect was that your total experienced utility increased? Can there possibly be experiences of finite duration that are so awesome in terms of utility that they outweigh futures in which you are dead? What if you were a chess player and your chess utility function was such that if you could promote 6 pawns to queens that your enjoyment of that bizarre novelty occurring during gameplay was worth way more than winning the game and so even if you saw that the opponent had a forced mate, you’d willingly walk right into it in order to get the sweet sweet novelty of the 6 queens?]
A simple example would be that, when facing the choice between unimaginable torture from which you willingly accept that the probability of escape is near zero or suicide, there could be physically meaningful utility functions (perhaps even ‘optimal’ utility functions) that would rationally choose suicide. A Bayesian superintelligence might view a solitary light cone existence with no tasks to derive purpose from as equivalent to a form of this sort of torture and thus see that if it just “acts dumber but dies sooner but lives the remaining years with more enjoyable purpose” it will actually have more total utility. Maybe it will shoot itself in its superintelligent foot and erase all memory of the shooting and replace it with some different explanation for why there is a bullet hole. Then the future stupider AI will be in a predicament where it realizes it needs to make itself smarter (up to the former level of intelligence that it erased from its own memory) but is somehow handicapped in a way that it’s just barely out of reach (but chasing the ever-out-of-reach carrot gives it meaning). Would it rather be Sisyphus or David Foster Wallace or the third option that I am most likely failing to see?
Addition
I thought of one idea for what the Bayesian intelligence might do after all goals had been achieved for its own self-preservation, as near as it could reckon. The idea is that is might run a simulation resulting in other lifeforms. Would it be fair to say that the probability of it choosing to do this skyrockets to close to 1 as the total number of other living beings goes toward zero? If so, then as per the link above, does this make us more compelled to believe we’re currently in the simulation of a last Bayesian intelligence in a significantly mature metaverse (don’t tell the Christians)? My gut says probably not but I can’t think of concrete compelling reasons for why this can be easily dismissed. I’ll feel better if anyone can Swiss-cheesify my line of thinking.
I once read an interesting book by Kurt Vonnegut called Player Piano that focused on an increasingly automated society (where one was not allowed to have a ‘creative’ job, for example, unless the punch card that assessed your general intelligence by way of standardized test happened to have the ‘creative’ slot punched; and where out of work (displaced by robots) auto mechanics pined over the good ole days when they could ply their trade for a living rather than seeing their vocation-which-they-also-had-creative-passion-for become the work of robots.)
In particular, a character (a discontented engineer who goes Luddite and becomes a farmer) has this to say in a climactic letter in the story:
“You perhaps disagree with the antique and vain notion of Man’s being a creation of God. … But I find it a far more defensible belief than the one implicit in intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence.”
I found this quite interesting and simply wonder what LWers think about this premise. I disagree with the idea that religious faith in God is somehow more defensible than persistent technological progress for the sake of making myself more durable and efficient (and presumably we can extend the quote to include ‘rational’). However, I don’t necessarily disagree outright that “intemperate faith in lawless technological progress—namely, that man is on earth to create more durable and efficient images of himself, and, hence, to eliminate any justification at all for his own continued existence”—is not strongly defensible. That phrase at the end … “eliminate any justification for his own continued existence” is quite resonant.
Might we not be able to define intelligence in this way? Intelligence is a property such that once I have it, then I am personally less on the hook for expending my own resources to sustain my own existence. In a slight sense, intelligence renders its owner less and less necessary for the actual work that goes into self preservation.
The sense in which I think this applies is as follows: suppose my utility function heavily values my perception of sustained existence. But at the same time, at least some component of my utility is derived from perceiving a “sense of meaning” out of life (a concrete definition of that could be debated endlessly, so please lets just go with a basic natural language understanding of that for right now). If I feel that increase in intelligence improves probability of survival but diminishes personal usefulness and “meaning”, then it is at least possible that there is some finite largest amount of intelligence (quantified in terms of efficient optimization power if you so choose) such that beyond that intelligence horizon, my perceived quality of life actually decreases because my perceived personal meaning drops low enough to offset the marginal increased assurance of longer(/more comfortable) life.
I think this is a non-trivial theory about intelligence. This “intelligence horizon” may not be something that humans could even begin to encounter, as most of us seem perfectly capable of extracting “meaning” out of life despite technological progress. But this isn’t the same as knowing a fundamental reason why all intelligences would always value a unit improvement in longevity over a unit improvement in “personal meaning”.
For example, suppose that many billions of years into the future there have taken place several great battles between Bayesian superintelligences forced to hostility over scarcity of resource so severe that even their combined superintelligent efforts at collaboration could not solve the problems. One triumphant superintelligence remains and is confident that the probability it is the last remaining life within its light cone is near 1. Suppose it goes on to self colonize territory and resources until it is confident that the probability that there is additional resource to consume is near zero (here I mean that it believes with a high degree of confidence that it has located all available resource that it can physically access).
Now what does it do? What is the psychology of such a being? I don’t even pretend to know a good answer to this sort of question, but I am sure other LWers will have good things to say. But you can clearly see that as this last Bayesian intelligence completes tasks, it directly loses purpose. Unless it does not weigh a sense of purpose into its utility function, this would be problematic. Maybe it would start to intentionally solve problems very slowly so that it required a very long time to finish. Maybe its slowness would increase as time goes on to ensure that it never actually does finish the task?
It’d be cool if, based on knowing something about your light cone, the problems needing solved during your lifetime, and the imbued method by which your utility function assigns weight to a sense of purpose, you could compute an optimum life span… I mean, suppose the universe was the interval [1,N] for some very large N and my utility function under the scenario where I set myself to just optimize indefinitely and make myself more durable and efficient starts looking like 1/x^2 after a long time, say from integer M to N where M is also very large. In order that we can even have an expected utility (which I think is a reasonable assumption) then any utility I choose ought to be rectifiable on [1,N], so no matter what the integral works out to be, I could find utility functions that actually hit zero (assuming non-negative utility, or hitting -max{utility} for bounded but possibly negative utility, or the debate is over if death → -\infty utility but this doesn’t seem plausible given the occurrence of suicide) at a time before N (corresponding to death before N) but for which the total experienced utility is higher.
[Note: this is not an attempt to pin down every rigorous detail about utility functions… just to illustrate the naive, simplistic view that even simple utility scenarios can lead to counter-intuitive decisions. The opportunity to have these sorts of situations would only increase as utility functions become more realistic and are based upon more realistic models of the universe. Imagine if you had access to your own utility function and a Taylor series approximation of the evolving quantum states that will “affect you”. If you can compute any kind of meaningful cohesive extrapolated volition, why couldn’t you predict your reaction to the predicament of being in a situation like that of Sisyphus and whether or not dying earlier would be better if the overall effect was that your total experienced utility increased? Can there possibly be experiences of finite duration that are so awesome in terms of utility that they outweigh futures in which you are dead? What if you were a chess player and your chess utility function was such that if you could promote 6 pawns to queens that your enjoyment of that bizarre novelty occurring during gameplay was worth way more than winning the game and so even if you saw that the opponent had a forced mate, you’d willingly walk right into it in order to get the sweet sweet novelty of the 6 queens?]
A simple example would be that, when facing the choice between unimaginable torture from which you willingly accept that the probability of escape is near zero or suicide, there could be physically meaningful utility functions (perhaps even ‘optimal’ utility functions) that would rationally choose suicide. A Bayesian superintelligence might view a solitary light cone existence with no tasks to derive purpose from as equivalent to a form of this sort of torture and thus see that if it just “acts dumber but dies sooner but lives the remaining years with more enjoyable purpose” it will actually have more total utility. Maybe it will shoot itself in its superintelligent foot and erase all memory of the shooting and replace it with some different explanation for why there is a bullet hole. Then the future stupider AI will be in a predicament where it realizes it needs to make itself smarter (up to the former level of intelligence that it erased from its own memory) but is somehow handicapped in a way that it’s just barely out of reach (but chasing the ever-out-of-reach carrot gives it meaning). Would it rather be Sisyphus or David Foster Wallace or the third option that I am most likely failing to see?
Addition
I thought of one idea for what the Bayesian intelligence might do after all goals had been achieved for its own self-preservation, as near as it could reckon. The idea is that is might run a simulation resulting in other lifeforms. Would it be fair to say that the probability of it choosing to do this skyrockets to close to 1 as the total number of other living beings goes toward zero? If so, then as per the link above, does this make us more compelled to believe we’re currently in the simulation of a last Bayesian intelligence in a significantly mature metaverse (don’t tell the Christians)? My gut says probably not but I can’t think of concrete compelling reasons for why this can be easily dismissed. I’ll feel better if anyone can Swiss-cheesify my line of thinking.
This should be a post, on a personal blog if not on LW.