One might expect self-improving systems to be highly unpredictable because the properties of the current version might change in the next version. Our analysis will instead show that self-improvement acts to create predictable regularities. It builds on the intellectual foundations of microeconomics, the science of preference and choice in the face of uncertainty. The basic theory was created by John von Neumann and Oskar Morgenstern in 1944 for situations with objective uncertainty and was later extended by Savage and Anscombe and Aumann to situations with subjective uncertainty. Our analysis shows that while the preferences of self-improving systems will depend on their origins, they will act on those preferences in predictable ways. Repeated self-improvement brings intelligent agents closer to an ideal that economists sometimes call “Homo Economicus”. Ironically, human behavior is not well described by this ideal and the field of “behavioral economics” has emerged in recent years to study how humans actually behave. The classical economic theory is much more applicable to self-improving systems because they will discover and eliminate their own irrationalities in ways that humans cannot.
Maybe it doesn’t belong. But I was thinking in terms of something like rationality being an attractor. Minds, whatever their origin, if capable of self-improving, will tend toward a pattern which human economists had already identified as being at the heart of human rationality.
The rational direction to guide your own improvement is toward greater rationality. Even if you are not all that rational to begin with. That means that the characteristics we assign to modeled “rational agents” may be universal—they are not just something invented by some lackey of a capitalist patron.
Unless Omohundro’s analysis is wrong and he just wrote it because he is a lackey, that is.
Some human irrationality seems adaptive. Humans apparently deceive themselves so they can manipulate others without actually lying—so as to avoid detection.
It may be that parts of our unconscious mind are well aware of many of our deceptions, and it is only our conscious mind that is fooled. Our conscious minds may even be like the public relations department of a corporation, whose main purpose is to present a certain coherent image to the outside, and not to make key
corporate policy decisions. - source
Some human irrationality seems adaptive. Humans apparently deceive themselves so they can manipulate others without actually lying—so as to avoid detection.
That does not directly contradict Omohundro. The quotation merely suggests that almost-rational humans will seek to self-modify in the direction of becoming less self-deceptive and better at lying. A look at the self-help literature tends to confirm Omohundro’s prediction.
That leaves the question, though, as to why Natural Selection didn’t take care of this ‘improvement’ itself. My guess is that it is a life-history, levels-of-selection, and kin-selection issue. Self-help books are purchased by adults. NS tries to optimize the whole life history. It is good for neither children nor their families that they become accomplished liars. Maybe self-deception in children has some advantages as well. Just speculating.
Some human irrationality seems adaptive. Humans apparently deceive
themselves so they can manipulate others without actually lying—so as
to avoid detection.
That does not directly contradict Omohundro. The quotation merely suggests that almost-rational humans will seek to self-modify in the direction of becoming less self-deceptive and better at lying. A look at the self-help literature tends to confirm Omohundro’s prediction.
Are the liars going to win, though? Nature subsidises both transparency and lie detectors, for reasons to do with promoting cooperation. In the future it may get even harder to convince others of things you don’t personally believe—as is dramatically portrayed in The Truth Machine.
Steve Omohundro, “The Nature of Self-Improving Artificial Intelligence” 2007
It’s an interesting topic, but what exactly makes this a rationality quote?
Maybe it doesn’t belong. But I was thinking in terms of something like rationality being an attractor. Minds, whatever their origin, if capable of self-improving, will tend toward a pattern which human economists had already identified as being at the heart of human rationality.
The rational direction to guide your own improvement is toward greater rationality. Even if you are not all that rational to begin with. That means that the characteristics we assign to modeled “rational agents” may be universal—they are not just something invented by some lackey of a capitalist patron.
Unless Omohundro’s analysis is wrong and he just wrote it because he is a lackey, that is.
Some human irrationality seems adaptive. Humans apparently deceive themselves so they can manipulate others without actually lying—so as to avoid detection.
That does not directly contradict Omohundro. The quotation merely suggests that almost-rational humans will seek to self-modify in the direction of becoming less self-deceptive and better at lying. A look at the self-help literature tends to confirm Omohundro’s prediction.
That leaves the question, though, as to why Natural Selection didn’t take care of this ‘improvement’ itself. My guess is that it is a life-history, levels-of-selection, and kin-selection issue. Self-help books are purchased by adults. NS tries to optimize the whole life history. It is good for neither children nor their families that they become accomplished liars. Maybe self-deception in children has some advantages as well. Just speculating.
Are the liars going to win, though? Nature subsidises both transparency and lie detectors, for reasons to do with promoting cooperation. In the future it may get even harder to convince others of things you don’t personally believe—as is dramatically portrayed in The Truth Machine.