I think you’re assuming that to give in to the mugging is the wrong answer in a one-shot game for a being that values all humans in existence equally, because it feels wrong to you, a being with a moral compass evolved in iterated multi-generational games.
Consider these possibilities, any one of which would create challenges for your reasoning:
1. Giving in is the right answer in a one-shot game, but the wrong answer in an iterated game. If you give in to the mugging, the outsider will keep mugging you and other rationalists until you’re all broke, leaving the universe’s future in the hands of “Marxians” and post-modernists.
2. Giving in is the right answer for a rational AI God, but evolved beings (under the Darwinian definition of “evolved”) can’t value all member of their species equally. They must value kin more than strangers. You would need a theory to explain why any being that evolved due to resource competition wouldn’t consider killing a large number of very distantly-related members of its species to be a good thing.
3. You should interpret the conflict between your intuition, and your desire for a rational God, not as showing that you’re reasoning badly because you’re evolved, but that you’re reasoning badly by desiring a rational God bound by a static utility function. This is complicated, so I’m gonna need more than one paragraph:
Intuitively, my argument boils down to applying the logic behind free markets, freedom of speech, and especially evolution, to the question of how to construct God’s utility function. This will be vague, but I think you can fill in the blanks.
Free-market economic theory developed only after millenia during which everyone believed that top-down control was the best way of allocating resources. Freedom of speech developed only after millenia during which everyone believed that it was rational for everyone to try to suppress any speech they disagreed with. Political liberalism developed only after millenia during which everybody believed that the best way to reform society was to figure out what the best society would be like, then force that on everyone. Evolution was conceived of—well, originally about 2500 years ago, probably by Democritus, but it became popular only after millenia during which everyone believed that life could be created only by design.
All of these developments came from empiricists. Empiricism is one of the two opposing philosophical traditions of Western thought. It originated, as far as we know, with Democritus (about whom Plato reportedly said that he wished all his works to be burned—which they eventually were). It went through the Skeptics, the Stoics, Lucretius, nominalism, the use of numeric measurements (re-introduced to the West circa 1300), the Renaissance and Enlightenment, and eventually (with the addition of evolution, probability, statistics, and operationalized terms) created modern science.
A key principle of empiricism, on which John Stuart Mill explicitly based his defense of free speech, is that we can never be certain. If you read about the skeptics and stoics today, you’ll read that they “believed nothing”, but that was because, to their opponents, “believe” meant “know something with 100% certainty”.
(The most-famous skeptic, Sextus Empiricus, was called “Empiricus” because he was of the empirical school of medicine, which taught learning from experience. Its opponent was the rational school of medicine, which used logic to interpret the dictums of the ancient authorities.)
The opposing philosophical tradition, founded by Plato—is rationalism. “Rational” does not mean “good thinking”. It has a very specific meaning, and it is not a good way of thinking. It means reasoning about the physical world the same way Euclid constructed geometric proofs. No measurements, no irrational numbers, no observation of the world, no operationalized nominalist definitions, no calculus or differential equations, no testing of hypotheses—just armchair a priori logic about universal categories, based on a set of unquestionable axioms, done in your favorite human language. Rationalism is the opposite of science, which is empirical. The pretense that “rational” means “right reasoning” is the greatest lie foisted on humanity by philosophers.
Dualist rationalism is inherently religious, as it relies on some concept of “spirit”, such as Plato’s Forms, Augustine’s God, Hegel’s World Spirit, or an almighty programmer converting sense data into LISP symbols, to connect the inexact, ambiguous, changeable things of this world to the precise, unambiguous, unchanging, and usually unquantified terms in its logic.
(Monist rationalists, like Buddha, Parmenides, and post-modernists, believe sense data can’t be divided unambiguously into categories, and thus we may not use categories. Modern empiricists categorize sense data using statistics.)
Rationalists support strict, rigid, top-down planning and control. This includes their opposition to free markets, free speech, gradual reform, and optimization and evolution in general. This is because rationalists believe they can prove things about the real world, and hence their conclusions are reliable, and they don’t need to mess around with slow, gradual improvements or with testing. (Of course each rationalist believes that every other rationalist was wrong, and should probably be burned at the stake.)
They oppose all randomness and disorder, because it makes strict top-down control difficult, and threatens to introduce change, which can only be bad once you’ve found the truth.
They have to classify every physical thing in the world into a discrete, structureless, atomic category, for use in their logic. That has led inevitably to theories which require all humans to ultimately have, at reflective equilibrium, the same values—as Plato, Augustine, Marx, and CEV all do.
You have, I think, picked up some of these bad inclinations from rationalism. When you say you want to find the “right” set of values (via CEV) and encode them into an AI God, that’s exactly like the rationalists who spent their lives trying to find the “right” way to live, and then suppress all other thoughts and enforce that “right way” on everyone, for all time. Whereas an empiricist would never claim to have found final truth, and would always leave room for new understandings and new developments.
Your objection to randomness is also typically rationalist. Randomness enables you to sample without bias. A rationalist believes he can achieve complete lack of bias; an empiricist believes that neither complete lack of bias nor complete randomness can be achieved, but that for a given amount of effort, you might achieve lower bias by working on your random number generator and using it to sample, than by hacking away at your biases.
So I don’t think we should build an FAI God who has a static set of values. We should build, if anything, an AI referee, who tries only to keep conditions in the universe that will enable evolution to keep on producing behaviors, concepts, and creatures of greater and greater complexity. Randomness must not be eliminated, for without randomness we can have no true exploration, and must be ruled forever by the beliefs and biases of the past.
Your overall point is right and important but most of your specific historical claims here are false—more mythical than real.
Free-market economic theory developed only after millenia during which everyone believed that top-down control was the best way of allocating resources.
Free market economic theory was developed during a period of rapid centralization of power, before which it was common sense that most resource allocation had to be done at the local level, letting peasants mostly alone to farm their own plots. To find a prior epoch of deliberate central resource management at scale you have to go back to the Bronze Age, with massive irrigation projects and other urban amenities built via palace economies, and even then there wasn’t really an ideology of centralization. A few Greek city-states like Sparta had tightly regulated mores for the elites, but the famously oppressed Helots were still probably mostly left alone. In Russia, Communism was a massive centralizing force—which implies that peasants had mostly been left alone beforehand. Centralization is about states trying to become more powerful (which is why Smith called his book The Wealth of Nations, pitching his message to the people who needed to be persuaded.) Tocqueville’s The Old Regime describes centralization in France before and after the Revolution. War and Peace has a good empirical treatment of the modernizing/centralizing force vs the old-fashioned empirical impulse in Russia. “Freedom” is not always decentralizing, though, as the book makes clear.
Freedom of speech developed only after millenia during which everyone believed that it was rational for everyone to try to suppress any speech they disagreed with.
There was something much like this in both the Athenian (and probably broader Greek) world (the democratic prerogative to publicly debate things), and the Israelite world (prophets normatively had something close to immunity from prosecution for speech, and there were no qualifications needed to prophesy). In both cases there were limits, but there are limits in our world too. The ideology of freedom of speech is new, but your characterization of the alternative is tendentious.
Political liberalism developed only after millenia during which everybody believed that the best way to reform society was to figure out what the best society would be like, then force that on everyone.
Political liberalism is not really an exception to this!
Evolution was conceived of—well, originally about 2500 years ago, probably by Democritus, but it became popular only after millenia during which everyone believed that life could be created only by design.
It’s really unclear what past generations meant by God, but this one is probably right.
I think you’re assuming that to give in to the mugging is the wrong answer in a one-shot game for a being that values all humans in existence equally, because it feels wrong to you, a being with a moral compass evolved in iterated multi-generational games.
Consider these possibilities, any one of which would create challenges for your reasoning:
1. Giving in is the right answer in a one-shot game, but the wrong answer in an iterated game. If you give in to the mugging, the outsider will keep mugging you and other rationalists until you’re all broke, leaving the universe’s future in the hands of “Marxians” and post-modernists.
2. Giving in is the right answer for a rational AI God, but evolved beings (under the Darwinian definition of “evolved”) can’t value all member of their species equally. They must value kin more than strangers. You would need a theory to explain why any being that evolved due to resource competition wouldn’t consider killing a large number of very distantly-related members of its species to be a good thing.
3. You should interpret the conflict between your intuition, and your desire for a rational God, not as showing that you’re reasoning badly because you’re evolved, but that you’re reasoning badly by desiring a rational God bound by a static utility function. This is complicated, so I’m gonna need more than one paragraph:
Intuitively, my argument boils down to applying the logic behind free markets, freedom of speech, and especially evolution, to the question of how to construct God’s utility function. This will be vague, but I think you can fill in the blanks.
Free-market economic theory developed only after millenia during which everyone believed that top-down control was the best way of allocating resources. Freedom of speech developed only after millenia during which everyone believed that it was rational for everyone to try to suppress any speech they disagreed with. Political liberalism developed only after millenia during which everybody believed that the best way to reform society was to figure out what the best society would be like, then force that on everyone. Evolution was conceived of—well, originally about 2500 years ago, probably by Democritus, but it became popular only after millenia during which everyone believed that life could be created only by design.
All of these developments came from empiricists. Empiricism is one of the two opposing philosophical traditions of Western thought. It originated, as far as we know, with Democritus (about whom Plato reportedly said that he wished all his works to be burned—which they eventually were). It went through the Skeptics, the Stoics, Lucretius, nominalism, the use of numeric measurements (re-introduced to the West circa 1300), the Renaissance and Enlightenment, and eventually (with the addition of evolution, probability, statistics, and operationalized terms) created modern science.
A key principle of empiricism, on which John Stuart Mill explicitly based his defense of free speech, is that we can never be certain. If you read about the skeptics and stoics today, you’ll read that they “believed nothing”, but that was because, to their opponents, “believe” meant “know something with 100% certainty”.
(The most-famous skeptic, Sextus Empiricus, was called “Empiricus” because he was of the empirical school of medicine, which taught learning from experience. Its opponent was the rational school of medicine, which used logic to interpret the dictums of the ancient authorities.)
The opposing philosophical tradition, founded by Plato—is rationalism. “Rational” does not mean “good thinking”. It has a very specific meaning, and it is not a good way of thinking. It means reasoning about the physical world the same way Euclid constructed geometric proofs. No measurements, no irrational numbers, no observation of the world, no operationalized nominalist definitions, no calculus or differential equations, no testing of hypotheses—just armchair a priori logic about universal categories, based on a set of unquestionable axioms, done in your favorite human language. Rationalism is the opposite of science, which is empirical. The pretense that “rational” means “right reasoning” is the greatest lie foisted on humanity by philosophers.
Dualist rationalism is inherently religious, as it relies on some concept of “spirit”, such as Plato’s Forms, Augustine’s God, Hegel’s World Spirit, or an almighty programmer converting sense data into LISP symbols, to connect the inexact, ambiguous, changeable things of this world to the precise, unambiguous, unchanging, and usually unquantified terms in its logic.
(Monist rationalists, like Buddha, Parmenides, and post-modernists, believe sense data can’t be divided unambiguously into categories, and thus we may not use categories. Modern empiricists categorize sense data using statistics.)
Rationalists support strict, rigid, top-down planning and control. This includes their opposition to free markets, free speech, gradual reform, and optimization and evolution in general. This is because rationalists believe they can prove things about the real world, and hence their conclusions are reliable, and they don’t need to mess around with slow, gradual improvements or with testing. (Of course each rationalist believes that every other rationalist was wrong, and should probably be burned at the stake.)
They oppose all randomness and disorder, because it makes strict top-down control difficult, and threatens to introduce change, which can only be bad once you’ve found the truth.
They have to classify every physical thing in the world into a discrete, structureless, atomic category, for use in their logic. That has led inevitably to theories which require all humans to ultimately have, at reflective equilibrium, the same values—as Plato, Augustine, Marx, and CEV all do.
You have, I think, picked up some of these bad inclinations from rationalism. When you say you want to find the “right” set of values (via CEV) and encode them into an AI God, that’s exactly like the rationalists who spent their lives trying to find the “right” way to live, and then suppress all other thoughts and enforce that “right way” on everyone, for all time. Whereas an empiricist would never claim to have found final truth, and would always leave room for new understandings and new developments.
Your objection to randomness is also typically rationalist. Randomness enables you to sample without bias. A rationalist believes he can achieve complete lack of bias; an empiricist believes that neither complete lack of bias nor complete randomness can be achieved, but that for a given amount of effort, you might achieve lower bias by working on your random number generator and using it to sample, than by hacking away at your biases.
So I don’t think we should build an FAI God who has a static set of values. We should build, if anything, an AI referee, who tries only to keep conditions in the universe that will enable evolution to keep on producing behaviors, concepts, and creatures of greater and greater complexity. Randomness must not be eliminated, for without randomness we can have no true exploration, and must be ruled forever by the beliefs and biases of the past.
Your overall point is right and important but most of your specific historical claims here are false—more mythical than real.
Free market economic theory was developed during a period of rapid centralization of power, before which it was common sense that most resource allocation had to be done at the local level, letting peasants mostly alone to farm their own plots. To find a prior epoch of deliberate central resource management at scale you have to go back to the Bronze Age, with massive irrigation projects and other urban amenities built via palace economies, and even then there wasn’t really an ideology of centralization. A few Greek city-states like Sparta had tightly regulated mores for the elites, but the famously oppressed Helots were still probably mostly left alone. In Russia, Communism was a massive centralizing force—which implies that peasants had mostly been left alone beforehand. Centralization is about states trying to become more powerful (which is why Smith called his book The Wealth of Nations, pitching his message to the people who needed to be persuaded.) Tocqueville’s The Old Regime describes centralization in France before and after the Revolution. War and Peace has a good empirical treatment of the modernizing/centralizing force vs the old-fashioned empirical impulse in Russia. “Freedom” is not always decentralizing, though, as the book makes clear.
There was something much like this in both the Athenian (and probably broader Greek) world (the democratic prerogative to publicly debate things), and the Israelite world (prophets normatively had something close to immunity from prosecution for speech, and there were no qualifications needed to prophesy). In both cases there were limits, but there are limits in our world too. The ideology of freedom of speech is new, but your characterization of the alternative is tendentious.
Political liberalism is not really an exception to this!
It’s really unclear what past generations meant by God, but this one is probably right.