It’s my contention that rationality should offer guidance in figuring out what goals you should have. A rationalist society will have goals closer to “defeat death and grasp the stars” than “gain ALL the status”. It’s not just rationalists who should win, it’s rational societies who should win. If you’re in a society that is insane then you may not be able to “win” as a rationalist. In that case your goal should not be “winning” in the social-traditional sense, it should be making society sane.
I don’t think that that’s a bad thing. The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”. The point of the post was to warn that not only answering our questions but figuring out which questions we should ask is an issue we have to tackle. We have to figure out what winning should be.
The reason that the immortal starfarers are better is that they’re trying to do that, so if all values aren’t created equally, they’re more likely to find out about it.
The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”.
Deciding that going somewhere is “winning” comes from your existing utility function.
Another person could judge that the civilization with the most rich and complex social hierarchy “wins”.
Rationality can help you search the space of actions, policies, and outcomes for those which produce the highest value for you. It cannot help you pass objective judgment on your values, or discover “better” ones.
I think that’s almost completely wrong. Being human offers guidance in figuring out what goals and values we should have. If the values of the society would be seen as insane by us, a rationalist will still be more likely to win over more of those socieities than average.
Can you explain this suspicion? I’m not saying that “Rationalists always win”: I am saying that they win more often than average.
Say you are in society X, which maximizes potential values [1, 2, 7] though mechanism P and minimzies potential values [4, 9, 13] through mechanism Q.
A rationalist (A) who values [1, 4, 9] will likely not do as well as a random agent (B) that values [1, 2, 7] under X, because the rationalist will only get limited help from P while having to counteract Q, while the other agent (rationalist or not) will recieve full benefit from P and no harm from Q. So it’s trivially true that a rationalist does not always do better than other agents: sometimes the game is set against them.
A rationalist (A) will do better than a non-rationalist (C) with values [1, 4, 9] if having an accurate perception of P allows you to maximize P for 1 or having an accurate perception of Q allows you to minimize Q for [4, 9]. In the world we live in, at least, this usually proves true.
But A will also do better than B in any society that isn’t X, unless B is also a rationalist. They will have a more accurate perception of the reality of the society they are in and thus be better able to maximize the mechanisms that aid their values while minimizing the mechanisms that countermand them.
That’s what I meant by “more likely to win over more of those societies than average.”
I haven’t thought about this carefully, so this may be a howler, but here is what I was thinking:
“Winning” is an optimization problem, so you can conceive of the problem of finding the winning strategy in terms of efficiently minimizing some cost function. Different sets of values—different utility functions—will correspond to different cost functions. Rationalism is a particular algorithm for searching for the minimum. Here I associate “rationalism” with the set of concrete epistemic tools recommended by LW; you could, of course, define “rationalism” so that whichever strategy most conduces to winning in a particular context is the rational one, but then your claim would be tautological.
The No Free Lunch Theorem for search and optimization says that all algorithms that search for the minimum of a cost function perform equally well when you average over all possible cost functions. So if you’re really allowing the possibility of any set of values, then the rationalism algorithm is no more likely to win on average than any other search algorithm.
Again, this is a pretty hasty argument, so I’m sure there are holes.
I suspect you are right if we are talking about epistemic rationality, but not instrumental rationality.
In practice, when attempting to maximize a value, once you know what sort of system you are in, most of your energy has to go into gaming the system: finding the cost of minimizing the costs and looking for exploits. This is more true the more times a game is iterated: if a game literally went on forever, any finite cost becomes justifiable for this sort of gaming of the system: you can spend any bounded amount of bits. (Conversely, if a game is unique, you are less justified in spending your bits on finding solutions: your budget roughly becomes what you can afford to spare.)
If we apply LW techniques of rationalism (as you’ve defined it) what we get is general methods, heuristics, and proofs on ways to find these exploits, a summation of this method being something like “know the rules of the world you are in” because your knowledge of a game directly affects your ability to manipulate its rules and scoring.
In other words, I suspect you are right if what we are talking about is simply finding the best situation for your algorithm: choosing the best restaurant in the available solution space. But when we are in a situation where the rules can be manipulated, used, or applied more effectively I believe this dissolves. You could probably convince me pretty quickly with a more formal argument, however.
It’s my contention that rationality should offer guidance in figuring out what goals you should have. A rationalist society will have goals closer to “defeat death and grasp the stars” than “gain ALL the status”. It’s not just rationalists who should win, it’s rational societies who should win. If you’re in a society that is insane then you may not be able to “win” as a rationalist. In that case your goal should not be “winning” in the social-traditional sense, it should be making society sane.
You’re priveliging your values when you judge which society—the status game players versus the immortal starfarers—is “winning”.
I don’t think that that’s a bad thing. The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”. The point of the post was to warn that not only answering our questions but figuring out which questions we should ask is an issue we have to tackle. We have to figure out what winning should be.
The reason that the immortal starfarers are better is that they’re trying to do that, so if all values aren’t created equally, they’re more likely to find out about it.
Deciding that going somewhere is “winning” comes from your existing utility function. Another person could judge that the civilization with the most rich and complex social hierarchy “wins”.
Rationality can help you search the space of actions, policies, and outcomes for those which produce the highest value for you. It cannot help you pass objective judgment on your values, or discover “better” ones.
I think that’s almost completely wrong. Being human offers guidance in figuring out what goals and values we should have. If the values of the society would be seen as insane by us, a rationalist will still be more likely to win over more of those socieities than average.
I suspect that, if rigorously formulated, this claim will run afoul of something like the No Free Lunch Theorem.
Can you explain this suspicion? I’m not saying that “Rationalists always win”: I am saying that they win more often than average.
Say you are in society X, which maximizes potential values [1, 2, 7] though mechanism P and minimzies potential values [4, 9, 13] through mechanism Q.
A rationalist (A) who values [1, 4, 9] will likely not do as well as a random agent (B) that values [1, 2, 7] under X, because the rationalist will only get limited help from P while having to counteract Q, while the other agent (rationalist or not) will recieve full benefit from P and no harm from Q. So it’s trivially true that a rationalist does not always do better than other agents: sometimes the game is set against them.
A rationalist (A) will do better than a non-rationalist (C) with values [1, 4, 9] if having an accurate perception of P allows you to maximize P for 1 or having an accurate perception of Q allows you to minimize Q for [4, 9]. In the world we live in, at least, this usually proves true.
But A will also do better than B in any society that isn’t X, unless B is also a rationalist. They will have a more accurate perception of the reality of the society they are in and thus be better able to maximize the mechanisms that aid their values while minimizing the mechanisms that countermand them.
That’s what I meant by “more likely to win over more of those societies than average.”
I haven’t thought about this carefully, so this may be a howler, but here is what I was thinking:
“Winning” is an optimization problem, so you can conceive of the problem of finding the winning strategy in terms of efficiently minimizing some cost function. Different sets of values—different utility functions—will correspond to different cost functions. Rationalism is a particular algorithm for searching for the minimum. Here I associate “rationalism” with the set of concrete epistemic tools recommended by LW; you could, of course, define “rationalism” so that whichever strategy most conduces to winning in a particular context is the rational one, but then your claim would be tautological.
The No Free Lunch Theorem for search and optimization says that all algorithms that search for the minimum of a cost function perform equally well when you average over all possible cost functions. So if you’re really allowing the possibility of any set of values, then the rationalism algorithm is no more likely to win on average than any other search algorithm.
Again, this is a pretty hasty argument, so I’m sure there are holes.
I suspect you are right if we are talking about epistemic rationality, but not instrumental rationality.
In practice, when attempting to maximize a value, once you know what sort of system you are in, most of your energy has to go into gaming the system: finding the cost of minimizing the costs and looking for exploits. This is more true the more times a game is iterated: if a game literally went on forever, any finite cost becomes justifiable for this sort of gaming of the system: you can spend any bounded amount of bits. (Conversely, if a game is unique, you are less justified in spending your bits on finding solutions: your budget roughly becomes what you can afford to spare.)
If we apply LW techniques of rationalism (as you’ve defined it) what we get is general methods, heuristics, and proofs on ways to find these exploits, a summation of this method being something like “know the rules of the world you are in” because your knowledge of a game directly affects your ability to manipulate its rules and scoring.
In other words, I suspect you are right if what we are talking about is simply finding the best situation for your algorithm: choosing the best restaurant in the available solution space. But when we are in a situation where the rules can be manipulated, used, or applied more effectively I believe this dissolves. You could probably convince me pretty quickly with a more formal argument, however.