“Winning” refers to making progress towards whatever goals you set for yourself. Rationality can help you achieve your goals but—unless you’res suffering from akrasia—offers little guidance in figuring out what goals you should have.
It’s a rule of epistemic rationality that, all other things being equal, one should adopt simpler theories. Why shouldn’t this also extend to practical rationality, and to the determination of our goals in particular? If our ultimate values involve arbitrary and ad hoc distinctions, then they are irrational. Consider, for instance, Parfit’s example of Future Tuesday Indifference:
A certain hedonist cares greatly about the quality of his future experiences. With one exception, he cares equally about all the parts of his future. The exception is that he has Future-Tuesday-Indifference. Throughout every Tuesday he cares in the normal way about what is happening to him. But he never cares about possible pains or pleasures on a future Tuesday.
I think that any account of practical rationality that does not rule Future Tuesday Indifference an irrational ultimate goal is incomplete. Consider also Eliezer’s argument in Transhumanism as Simplified Humanism.
Of course, this doesn’t apply directly to the point raised by Eneasz, since the distinction between values he is talking about can’t obviously be cashed out in terms of simplicity. But I think there’s good reason to reject the general Humean principle that our ultimate values are not open to rational criticism (except perhaps on grounds of inconsistency), and once that is allowed, positions like the one held by Eneasz are not obviously wrong.
Having a high quality experience at all times other than Tuesdays seems to be a strange goal, but one that a person could coherently optimize for (given a suitable meaning of “high quality experience). The problem with Future Tuesday Indifference is that at different times, the person places different values on the same experience on the same Tuesday.
Yeah, I see that Future Tuesday Indifference is a bad example. Not precisely for the reason you give, though, because that would also entail that any discounting of future goods is irrational, and that doesn’t seem right. But Future Tuesday Indifference would involve the sort of preference switching you see with hyperbolic discounting, which is more obviously irrational and might be confounding intuitions in this case.
So here’s a better example: a person only assigns value to the lives of people who were born within a five-mile radius of the Leaning Tower of Pisa. This is an ultimate value, not an instrumental one. There’s no obvious incoherence involved here. A person could coherently optimize for this goal. But my point is that this does not exhaust our avenues for rational criticism of goals. The fact that this person has an ultimate value that relies on such a highly specific and arbitrary distinction is grounds for criticism, just as it would be if the person adopted a scientific theory which (despite being empirically adequate) postulated such a distinction.
Not precisely for the reason you give, though, because that would also entail that any discounting of future goods is irrational, and that doesn’t seem right.
Discounting of future goods does not involve assigning different values to the same goods at the same time.
So here’s a better example: a person only assigns value to the lives of people who were born within a five-mile radius of the Leaning Tower of Pisa. This is an ultimate value, not an instrumental one. There’s no obvious incoherence involved here. A person could coherently optimize for this goal. But my point is that this does not exhaust our avenues for rational criticism of goals. The fact that this person has an ultimate value that relies on such a highly specific and arbitrary distinction is grounds for criticism, just as it would be if the person adopted a scientific theory which (despite being empirically adequate) postulated such a distinction.
I would not criticize this goal for being “irrational”, though I would oppose it because it conflicts with my own goals. My opposition is not because it is arbitrary, I am perfectly happy with arbitrariness in goal systems that aligns with my own goals.
Discounting of future goods does not involve assigning different values to the same goods at the same time.
The qualifier “at the same time” is ambiguous here.
If you mean that different values are assigned at the same time, so that the agent has conflicting utilities for a goal at a single time, then you’re right that discounting does not involve this. But neither does Future Tuesday Indifference,. so I don’t see the relevance.
If “at the same time” is meant to modify “the same goods”, so that what you’re saying is that discounting does not involve assigning different values to “good-g-at-time-t”, then this is false. Depending on the time at which the valuation is made, discounting entails that different values will be assigned to “good-g-at-time-t”.
If “at the same time” is meant to modify “the same goods”, so that what you’re saying is that discounting does not involve assigning different values to “good-g-at-time-t”, then this is false. Depending on the time at which the valuation is made, discounting entails that different values will be assigned to “good-g-at-time-t”.
Suppose an agend with exponential time discounting assigns goods G at a time T a utility of U0(G)*exp(a*(T0-T)). Then that is the utility the agent at any time assigns those goods at that time. You may be thinking that the agent at time TA assigns a utility to the goods G at the same time T of U0(G)*exp(a*(TA-T)) and thus the agent at different times is assigning different utilities, but these utility functions differ only by the constant (over states of the universe) factor exp(a*(TA-T0)), which being an affine transformation, doesn’t matter. The discounting agent’s equivalency class of utility functions representing its values really is constant over the agent’s subjective time.
It’s my contention that rationality should offer guidance in figuring out what goals you should have. A rationalist society will have goals closer to “defeat death and grasp the stars” than “gain ALL the status”. It’s not just rationalists who should win, it’s rational societies who should win. If you’re in a society that is insane then you may not be able to “win” as a rationalist. In that case your goal should not be “winning” in the social-traditional sense, it should be making society sane.
I don’t think that that’s a bad thing. The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”. The point of the post was to warn that not only answering our questions but figuring out which questions we should ask is an issue we have to tackle. We have to figure out what winning should be.
The reason that the immortal starfarers are better is that they’re trying to do that, so if all values aren’t created equally, they’re more likely to find out about it.
The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”.
Deciding that going somewhere is “winning” comes from your existing utility function.
Another person could judge that the civilization with the most rich and complex social hierarchy “wins”.
Rationality can help you search the space of actions, policies, and outcomes for those which produce the highest value for you. It cannot help you pass objective judgment on your values, or discover “better” ones.
I think that’s almost completely wrong. Being human offers guidance in figuring out what goals and values we should have. If the values of the society would be seen as insane by us, a rationalist will still be more likely to win over more of those socieities than average.
Can you explain this suspicion? I’m not saying that “Rationalists always win”: I am saying that they win more often than average.
Say you are in society X, which maximizes potential values [1, 2, 7] though mechanism P and minimzies potential values [4, 9, 13] through mechanism Q.
A rationalist (A) who values [1, 4, 9] will likely not do as well as a random agent (B) that values [1, 2, 7] under X, because the rationalist will only get limited help from P while having to counteract Q, while the other agent (rationalist or not) will recieve full benefit from P and no harm from Q. So it’s trivially true that a rationalist does not always do better than other agents: sometimes the game is set against them.
A rationalist (A) will do better than a non-rationalist (C) with values [1, 4, 9] if having an accurate perception of P allows you to maximize P for 1 or having an accurate perception of Q allows you to minimize Q for [4, 9]. In the world we live in, at least, this usually proves true.
But A will also do better than B in any society that isn’t X, unless B is also a rationalist. They will have a more accurate perception of the reality of the society they are in and thus be better able to maximize the mechanisms that aid their values while minimizing the mechanisms that countermand them.
That’s what I meant by “more likely to win over more of those societies than average.”
I haven’t thought about this carefully, so this may be a howler, but here is what I was thinking:
“Winning” is an optimization problem, so you can conceive of the problem of finding the winning strategy in terms of efficiently minimizing some cost function. Different sets of values—different utility functions—will correspond to different cost functions. Rationalism is a particular algorithm for searching for the minimum. Here I associate “rationalism” with the set of concrete epistemic tools recommended by LW; you could, of course, define “rationalism” so that whichever strategy most conduces to winning in a particular context is the rational one, but then your claim would be tautological.
The No Free Lunch Theorem for search and optimization says that all algorithms that search for the minimum of a cost function perform equally well when you average over all possible cost functions. So if you’re really allowing the possibility of any set of values, then the rationalism algorithm is no more likely to win on average than any other search algorithm.
Again, this is a pretty hasty argument, so I’m sure there are holes.
I suspect you are right if we are talking about epistemic rationality, but not instrumental rationality.
In practice, when attempting to maximize a value, once you know what sort of system you are in, most of your energy has to go into gaming the system: finding the cost of minimizing the costs and looking for exploits. This is more true the more times a game is iterated: if a game literally went on forever, any finite cost becomes justifiable for this sort of gaming of the system: you can spend any bounded amount of bits. (Conversely, if a game is unique, you are less justified in spending your bits on finding solutions: your budget roughly becomes what you can afford to spare.)
If we apply LW techniques of rationalism (as you’ve defined it) what we get is general methods, heuristics, and proofs on ways to find these exploits, a summation of this method being something like “know the rules of the world you are in” because your knowledge of a game directly affects your ability to manipulate its rules and scoring.
In other words, I suspect you are right if what we are talking about is simply finding the best situation for your algorithm: choosing the best restaurant in the available solution space. But when we are in a situation where the rules can be manipulated, used, or applied more effectively I believe this dissolves. You could probably convince me pretty quickly with a more formal argument, however.
“Winning” refers to making progress towards whatever goals you set for yourself. Rationality can help you achieve your goals but—unless you’res suffering from akrasia—offers little guidance in figuring out what goals you should have.
It’s a rule of epistemic rationality that, all other things being equal, one should adopt simpler theories. Why shouldn’t this also extend to practical rationality, and to the determination of our goals in particular? If our ultimate values involve arbitrary and ad hoc distinctions, then they are irrational. Consider, for instance, Parfit’s example of Future Tuesday Indifference:
I think that any account of practical rationality that does not rule Future Tuesday Indifference an irrational ultimate goal is incomplete. Consider also Eliezer’s argument in Transhumanism as Simplified Humanism.
Of course, this doesn’t apply directly to the point raised by Eneasz, since the distinction between values he is talking about can’t obviously be cashed out in terms of simplicity. But I think there’s good reason to reject the general Humean principle that our ultimate values are not open to rational criticism (except perhaps on grounds of inconsistency), and once that is allowed, positions like the one held by Eneasz are not obviously wrong.
Having a high quality experience at all times other than Tuesdays seems to be a strange goal, but one that a person could coherently optimize for (given a suitable meaning of “high quality experience). The problem with Future Tuesday Indifference is that at different times, the person places different values on the same experience on the same Tuesday.
Yeah, I see that Future Tuesday Indifference is a bad example. Not precisely for the reason you give, though, because that would also entail that any discounting of future goods is irrational, and that doesn’t seem right. But Future Tuesday Indifference would involve the sort of preference switching you see with hyperbolic discounting, which is more obviously irrational and might be confounding intuitions in this case.
So here’s a better example: a person only assigns value to the lives of people who were born within a five-mile radius of the Leaning Tower of Pisa. This is an ultimate value, not an instrumental one. There’s no obvious incoherence involved here. A person could coherently optimize for this goal. But my point is that this does not exhaust our avenues for rational criticism of goals. The fact that this person has an ultimate value that relies on such a highly specific and arbitrary distinction is grounds for criticism, just as it would be if the person adopted a scientific theory which (despite being empirically adequate) postulated such a distinction.
Discounting of future goods does not involve assigning different values to the same goods at the same time.
I would not criticize this goal for being “irrational”, though I would oppose it because it conflicts with my own goals. My opposition is not because it is arbitrary, I am perfectly happy with arbitrariness in goal systems that aligns with my own goals.
The qualifier “at the same time” is ambiguous here.
If you mean that different values are assigned at the same time, so that the agent has conflicting utilities for a goal at a single time, then you’re right that discounting does not involve this. But neither does Future Tuesday Indifference,. so I don’t see the relevance.
If “at the same time” is meant to modify “the same goods”, so that what you’re saying is that discounting does not involve assigning different values to “good-g-at-time-t”, then this is false. Depending on the time at which the valuation is made, discounting entails that different values will be assigned to “good-g-at-time-t”.
Suppose an agend with exponential time discounting assigns goods G at a time T a utility of U0(G)*exp(a*(T0-T)). Then that is the utility the agent at any time assigns those goods at that time. You may be thinking that the agent at time TA assigns a utility to the goods G at the same time T of U0(G)*exp(a*(TA-T)) and thus the agent at different times is assigning different utilities, but these utility functions differ only by the constant (over states of the universe) factor exp(a*(TA-T0)), which being an affine transformation, doesn’t matter. The discounting agent’s equivalency class of utility functions representing its values really is constant over the agent’s subjective time.
Ah, I see. You’re right. Comment retracted.
It’s my contention that rationality should offer guidance in figuring out what goals you should have. A rationalist society will have goals closer to “defeat death and grasp the stars” than “gain ALL the status”. It’s not just rationalists who should win, it’s rational societies who should win. If you’re in a society that is insane then you may not be able to “win” as a rationalist. In that case your goal should not be “winning” in the social-traditional sense, it should be making society sane.
You’re priveliging your values when you judge which society—the status game players versus the immortal starfarers—is “winning”.
I don’t think that that’s a bad thing. The immortal starfarers necessarily go somewhere; the status game players don’t necessarily go anywhere. Hence “winning”. The point of the post was to warn that not only answering our questions but figuring out which questions we should ask is an issue we have to tackle. We have to figure out what winning should be.
The reason that the immortal starfarers are better is that they’re trying to do that, so if all values aren’t created equally, they’re more likely to find out about it.
Deciding that going somewhere is “winning” comes from your existing utility function. Another person could judge that the civilization with the most rich and complex social hierarchy “wins”.
Rationality can help you search the space of actions, policies, and outcomes for those which produce the highest value for you. It cannot help you pass objective judgment on your values, or discover “better” ones.
I think that’s almost completely wrong. Being human offers guidance in figuring out what goals and values we should have. If the values of the society would be seen as insane by us, a rationalist will still be more likely to win over more of those socieities than average.
I suspect that, if rigorously formulated, this claim will run afoul of something like the No Free Lunch Theorem.
Can you explain this suspicion? I’m not saying that “Rationalists always win”: I am saying that they win more often than average.
Say you are in society X, which maximizes potential values [1, 2, 7] though mechanism P and minimzies potential values [4, 9, 13] through mechanism Q.
A rationalist (A) who values [1, 4, 9] will likely not do as well as a random agent (B) that values [1, 2, 7] under X, because the rationalist will only get limited help from P while having to counteract Q, while the other agent (rationalist or not) will recieve full benefit from P and no harm from Q. So it’s trivially true that a rationalist does not always do better than other agents: sometimes the game is set against them.
A rationalist (A) will do better than a non-rationalist (C) with values [1, 4, 9] if having an accurate perception of P allows you to maximize P for 1 or having an accurate perception of Q allows you to minimize Q for [4, 9]. In the world we live in, at least, this usually proves true.
But A will also do better than B in any society that isn’t X, unless B is also a rationalist. They will have a more accurate perception of the reality of the society they are in and thus be better able to maximize the mechanisms that aid their values while minimizing the mechanisms that countermand them.
That’s what I meant by “more likely to win over more of those societies than average.”
I haven’t thought about this carefully, so this may be a howler, but here is what I was thinking:
“Winning” is an optimization problem, so you can conceive of the problem of finding the winning strategy in terms of efficiently minimizing some cost function. Different sets of values—different utility functions—will correspond to different cost functions. Rationalism is a particular algorithm for searching for the minimum. Here I associate “rationalism” with the set of concrete epistemic tools recommended by LW; you could, of course, define “rationalism” so that whichever strategy most conduces to winning in a particular context is the rational one, but then your claim would be tautological.
The No Free Lunch Theorem for search and optimization says that all algorithms that search for the minimum of a cost function perform equally well when you average over all possible cost functions. So if you’re really allowing the possibility of any set of values, then the rationalism algorithm is no more likely to win on average than any other search algorithm.
Again, this is a pretty hasty argument, so I’m sure there are holes.
I suspect you are right if we are talking about epistemic rationality, but not instrumental rationality.
In practice, when attempting to maximize a value, once you know what sort of system you are in, most of your energy has to go into gaming the system: finding the cost of minimizing the costs and looking for exploits. This is more true the more times a game is iterated: if a game literally went on forever, any finite cost becomes justifiable for this sort of gaming of the system: you can spend any bounded amount of bits. (Conversely, if a game is unique, you are less justified in spending your bits on finding solutions: your budget roughly becomes what you can afford to spare.)
If we apply LW techniques of rationalism (as you’ve defined it) what we get is general methods, heuristics, and proofs on ways to find these exploits, a summation of this method being something like “know the rules of the world you are in” because your knowledge of a game directly affects your ability to manipulate its rules and scoring.
In other words, I suspect you are right if what we are talking about is simply finding the best situation for your algorithm: choosing the best restaurant in the available solution space. But when we are in a situation where the rules can be manipulated, used, or applied more effectively I believe this dissolves. You could probably convince me pretty quickly with a more formal argument, however.