The concept “achieving your values” doesn’t deserve the term “instrumental rationality”. If it does, then, as you point out, works about instrumental rationality are merely works about how to do stuff. You’re giving a fancy new name to an old concept.
ETA: Not that that’s always exactly what we mean when we say “instrumental rationality”, of course …
How about the given definition of “epistemic rationality”? This is also really general: it’s how to know stuff. Granted, that’s precisely what being less wrong means, but we’re not interested in general education. Granted again, the top-rated post of all time, “Generalizing From One Example”, is definitely epistemic rationality but not obviously any other type of rationality.
So, here I propose some other definitions of “rationality”:
Aumann rationality: a person is Aumann rational if they are rational (don’t interpret this circularly!), they believe other people are Aumann rational, and other people believe they are Aumann rational. Perfect Aumann rationality causes people to never disagree with each other, but it’s a spectrum. Eliezer Yudkowsky is relatively Aumann rational; people on Less Wrong are expected to be quite Aumann rational with each other; people in political debates have very little Aumann rationality.
Rational neutrality: though people who are rational-neutral discard evidence regarding statements, as any intelligent being must, their decision whether to discard a piece of evidence or not is not based on the direction/magnitude of it—if they ignore an observation, they do so without first seeing what it is.
Krasia: quite unrelated to any other type of rationality, people with high krasia are good at going from believing that an action would result in high expected utility to actually taking that action.
people who are rational-neutral discard evidence regarding statements, as any intelligent being must
I feel like I should be able to find this out on my own, but I’ve had no success so far. Does “evidence regarding statements” refer to statements that are evidence-regarding, or evidence that regards statements? Either way I can’t figure out an obvious reason to reject such things. Is it the idea that evidence shouldn’t be discussed on any aspect beyond validity? I feel I’m missing something, many thanks to anyone who can throw me a link or other resource.
(If this post is too long, read only the last paragraph.)
Evidence that regards statements. I guess the “regarding statements” bit was redundant. Anyway, let me try to give some examples.
First, let me postulate a guy named Delta. Delta is an extremely rational robot who, given the evidence, always comes up with the best possible conclusion.
Andy the Apathetic is presented with a court case. Before he ever looks at the case, he decides that the probability the defendant is guilty is 50%. In fact, he never looks at the case; he tosses it aside and gives that 50% as his final judgement. Andy is rational-neutral, as he discarded evidence regardless of its direction; his probability is useless, but if I told Delta how Andy works and Andy’s final judgement, Delta would agree with it.
Barney the Biased is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he decides to discard everything suggesting that the defendant is innocent; he concludes that the defendant has a 99.99% chance of being guilty and gives that as his final judgement. Barney is not rational-neutral, as he discarded evidence with regard to its direction; his probability is almost useless (but not as useless as Andy’s), and if I told Delta how Barney works and Barney’s final judgement, Delta might give a probability of only 45%.
Finally, Charlie the Careful is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he takes absolutely everything into account, running the numbers and keeping Bayes’ law between his eyes at all times; eventually, after running a complete analysis, he decides that the probability that the defendant is guilty is 23.14159265%. Charlie is rational-neutral, as he discarded evidence regardless of its direction (in fact, he discarded no evidence); if I told Delta how Charliie works and Charlie’s final judgement, Delta would agree with it.
So, here’s another definition of rational neutrality I came up with by writing this: you are rational-neutral if, given only your source code, it’s impossible to come up with a function that takes one of your probability estimates and returns a better probability estimate.
It might be useful to revise this concept to account for computational resources (see AI work on ‘limited rationality’, e.g. Russell and Wefald’s “Do the Right Thing” book).
Upon thinking about that second definition of rational neutrality, I find myself thinking that that can’t be right. It’s identical to calibration. And even a rational-neutral agent that’s been “repaired” by applying the best possible probability estimate adjustment function will still return the same ordinal probabilities: Barney the Biased, even after adjustment, will return higher probabilities for statements he is biased toward than statements he is biased against.
I would have said this:
So, here’s another definition of rational neutrality I came up with by writing this: you are rational-neutral if, given only your source code and your probability estimates, it’s impossible for someone to come up with better probability estimates.
...but that definition doesn’t rule out the possibility that an agent would look at your probability estimates, figure out what the problem is, and come up with a better solution on its own. In the extreme case, no agent would be considered rational-neutral unless it had a full knowledge of all mathematical results. That’s not what I want; therefore, I stick by my original definition.
The concept “achieving your values” doesn’t deserve the term “instrumental rationality”. If it does, then, as you point out, works about instrumental rationality are merely works about how to do stuff. You’re giving a fancy new name to an old concept.
ETA: Not that that’s always exactly what we mean when we say “instrumental rationality”, of course …
How about the given definition of “epistemic rationality”? This is also really general: it’s how to know stuff. Granted, that’s precisely what being less wrong means, but we’re not interested in general education. Granted again, the top-rated post of all time, “Generalizing From One Example”, is definitely epistemic rationality but not obviously any other type of rationality.
So, here I propose some other definitions of “rationality”:
Aumann rationality: a person is Aumann rational if they are rational (don’t interpret this circularly!), they believe other people are Aumann rational, and other people believe they are Aumann rational. Perfect Aumann rationality causes people to never disagree with each other, but it’s a spectrum. Eliezer Yudkowsky is relatively Aumann rational; people on Less Wrong are expected to be quite Aumann rational with each other; people in political debates have very little Aumann rationality.
Rational neutrality: though people who are rational-neutral discard evidence regarding statements, as any intelligent being must, their decision whether to discard a piece of evidence or not is not based on the direction/magnitude of it—if they ignore an observation, they do so without first seeing what it is.
Krasia: quite unrelated to any other type of rationality, people with high krasia are good at going from believing that an action would result in high expected utility to actually taking that action.
I expect that anyone who expected this has already been quite disappointed. ;-)
I feel like I should be able to find this out on my own, but I’ve had no success so far. Does “evidence regarding statements” refer to statements that are evidence-regarding, or evidence that regards statements? Either way I can’t figure out an obvious reason to reject such things. Is it the idea that evidence shouldn’t be discussed on any aspect beyond validity? I feel I’m missing something, many thanks to anyone who can throw me a link or other resource.
(If this post is too long, read only the last paragraph.)
Evidence that regards statements. I guess the “regarding statements” bit was redundant. Anyway, let me try to give some examples.
First, let me postulate a guy named Delta. Delta is an extremely rational robot who, given the evidence, always comes up with the best possible conclusion.
Andy the Apathetic is presented with a court case. Before he ever looks at the case, he decides that the probability the defendant is guilty is 50%. In fact, he never looks at the case; he tosses it aside and gives that 50% as his final judgement. Andy is rational-neutral, as he discarded evidence regardless of its direction; his probability is useless, but if I told Delta how Andy works and Andy’s final judgement, Delta would agree with it.
Barney the Biased is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he decides to discard everything suggesting that the defendant is innocent; he concludes that the defendant has a 99.99% chance of being guilty and gives that as his final judgement. Barney is not rational-neutral, as he discarded evidence with regard to its direction; his probability is almost useless (but not as useless as Andy’s), and if I told Delta how Barney works and Barney’s final judgement, Delta might give a probability of only 45%.
Finally, Charlie the Careful is presented with the same court case. Before he ever looks at the case, he decides that the probability that the defendant is guilty is 50%. Looking through the evidence, he takes absolutely everything into account, running the numbers and keeping Bayes’ law between his eyes at all times; eventually, after running a complete analysis, he decides that the probability that the defendant is guilty is 23.14159265%. Charlie is rational-neutral, as he discarded evidence regardless of its direction (in fact, he discarded no evidence); if I told Delta how Charliie works and Charlie’s final judgement, Delta would agree with it.
So, here’s another definition of rational neutrality I came up with by writing this: you are rational-neutral if, given only your source code, it’s impossible to come up with a function that takes one of your probability estimates and returns a better probability estimate.
It might be useful to revise this concept to account for computational resources (see AI work on ‘limited rationality’, e.g. Russell and Wefald’s “Do the Right Thing” book).
I’ll try my best to get my hands on a copy of that book.
Upon thinking about that second definition of rational neutrality, I find myself thinking that that can’t be right. It’s identical to calibration. And even a rational-neutral agent that’s been “repaired” by applying the best possible probability estimate adjustment function will still return the same ordinal probabilities: Barney the Biased, even after adjustment, will return higher probabilities for statements he is biased toward than statements he is biased against.
I would have said this:
...but that definition doesn’t rule out the possibility that an agent would look at your probability estimates, figure out what the problem is, and come up with a better solution on its own. In the extreme case, no agent would be considered rational-neutral unless it had a full knowledge of all mathematical results. That’s not what I want; therefore, I stick by my original definition.
It took two read-throughs to get this, but I’m fairly sure that’s the concept and not your handling.Thanks for the explanation!
I’m going to start using “krasia”. I hadn’t encountered it before but apparently it’s had some currency in epistemology.