I have rewritten the first paragraph based on your feedback.
Mea culpa, but the rewrite doesn’t look great to me. Before, you first paragraph had some zing. People like me could and did find fault with it, sure, but at least it was energetic. And now the first two sentences are followed by a lot of hemming and hawing which sounds defensive and is entirely uninspiring.
“Ensuring that resource usage and behaviour/thought cordination is directed towards the fulfillment of the agents goals” was already being taught by senior slave-drivers to junior slave-drivers when the pyramids were being built. In trying to avoid rationality be just prediction, you made it be just effectiveness.
I don’t have a good suggestion for you, in fact I’m not sure that the so-called epistemic rationality (aka science) and instrumental rationality (aka pragmatism and keeping your eye on the ball) can be usefully joined together into a single concept. But since you are writing a compendium, you probably should come up with a reasonable definition for rationality, since it is, y’know, a core concept.
In terms of the defintiion, it is in the title. As to what is is. I am basically trying to convey the idea that rationality is optimal thinking. Although, I suppose I am also happy with how its defined in this book. If you think the below definitions are better, let me know.
Rationality: the property of a system which does the “right thing” given what it knows.
It is an gent that acts to maximize its expected performance measure. That is, it does the “right thing”.
For each possible percept sequence, an ideal rational agent should do whatever action is expected to maximize its performance measure, on the basis of the evidence provided by the percept sequence and whatever built-in knowledge the agent has
An omniscient agent knows the actual outcome of its actions, and can act accordingly, but omniscience is impossible in reality. It is important to distinguish rationality and omniscience. This is because we should not blame an agent for failing to take into account something it could not perceive, or for failing to take an action that it is incapable of taking.
That’s not really a definition: you just shifted the entire burden onto the word “optimally”. A basic use of a definition is to see if something fits it—if we defined a class A, is object z a member of that class? So let’s say I’m considering some action. Is it rational? Well, it is if it’s optimal. Err.. and what does that mean? To answer I need to define optimality and that is not trivial. And if you say that optimality is maximizing (expected) utility, we’re back to your original definition which I poked at and you abandoned.
the property of a system which does the “right thing”
That’s exactly the same thing—replacing one word with another (or two) without clarifying anything.
whatever action is expected to maximize its performance measure
That’s maximizing expected utility, again.
we should not blame an agent
Yes, of course, but that doesn’t help you with a definition.
Before we get too deeply into problems with whatever definition I might use, I want to make sure that you agree that what I am trying to say is right. Once that is confirmed, then I can think more about how to say it well.
This is basically what I am trying to say in the post. ‘Rationality’ is ‘optimal reasoning’ which we know of as normative rationality, i.e. the laws of thought and action or what the perfect agent would do. A caveat is that when we talk about rationality in regards to limited agents we are really talking about bounded rationality. Hence, rationality in this case is really ‘bounded optimal reasoning’. So for limited agents, rationality is about the reasoning that best approximates the results of the normative rationality. Also, for limited agents we consider those types of thinking that lead to more optimal reasoning in the future to be rational as well. This is the basis for epistemic and instrumental rationality and is why curiosity is rational. Curiosity often leads to better maps. Better maps often leads to better decisions and, therefore, we consider curiosity to be rational. In terms of costs, the goal of rationality in limited agents is to best approximate the normative rationality. This requires the highest return on investment on the resources that you make use of. Basically the resource usage to expected utiltity ratio should be high since, we only have a limited amount of resources that we can make us of. If there is an alternative way to spend the same resources which has a higher usage to expected utility ratio, then you are not as close as possible.to the normative rationality.
Before we get too deeply into problems with whatever definition I might use, I want to make sure that you agree that what I am trying to say is right.
A good idea. I’m not nipicking about wording, at issue is actually meaning.
First, as I mentioned before, I am not sure how to combine epistemic and instrumental rationality together into one useful concept. I am not saying it’s impossible, just that nothing comes to my mind.
One issue, for example, is that they belong to different categories: one is about knowing and the other is about doing. Yes, you can trivially stick them together by saying that epistemic rationality is just instrumental rationality with the goal of constructing a good map, but I don’t know what you gain by that. Constructing a good map is, basically, the scientific method and it is not a decision theory.
Second, I have problems with the “what the perfect agent would do”. An immediate issue is that the answer to that is “You don’t know and you will never know” for any noticeably complex problem[1], especially one that concerns the messy real world and not, for example, the neat and well-defined world of mathematics. That’s an issue because you set it up as a standard and as a limit to which “bounded optimal reasoning” should converge. But if you don’t know what it is, you don’t know what you should converge to and don’t have a good method to adjudicate competing claims about what is rational.
There are also questions about defining rationality as optimality. Optimality typically involves maximizing some measure, but in a lot of situations what matters is not how to reach the maximum, but rather what is it that you optimize. Is it “rational” to arrive at an optimum for the wrong thing? How do you know what to optimize for? Handwaving about utility remains handwaving because the only utility functions I have seen which actually produce a specific numerical estimate are economic utility functions and they solely care about money.
Moreover, you rarely have the luxury of optimizing for one thing. Typically you have multiple conflicting goals with a mix of different costs to all actions, so deciding how are you going to balance goals and summarize costs is very important and I have no idea what is the “rational” way to go about it.
All in all, I am not satisfied by the “perfect agent” or “optimize utility” definitions of rationality. The perfect agent approach is essentially WWJD—What Would Jesus Do—only without the religious baggage, and optimizing utility doesn’t tell me how to actually, in practice do that.
Notice that keeping epistemic and instrumental rationality separate works much better. The criterion for epistemic rationality is the match between the map and the reality—this is specific and observable. The criterion for instrumental rationality is whether you reach your goals at a reasonable cost. This is more complicated because of uncertainty of the future: good decisions don’t always lead to good outcomes and good outcomes do not necessarily follow from good decisions. But even here there are things we can look at and handles we can grab and manipulate. But “emulate perfection” or “maximize utility”—I have no idea how to even start doing that.
[1] This is commonly held as entirely obvious in LW—only in the context of AI boxing :-)
First, as I mentioned before, I am not sure how to combine epistemic and instrumental rationality together into one useful concept. I am not saying it’s impossible, just that nothing comes to my mind. One issue, for example, is that they belong to different categories: one is about knowing and the other is about doing.
If we forget about epistemic and instrumental rationality for a moment and think about what reasoning is aimed at achieving. That is, why we care about it. Then, I think we can get closer to understanding how epistemic and instrumental rationality might work together to become part of something larger. They are ,of course, still different techniques.
Do you think that the below areas describe what it means to reason well?
It is about actually achieving desired outcomes, so if you want x to occur it is about making sure that you initiate a series of thoughts and actions that lead to x occurring instead of y or z or a plethora of other possibilities.
It is about achieving outcomes at a reasonable cost. If you can achieve your desired outcomes with cheaper costs, then you will be able to achieve more outcomes.
It is about choosing the best outcomes to achieve. We all have limited resources and to achieve any outcomes we need to use resources. This means that we can only chose a limited amount of outcomes to pursue. Making this choice wisely is what this area is about.
It is about valuing the outcomes appropriately. This would be about making your values coherent and correctly valuing the things that matter so that they get priority.
Epistemic/Instrumental rationality is really about certain types of skills that allow us to do well in the above areas (I think I covered all of them). I think that there might also be more types of skills. I have a general idea about what another one might be, I am not sure how many others there might be. Although, I want the compendium to cover existing and established ideas only. That is why I am referring to epistemic and instrumental rationality and not anything else.
All in all, I am not satisfied by the “perfect agent” or “optimize utility” definitions of rationality. The perfect agent approach is essentially WWJD—What Would Jesus Do—only without the religious baggage, and optimizing utility doesn’t tell me how to actually, in practice do that.
What do you think about defining rationality as a property that is attributed by agents to certain thoughts and behaviours? This would mean that it is not only bounded by the agent’s abilities and the information it has, but also by its understanding of what it means to be rational. Essentially, it would mean that ‘rationality’ is subjective. To avoid the fallacy of the grey there needs to be some objective way to judge different agents understanding of what it means to be rational. This objective way is basically our best overall guess at what perfectly optimal reasoning or optimal reasoning for humans would be. For humans, this way is the scientific method with the current body of work pointing to logic, probability and decision theory as having the closest answers on what it means to reason optimally, i.e. be rational. These answers aren’t necessarily correct due to negative pragmatism etc. They are just our current, best and most informed guesses.
There are also questions about defining rationality as optimality. Optimality typically involves maximizing some measure, but in a lot of situations what matters is not how to reach the maximum, but rather what is it that you optimize. Is it “rational” to arrive at an optimum for the wrong thing?
I always thought that this was a part of what it means to be instrumentally rational. Basically to have optimal goals as well. This is my problem with instrumental rationality as it’s talked about on less wrong is about achieving what you value at a reasonable cost or is it about making your values coherent and in line with what you innately value or is it a combination of the two. I have always felt that instrumental rationality is a bit too overreaching and encompassing. Do you think I should split it into two types of instrumental rationality? One for costs and one for value alignment or am I not interpreting it correctly.
Notice that keeping epistemic and instrumental rationality separate works much better.
I will have other posts where I go into detail on each of these separately. I think that they are separate skills or areas of expertise, but I also think that there should be a base reason for why we should care about them.
and think about what reasoning is aimed at achieving
So do you want to define “rationality” as a kind of reasoning? Reasoning is an opaque mental process and, for example, does not include acting which is a large part of instrumental rationality. Procrastination is a classic LW sin, but it’s not a reasoning problem. And what would be non-rational reasoning besides straightforward logical errors? The great majority of thinking people do throughout the day is not formalizable into a neat system of propositions and conclusions.
It is about actually achieving desired outcomes …at a reasonable cost
Yes, that’s the definition of instrumental rationality.
It is about choosing the best outcomes to achieve. … It is about valuing the outcomes appropriately.
Hold on, that’s new. Are you claiming that (proper) values are a part of rationality and that rationality will tell you what your values should be? I think I am going to loudly object to that. Maybe you can provide an example to show what you mean?
is really about certain types of skills that allow us to do well
Hm, that’s an interesting approach. Then you’d consider rationality a kind of skill—a skill like writing essays or programming? This is probably worth exploring further.
Essentially, it would mean that ‘rationality’ is subjective.
Not sure I want to go that way. You wouldn’t have many counterarguments to a bloke which declares himself perfectly rational as he goes to pray to Jesus so that he wins the lottery. And once you introduce an “objective way to judge” there doesn’t seem to be any point to the subjectivity any more.
I always thought that this was a part of what it means to be instrumentally rational. Basically to have optimal goals as well.
See above—goals are a direct function of values and I have very strong doubts that rationality can tell you what your values should be.
it about making your values coherent
Humans don’t have coherent values. In fact, I don’t think you can make system of values complex enough to deal with real life fully coherent (people who come close to that are usually called “crazy fanatics”). Instead, what people do is trade off different values against each other and come up with an end-result balance where they are willing to sacrifice some A, B, and C but gain X, Y, and Z. As a crude approximation you can think about it as summing different vectors and acting according to where the summed vector points.
I think that to what degree rationality applies here is a hard question. On the one hand, there is no basis for rationality to say “you need to value this and not value that”. On the other hand, values and their weights are not stable across time, and part of rationality is juggling short-term and long-term desires and consequences—usually pointing out that it’s not smart to pay with a lot of long-term pain for a jolt of short-term pleasure. That’s where this whole bit about “imagine yourself as a very smart, calm, capable human being—what would she choose?” comes in.
So, yes, it’s complicated. I have issues with listening to “It’s not rational to value/desire this”, but I have much less issues with “The price for this action that you want to do is really high, are you quite sure you want to pay it, that doesn’t look rational”. I am not sure where the proper boundary is.
So do you want to define “rationality” as a kind of reasoning? Reasoning is an opaque mental process and, for example, does not include acting which is a large part of instrumental rationality.
When I use the word reasoning, I really mean both the system 1 and 2 cognitive processes. By rational I basically mean reasoning (system 1 and 2) done well. Where done well, is defined based on your most trusted source. For us this is science, so logic, probability, decision theory etc. for system 2.
Hold on, that’s new. Are you claiming that (proper) values are a part of rationality and that rationality will tell you what your values should be? I think I am going to loudly object to that. Maybe you can provide an example to show what you mean?
I don’t know what “proper” would mean. I am talking about coherence which means that its “properness”, I suppose, depends on its context, i.e. the other pre-existing values. I will give you some examples. I will assume that you already know the difference between wanting and liking.
Excessive Wanting—an example is drug addiction: “Only ‘wanting’ systems sensitize, and so ‘wanting’ can increase and become quite intense due to sensitization, regardless of whether a drug still remains ‘liked’ after many repeated uses”.
Not liking things that you should or could—examples are bad experiences that cause aversion conditioning to something that you used to or could like. My general view is that if you don’t like something and you could then this is a limitation.
Not wanting things you like—ugh fields are an example of this.
Conflicting wants—this is often inevitable like you say value is complex. But, I think it is important to look at what the fundamental human values or needs are and try to align with those. If you don’t, then in general there is a going to be a greater amount of conflict.
I would need to write a full post on the details, but that is just a general idea of what I mean. You also consider the values of others that you are interconnected with and care about.
Hm, that’s an interesting approach. Then you’d consider rationality a kind of skill—a skill like writing essays or programming? This is probably worth exploring further.
I don’t see how you can view it as anything but a skill. This is because epistemic rationality, for example, is only valuable instrumentally. It helps makes more rational decisions, but the truer beliefs it causes need to be applied to actually be useful and improve your rationality. If you spend lots of effort creating true beliefs and then compartmentalize that knowledge and don’t apply it, you have effectively gained nothing in terms of rationality. That’s my view anyway. I don’t know how many people would agree. An example is Aumann, he knows a lot about rationality, but I don’t think he is rational because it looks to me like he believes in non overlapping magisteria.
So, yes, it’s complicated. I have issues with listening to “It’s not rational to value/desire this”, but I have much less issues with “The price for this action that you want to do is really high, are you quite sure you want to pay it, that doesn’t look rational”. I am not sure where the proper boundary is.
I agree with you on this and your other points on how value is complex. I think that to say that: “it is rational to value/desire this” there needs to a ‘because’ after that statement. No value/desire is rational or irrational in and of itself. It is only irrational or irrational in a context. That is, because of its relation to other values or the costs to fulfil it etc.
Right now, I am thinking that I need to make the base concepts of rationality more solid before I can move into what rationality is for this compendium.
This is my first attempt at defining things. My goal is to define things in a programatic kind of way. This means that the concepts should follow: single responsibility, loose coupling, yagni etc. Let me know what you think.
The goal of the definitions is just to highlight the right areas in concept space. They are drafts and will obviously need more detail. I would also need to submit them as posts and see if others agree.
I am thinking that there should be two basic areas: system 1 and system 2 rationality. Where rationality, in its most basic form, means done well (this will need to be expanded upon). The goal of the two areas is to define what it is we are referring to when we say that something is rational or irrational. There are two areas so that we can distinguish rationality/irrationality in formal reasoning vs. your intuitions or what you actually do vs. what you think you should do.
There are also skills or general topics which describe groups of techniques and methods that can be used to improve your rationality in one or both of the two areas of it. Using these skills means that you apply them using volitional effort. It is noted, however that if you use these skills often enough they are likely to become embedded in your system 1 processes.
There may be more skills, but I think the main ones are below:
Epistemic rationality—true beliefs and all that
Instrumental rationality - (restricted to reasonable costs)
Value coherence rationality—I gave some examples, but it basically means noticing when your values and desires are out of alignment or could become so if you did some action.
Distributive rationality—this is basically what you are talking about in the above quote. Once you have a semi-sufficient valuation system in place how can you actually distribute resources so that you achieve what you value.
Perspectival rationality—no matter how great you are at being rational you are limited by the ideas that you can come up with. You are limited by your paradigms and perspectives. Perspectival rationality is about knowing when to look at a situation from multiple perspectives and having the ability to model the territory or map accurately from another perspective. By modelling the map from another perspective, it is meant that you are thinking about what the maps of someone else or yourself in a future or past tense would be like for a given situation. By modelling the territory, it is meant that you are thinking about what the territory will be like if some situation occurs. An important part of perspectival rationality is being able to coalesce the information from multiple perspectives into a coherent whole. The aim of prrapectival rationality is greater novelty in your ideas, broader utilities in solutions and more pragmatic results. It also includes understanding the necessarily flawed and limited nature of your perspective. You need to constantly be seeking feedback and other perspectives. It would relate to complexity theory, agile software development, systems dynamics, boydian thinking and mental models/schemas/scripts (whatever you want to call it). I plan to write some posts around this idea.
Communicative rationality—how can you communicate well. I will need to look into this one, but I think it’s important.
Applied rationality—This relates to when you already know what the best thing to do is and is about how you can get yourself to actually do it. Examples of this are training will power or courage (doing something you don’t want to, but believe you should), dealing with ugh fields.
rational I basically mean reasoning (system 1 and 2) done well. Where done well, is defined based on your most trusted source.
I am not sure I understand—is “most trusted source” subjective? What if Jesus is my most trusted source? And He is for a great deal of people.
I am talking about coherence which means that its “properness”, I suppose, depends on its context, i.e. the other pre-existing values.
Do you think it could be reformulated in the framework where values form tree-like networks with some values being “deep” or “primary” and other values being “shallow” or “derived” or “secondary”? Then you might be able to argue that a conflict between a deep and a shallow value should be resolved by the declaring the shallow value not rational.
I don’t see how you can view it as anything but a skill
I meant this more specifically in the looking for a definition context.
One very common way of making a definition is to point to a well-known class, say, Bet and then define a sub-class beta by listing a set of features {X} which allow you to decide whether a particular object b from the super-class Bet belongs to the sub-class beta or not. Such definitions are sometimes called is-a-kind-of definitions: beta is a kind of Bet.
So if we were to try to give an is-a-kind-of defintion of rationality, what is the super-class? Is it reasoning? Is it skills? Something else?
No value/desire is rational or irrational in and of itself. It is only irrational or irrational in a context. That is, because of its relation to other values or the costs to fulfil it etc.
So how to avoid being caught in a loop: values depend on values which depend on values that depend on values..?
This means that the concepts should follow: single responsibility, loose coupling, yagni etc.
Not sure about yagni, since it is not the case that you can always go back to a core definition and easily update it for your new needs. If there’s already a structure built on top of that core definition, changing it might prove to be quite troublesome. Loose coupling and such—sure, if you can pull it off :-) Software architecture is… much less constrained by reality :-)
two basic areas: system 1 and system 2 rationality
What do you mean by system 2 rationality? Intuitions that work particularly well? Successful hunches?
I think the main ones are below
That’s a very wide reach. Are you sure you’re not using “rationality” just as a synonym for “doing something really well”?
That’s a very wide reach. Are you sure you’re not using “rationality” just as a synonym for “doing something really well”?
I mean do well in the areas I talked about before. In summary, I basically mean do well at coming up with solutions to problems or choosing/being able to go through with the best solution, out of all of the solutions you have come up with, to a problem.
I will try to define it again.
First off, there is comprehensive rationality or normative rationality. This does not consider agent limitations. It can be thought of as having two types.
Prescient—outcomes are known and fixed. The decision makers maximise the outcomes with the highest utilities (discounted by costs).
Non-prescient—like the prescient model, but it integrates risk and uncertainty by associating a probability distribution with the models where the probability is estimated by the decision maker.
In both cases, choices among competing goals are handled by something like indifference curves.
We could say that under the comprehensive rational model a rational agent is one that maximizes its expected utility, given its current knowledge.
When we talk about rationality, though, we normally mean in regards to humans. This means that we are talking about bounded rationality. Like comprehensive rationality, bounded rationality assumes that agents are goal-oriented, but bounded rationality also takes into account the cognitive limitations of decision makers in attempting to achieve those goals.
Bounded rationality deals with agents that are limited in many ways which include being:
Unable to determine all outcomes. Organisms with cognitive limitations have a need to satisfice and an inability to consider long sequential outcomes that are inextricably tied. There is also a tendency to focus on a specific set of the overall goals or outcomes due to priming.framing.
Unable to determine all of the pertinent information.
Unable to determine all of the possible inferences.
The big difference between bounded rationality and normative rationality is that in bounded rationality you also consider the agent improving its ability to choose or come up with the best outcomes as rational as long as there are no costs or missed opportunities involved.. Therefore, a rational agent, in the bounded sense, is one that has three characteristics:
It has a honed ability to return decent sets of outcomes from its searches for outcomes
The expected utiltity it assigns to outcomes accurately matches the actual expected utiltity
It chooses the best outcome returned by its searches for outcomes. The best outcome is that one with the highest expected utiltity (discounted by costs)
Do you think it could be reformulated in the framework where values form tree-like networks with some values being “deep” or “primary” and other values being “shallow” or “derived” or “secondary”? Then you might be able to argue that a conflict between a deep and a shallow value should be resolved by the declaring the shallow value not rational.
I think that once a value is in. It is in and works just like all the others in terms of its impact on valuation. However, a distinction like the one you talked about makes sense. But, I would not have ‘deep’ and ‘shallow’ because I have no idea how to determine that. Perhaps, ‘changeable’ vs ‘non-changeable’ would be better. Then, you can look at some conflicting values, i.e. ones that lead you to want opposite things, and ask if any of them are changeable and what the impact is from changing them. The values that relate to what you actually need are non-changeable or at least would cause languishing if you tried to repress them. I think the problem with the tree view is that values are complex, like you were talking about before, one value may conflict with multiple other values.
So how to avoid being caught in a loop: values depend on values which depend on values that depend on values..?
I don’t see the loop. This is because there is no ‘value’. There is only coherence which is just how much it conflicts with the other values. I don’t know how to describe this without an eidetic example. Please let me know if this doesn’t work. Imagine one of those old style screensavers where you have a ball moving across the screen and when it hits the side of the screen it bounces in a random direction. Now, when you have a single ball it can go in any direction at all. There is no concept of coherence because there is only one ball. It is when you introduce another ball that the direction starts to matter as there is now the factor of coherence between the balls. By coherence I mean simply that you don’t want the balls to hit each other. This restricts their movement and it now becomes optimal for them to move in some kind of pattern with vertical or horizontal lines being the simplest,
What this means for values is that you want them to basically be directed towards the same or similar targets or at least targets that are not conflicting. A potential indicator of an irrational value is one that conflicts with other values, Of course, human values are not coherent. But, incoherence is still an indicator of potential irrationality.
Unrelated to the above examples is that you would need to think about if the target of the value is actually valuable and is worth the costs you have to pay to achieve it, this is harder to find out, but you can look at the fundamental human needs. Maybe, your deep vs. shallow distinction would be useful in this context.
I am not sure I understand—is “most trusted source” subjective? What if Jesus is my most trusted source? And He is for a great deal of people.
I don’t think I am conveying this point well. I am trying to say that we only have an incomplete answer as to what is rational and that science provides the best answer we have.
One very common way of making a definition is to point to a well-known class,
I think instead of that type of defintion I would rather say that rationality means doing well in the areas of X, Y and Z. and then have a list of skills or domains that improve your ability in the areas of rationality.
Do you think that there are many types of rationality? I think that there are many types of methods to achieve rationality, but I don’t think there are many types of rationality.
So if we were to try to give an is-a-kind-of defintion of rationality, what is the super-class? Is it reasoning? Is it skills? Something else?
I would say reasoning or maybe problem solving and outcome generation/choosing better convey the idea of it.
Sorry, if I was meandering and repeating my points. I wasn’t viewing this as an argument, so I don’t view it as going in circles, but as going through a series of drafts. Maybe, I will need to be more careful in the future.
I appreciate your feedback.
In regards to what we talked about, I am not really that happy with how rationality is defined in the literature, but I am also not sure of what a better way to define it would be. I guess I will have to look into the bounded types of rationality.
No, that’s perfectly fine, I wasn’t treating it as an argument, either. It’s just that you are spending a lot of time thinking about it, and I’m spending less time, so, having made some points, I really don’t have much more to contribute and I don’t want to fisk your thinking notes. No need to be careful in drafts, that’s not what they are for :-)
Mea culpa, but the rewrite doesn’t look great to me. Before, you first paragraph had some zing. People like me could and did find fault with it, sure, but at least it was energetic. And now the first two sentences are followed by a lot of hemming and hawing which sounds defensive and is entirely uninspiring.
“Ensuring that resource usage and behaviour/thought cordination is directed towards the fulfillment of the agents goals” was already being taught by senior slave-drivers to junior slave-drivers when the pyramids were being built. In trying to avoid rationality be just prediction, you made it be just effectiveness.
I don’t have a good suggestion for you, in fact I’m not sure that the so-called epistemic rationality (aka science) and instrumental rationality (aka pragmatism and keeping your eye on the ball) can be usefully joined together into a single concept. But since you are writing a compendium, you probably should come up with a reasonable definition for rationality, since it is, y’know, a core concept.
I changed it again.
In terms of the defintiion, it is in the title. As to what is is. I am basically trying to convey the idea that rationality is optimal thinking. Although, I suppose I am also happy with how its defined in this book. If you think the below definitions are better, let me know.
Rationality: the property of a system which does the “right thing” given what it knows.
It is an gent that acts to maximize its expected performance measure. That is, it does the “right thing”.
For each possible percept sequence, an ideal rational agent should do whatever action is expected to maximize its performance measure, on the basis of the evidence provided by the percept sequence and whatever built-in knowledge the agent has
An omniscient agent knows the actual outcome of its actions, and can act accordingly, but omniscience is impossible in reality. It is important to distinguish rationality and omniscience. This is because we should not blame an agent for failing to take into account something it could not perceive, or for failing to take an action that it is incapable of taking.
That’s not really a definition: you just shifted the entire burden onto the word “optimally”. A basic use of a definition is to see if something fits it—if we defined a class A, is object z a member of that class? So let’s say I’m considering some action. Is it rational? Well, it is if it’s optimal. Err.. and what does that mean? To answer I need to define optimality and that is not trivial. And if you say that optimality is maximizing (expected) utility, we’re back to your original definition which I poked at and you abandoned.
That’s exactly the same thing—replacing one word with another (or two) without clarifying anything.
That’s maximizing expected utility, again.
Yes, of course, but that doesn’t help you with a definition.
Before we get too deeply into problems with whatever definition I might use, I want to make sure that you agree that what I am trying to say is right. Once that is confirmed, then I can think more about how to say it well.
This is basically what I am trying to say in the post. ‘Rationality’ is ‘optimal reasoning’ which we know of as normative rationality, i.e. the laws of thought and action or what the perfect agent would do. A caveat is that when we talk about rationality in regards to limited agents we are really talking about bounded rationality. Hence, rationality in this case is really ‘bounded optimal reasoning’. So for limited agents, rationality is about the reasoning that best approximates the results of the normative rationality. Also, for limited agents we consider those types of thinking that lead to more optimal reasoning in the future to be rational as well. This is the basis for epistemic and instrumental rationality and is why curiosity is rational. Curiosity often leads to better maps. Better maps often leads to better decisions and, therefore, we consider curiosity to be rational. In terms of costs, the goal of rationality in limited agents is to best approximate the normative rationality. This requires the highest return on investment on the resources that you make use of. Basically the resource usage to expected utiltity ratio should be high since, we only have a limited amount of resources that we can make us of. If there is an alternative way to spend the same resources which has a higher usage to expected utility ratio, then you are not as close as possible.to the normative rationality.
A good idea. I’m not nipicking about wording, at issue is actually meaning.
First, as I mentioned before, I am not sure how to combine epistemic and instrumental rationality together into one useful concept. I am not saying it’s impossible, just that nothing comes to my mind.
One issue, for example, is that they belong to different categories: one is about knowing and the other is about doing. Yes, you can trivially stick them together by saying that epistemic rationality is just instrumental rationality with the goal of constructing a good map, but I don’t know what you gain by that. Constructing a good map is, basically, the scientific method and it is not a decision theory.
Second, I have problems with the “what the perfect agent would do”. An immediate issue is that the answer to that is “You don’t know and you will never know” for any noticeably complex problem[1], especially one that concerns the messy real world and not, for example, the neat and well-defined world of mathematics. That’s an issue because you set it up as a standard and as a limit to which “bounded optimal reasoning” should converge. But if you don’t know what it is, you don’t know what you should converge to and don’t have a good method to adjudicate competing claims about what is rational.
There are also questions about defining rationality as optimality. Optimality typically involves maximizing some measure, but in a lot of situations what matters is not how to reach the maximum, but rather what is it that you optimize. Is it “rational” to arrive at an optimum for the wrong thing? How do you know what to optimize for? Handwaving about utility remains handwaving because the only utility functions I have seen which actually produce a specific numerical estimate are economic utility functions and they solely care about money.
Moreover, you rarely have the luxury of optimizing for one thing. Typically you have multiple conflicting goals with a mix of different costs to all actions, so deciding how are you going to balance goals and summarize costs is very important and I have no idea what is the “rational” way to go about it.
All in all, I am not satisfied by the “perfect agent” or “optimize utility” definitions of rationality. The perfect agent approach is essentially WWJD—What Would Jesus Do—only without the religious baggage, and optimizing utility doesn’t tell me how to actually, in practice do that.
Notice that keeping epistemic and instrumental rationality separate works much better. The criterion for epistemic rationality is the match between the map and the reality—this is specific and observable. The criterion for instrumental rationality is whether you reach your goals at a reasonable cost. This is more complicated because of uncertainty of the future: good decisions don’t always lead to good outcomes and good outcomes do not necessarily follow from good decisions. But even here there are things we can look at and handles we can grab and manipulate. But “emulate perfection” or “maximize utility”—I have no idea how to even start doing that.
[1] This is commonly held as entirely obvious in LW—only in the context of AI boxing :-)
If we forget about epistemic and instrumental rationality for a moment and think about what reasoning is aimed at achieving. That is, why we care about it. Then, I think we can get closer to understanding how epistemic and instrumental rationality might work together to become part of something larger. They are ,of course, still different techniques.
Do you think that the below areas describe what it means to reason well?
It is about actually achieving desired outcomes, so if you want x to occur it is about making sure that you initiate a series of thoughts and actions that lead to x occurring instead of y or z or a plethora of other possibilities.
It is about achieving outcomes at a reasonable cost. If you can achieve your desired outcomes with cheaper costs, then you will be able to achieve more outcomes.
It is about choosing the best outcomes to achieve. We all have limited resources and to achieve any outcomes we need to use resources. This means that we can only chose a limited amount of outcomes to pursue. Making this choice wisely is what this area is about.
It is about valuing the outcomes appropriately. This would be about making your values coherent and correctly valuing the things that matter so that they get priority.
Epistemic/Instrumental rationality is really about certain types of skills that allow us to do well in the above areas (I think I covered all of them). I think that there might also be more types of skills. I have a general idea about what another one might be, I am not sure how many others there might be. Although, I want the compendium to cover existing and established ideas only. That is why I am referring to epistemic and instrumental rationality and not anything else.
What do you think about defining rationality as a property that is attributed by agents to certain thoughts and behaviours? This would mean that it is not only bounded by the agent’s abilities and the information it has, but also by its understanding of what it means to be rational. Essentially, it would mean that ‘rationality’ is subjective. To avoid the fallacy of the grey there needs to be some objective way to judge different agents understanding of what it means to be rational. This objective way is basically our best overall guess at what perfectly optimal reasoning or optimal reasoning for humans would be. For humans, this way is the scientific method with the current body of work pointing to logic, probability and decision theory as having the closest answers on what it means to reason optimally, i.e. be rational. These answers aren’t necessarily correct due to negative pragmatism etc. They are just our current, best and most informed guesses.
I always thought that this was a part of what it means to be instrumentally rational. Basically to have optimal goals as well. This is my problem with instrumental rationality as it’s talked about on less wrong is about achieving what you value at a reasonable cost or is it about making your values coherent and in line with what you innately value or is it a combination of the two. I have always felt that instrumental rationality is a bit too overreaching and encompassing. Do you think I should split it into two types of instrumental rationality? One for costs and one for value alignment or am I not interpreting it correctly.
I will have other posts where I go into detail on each of these separately. I think that they are separate skills or areas of expertise, but I also think that there should be a base reason for why we should care about them.
So do you want to define “rationality” as a kind of reasoning? Reasoning is an opaque mental process and, for example, does not include acting which is a large part of instrumental rationality. Procrastination is a classic LW sin, but it’s not a reasoning problem. And what would be non-rational reasoning besides straightforward logical errors? The great majority of thinking people do throughout the day is not formalizable into a neat system of propositions and conclusions.
Yes, that’s the definition of instrumental rationality.
Hold on, that’s new. Are you claiming that (proper) values are a part of rationality and that rationality will tell you what your values should be? I think I am going to loudly object to that. Maybe you can provide an example to show what you mean?
Hm, that’s an interesting approach. Then you’d consider rationality a kind of skill—a skill like writing essays or programming? This is probably worth exploring further.
Not sure I want to go that way. You wouldn’t have many counterarguments to a bloke which declares himself perfectly rational as he goes to pray to Jesus so that he wins the lottery. And once you introduce an “objective way to judge” there doesn’t seem to be any point to the subjectivity any more.
See above—goals are a direct function of values and I have very strong doubts that rationality can tell you what your values should be.
Humans don’t have coherent values. In fact, I don’t think you can make system of values complex enough to deal with real life fully coherent (people who come close to that are usually called “crazy fanatics”). Instead, what people do is trade off different values against each other and come up with an end-result balance where they are willing to sacrifice some A, B, and C but gain X, Y, and Z. As a crude approximation you can think about it as summing different vectors and acting according to where the summed vector points.
I think that to what degree rationality applies here is a hard question. On the one hand, there is no basis for rationality to say “you need to value this and not value that”. On the other hand, values and their weights are not stable across time, and part of rationality is juggling short-term and long-term desires and consequences—usually pointing out that it’s not smart to pay with a lot of long-term pain for a jolt of short-term pleasure. That’s where this whole bit about “imagine yourself as a very smart, calm, capable human being—what would she choose?” comes in.
So, yes, it’s complicated. I have issues with listening to “It’s not rational to value/desire this”, but I have much less issues with “The price for this action that you want to do is really high, are you quite sure you want to pay it, that doesn’t look rational”. I am not sure where the proper boundary is.
When I use the word reasoning, I really mean both the system 1 and 2 cognitive processes. By rational I basically mean reasoning (system 1 and 2) done well. Where done well, is defined based on your most trusted source. For us this is science, so logic, probability, decision theory etc. for system 2.
I don’t know what “proper” would mean. I am talking about coherence which means that its “properness”, I suppose, depends on its context, i.e. the other pre-existing values. I will give you some examples. I will assume that you already know the difference between wanting and liking.
Excessive Wanting—an example is drug addiction: “Only ‘wanting’ systems sensitize, and so ‘wanting’ can increase and become quite intense due to sensitization, regardless of whether a drug still remains ‘liked’ after many repeated uses”.
Not liking things that you should or could—examples are bad experiences that cause aversion conditioning to something that you used to or could like. My general view is that if you don’t like something and you could then this is a limitation.
Not wanting things you like—ugh fields are an example of this.
Conflicting wants—this is often inevitable like you say value is complex. But, I think it is important to look at what the fundamental human values or needs are and try to align with those. If you don’t, then in general there is a going to be a greater amount of conflict.
I would need to write a full post on the details, but that is just a general idea of what I mean. You also consider the values of others that you are interconnected with and care about.
I don’t see how you can view it as anything but a skill. This is because epistemic rationality, for example, is only valuable instrumentally. It helps makes more rational decisions, but the truer beliefs it causes need to be applied to actually be useful and improve your rationality. If you spend lots of effort creating true beliefs and then compartmentalize that knowledge and don’t apply it, you have effectively gained nothing in terms of rationality. That’s my view anyway. I don’t know how many people would agree. An example is Aumann, he knows a lot about rationality, but I don’t think he is rational because it looks to me like he believes in non overlapping magisteria.
I agree with you on this and your other points on how value is complex. I think that to say that: “it is rational to value/desire this” there needs to a ‘because’ after that statement. No value/desire is rational or irrational in and of itself. It is only irrational or irrational in a context. That is, because of its relation to other values or the costs to fulfil it etc.
Right now, I am thinking that I need to make the base concepts of rationality more solid before I can move into what rationality is for this compendium.
This is my first attempt at defining things. My goal is to define things in a programatic kind of way. This means that the concepts should follow: single responsibility, loose coupling, yagni etc. Let me know what you think.
The goal of the definitions is just to highlight the right areas in concept space. They are drafts and will obviously need more detail. I would also need to submit them as posts and see if others agree.
I am thinking that there should be two basic areas: system 1 and system 2 rationality. Where rationality, in its most basic form, means done well (this will need to be expanded upon). The goal of the two areas is to define what it is we are referring to when we say that something is rational or irrational. There are two areas so that we can distinguish rationality/irrationality in formal reasoning vs. your intuitions or what you actually do vs. what you think you should do.
There are also skills or general topics which describe groups of techniques and methods that can be used to improve your rationality in one or both of the two areas of it. Using these skills means that you apply them using volitional effort. It is noted, however that if you use these skills often enough they are likely to become embedded in your system 1 processes.
There may be more skills, but I think the main ones are below:
Epistemic rationality—true beliefs and all that
Instrumental rationality - (restricted to reasonable costs)
Value coherence rationality—I gave some examples, but it basically means noticing when your values and desires are out of alignment or could become so if you did some action.
Distributive rationality—this is basically what you are talking about in the above quote. Once you have a semi-sufficient valuation system in place how can you actually distribute resources so that you achieve what you value.
Perspectival rationality—no matter how great you are at being rational you are limited by the ideas that you can come up with. You are limited by your paradigms and perspectives. Perspectival rationality is about knowing when to look at a situation from multiple perspectives and having the ability to model the territory or map accurately from another perspective. By modelling the map from another perspective, it is meant that you are thinking about what the maps of someone else or yourself in a future or past tense would be like for a given situation. By modelling the territory, it is meant that you are thinking about what the territory will be like if some situation occurs. An important part of perspectival rationality is being able to coalesce the information from multiple perspectives into a coherent whole. The aim of prrapectival rationality is greater novelty in your ideas, broader utilities in solutions and more pragmatic results. It also includes understanding the necessarily flawed and limited nature of your perspective. You need to constantly be seeking feedback and other perspectives. It would relate to complexity theory, agile software development, systems dynamics, boydian thinking and mental models/schemas/scripts (whatever you want to call it). I plan to write some posts around this idea.
Communicative rationality—how can you communicate well. I will need to look into this one, but I think it’s important.
Applied rationality—This relates to when you already know what the best thing to do is and is about how you can get yourself to actually do it. Examples of this are training will power or courage (doing something you don’t want to, but believe you should), dealing with ugh fields.
I am not sure I understand—is “most trusted source” subjective? What if Jesus is my most trusted source? And He is for a great deal of people.
Do you think it could be reformulated in the framework where values form tree-like networks with some values being “deep” or “primary” and other values being “shallow” or “derived” or “secondary”? Then you might be able to argue that a conflict between a deep and a shallow value should be resolved by the declaring the shallow value not rational.
I meant this more specifically in the looking for a definition context.
One very common way of making a definition is to point to a well-known class, say, Bet and then define a sub-class beta by listing a set of features {X} which allow you to decide whether a particular object b from the super-class Bet belongs to the sub-class beta or not. Such definitions are sometimes called is-a-kind-of definitions: beta is a kind of Bet.
So if we were to try to give an is-a-kind-of defintion of rationality, what is the super-class? Is it reasoning? Is it skills? Something else?
So how to avoid being caught in a loop: values depend on values which depend on values that depend on values..?
Not sure about yagni, since it is not the case that you can always go back to a core definition and easily update it for your new needs. If there’s already a structure built on top of that core definition, changing it might prove to be quite troublesome. Loose coupling and such—sure, if you can pull it off :-) Software architecture is… much less constrained by reality :-)
What do you mean by system 2 rationality? Intuitions that work particularly well? Successful hunches?
That’s a very wide reach. Are you sure you’re not using “rationality” just as a synonym for “doing something really well”?
I mean do well in the areas I talked about before. In summary, I basically mean do well at coming up with solutions to problems or choosing/being able to go through with the best solution, out of all of the solutions you have come up with, to a problem.
I will try to define it again.
First off, there is comprehensive rationality or normative rationality. This does not consider agent limitations. It can be thought of as having two types.
Prescient—outcomes are known and fixed. The decision makers maximise the outcomes with the highest utilities (discounted by costs).
Non-prescient—like the prescient model, but it integrates risk and uncertainty by associating a probability distribution with the models where the probability is estimated by the decision maker.
In both cases, choices among competing goals are handled by something like indifference curves.
We could say that under the comprehensive rational model a rational agent is one that maximizes its expected utility, given its current knowledge.
When we talk about rationality, though, we normally mean in regards to humans. This means that we are talking about bounded rationality. Like comprehensive rationality, bounded rationality assumes that agents are goal-oriented, but bounded rationality also takes into account the cognitive limitations of decision makers in attempting to achieve those goals.
Bounded rationality deals with agents that are limited in many ways which include being:
Unable to determine all outcomes. Organisms with cognitive limitations have a need to satisfice and an inability to consider long sequential outcomes that are inextricably tied. There is also a tendency to focus on a specific set of the overall goals or outcomes due to priming.framing.
Unable to determine all of the pertinent information.
Unable to determine all of the possible inferences.
The big difference between bounded rationality and normative rationality is that in bounded rationality you also consider the agent improving its ability to choose or come up with the best outcomes as rational as long as there are no costs or missed opportunities involved.. Therefore, a rational agent, in the bounded sense, is one that has three characteristics:
It has a honed ability to return decent sets of outcomes from its searches for outcomes
The expected utiltity it assigns to outcomes accurately matches the actual expected utiltity
It chooses the best outcome returned by its searches for outcomes. The best outcome is that one with the highest expected utiltity (discounted by costs)
I think that once a value is in. It is in and works just like all the others in terms of its impact on valuation. However, a distinction like the one you talked about makes sense. But, I would not have ‘deep’ and ‘shallow’ because I have no idea how to determine that. Perhaps, ‘changeable’ vs ‘non-changeable’ would be better. Then, you can look at some conflicting values, i.e. ones that lead you to want opposite things, and ask if any of them are changeable and what the impact is from changing them. The values that relate to what you actually need are non-changeable or at least would cause languishing if you tried to repress them. I think the problem with the tree view is that values are complex, like you were talking about before, one value may conflict with multiple other values.
I don’t see the loop. This is because there is no ‘value’. There is only coherence which is just how much it conflicts with the other values. I don’t know how to describe this without an eidetic example. Please let me know if this doesn’t work. Imagine one of those old style screensavers where you have a ball moving across the screen and when it hits the side of the screen it bounces in a random direction. Now, when you have a single ball it can go in any direction at all. There is no concept of coherence because there is only one ball. It is when you introduce another ball that the direction starts to matter as there is now the factor of coherence between the balls. By coherence I mean simply that you don’t want the balls to hit each other. This restricts their movement and it now becomes optimal for them to move in some kind of pattern with vertical or horizontal lines being the simplest,
What this means for values is that you want them to basically be directed towards the same or similar targets or at least targets that are not conflicting. A potential indicator of an irrational value is one that conflicts with other values, Of course, human values are not coherent. But, incoherence is still an indicator of potential irrationality.
Unrelated to the above examples is that you would need to think about if the target of the value is actually valuable and is worth the costs you have to pay to achieve it, this is harder to find out, but you can look at the fundamental human needs. Maybe, your deep vs. shallow distinction would be useful in this context.
I don’t think I am conveying this point well. I am trying to say that we only have an incomplete answer as to what is rational and that science provides the best answer we have.
I think instead of that type of defintion I would rather say that rationality means doing well in the areas of X, Y and Z. and then have a list of skills or domains that improve your ability in the areas of rationality.
Do you think that there are many types of rationality? I think that there are many types of methods to achieve rationality, but I don’t think there are many types of rationality.
I would say reasoning or maybe problem solving and outcome generation/choosing better convey the idea of it.
I have a feeling we’re starting to go in circles. But it was an interesting conversation and I hope it was useful to you :-)
Sorry, if I was meandering and repeating my points. I wasn’t viewing this as an argument, so I don’t view it as going in circles, but as going through a series of drafts. Maybe, I will need to be more careful in the future. I appreciate your feedback.
In regards to what we talked about, I am not really that happy with how rationality is defined in the literature, but I am also not sure of what a better way to define it would be. I guess I will have to look into the bounded types of rationality.
No, that’s perfectly fine, I wasn’t treating it as an argument, either. It’s just that you are spending a lot of time thinking about it, and I’m spending less time, so, having made some points, I really don’t have much more to contribute and I don’t want to fisk your thinking notes. No need to be careful in drafts, that’s not what they are for :-)