Robin has a post that in part addresses the question of how much value sharing can improve cooperation:
On the general abstract argument, we see a common pattern in both the evolution of species and human organizations — while winning systems often enforce substantial value sharing and loyalty on small scales, they achieve much less on larger scales. Values tend to be more integrated in a single organism’s brain, relative to larger families or species, and in a team or firm, relative to a nation or world. Value coordination seems hard, especially on larger scales.
This is not especially puzzling theoretically. While there can be huge gains to coordination, especially in war, it is far less obvious just how much one needs value sharing to gain action coordination. There are many other factors that influence coordination, after all; even perfect value matching is consistent with quite poor coordination. It is also far from obvious that values in generic large minds can easily be separated from other large mind parts. When the parts of large systems evolve independently, to adapt to differing local circumstances, their values may also evolve independently. Detecting and eliminating value divergences might in general be quite expensive.
In general, it is not at all obvious that the benefits of more value sharing are worth these costs. And even if more value sharing is worth the costs, that would only imply that value-sharing entities should be a bit larger than they are now, not that they should shift to a world-encompassing extreme.
My own intuition is that high fidelity value sharing (the kind made possible by mind copying / resets) would be a major breakthrough, and not just an incremental improvement as Robin suggests.
My own intuition is that high fidelity value sharing (the kind made possible by mind copying / resets) would be a major breakthrough, and not just an incremental improvement as Robin suggests.
Wouldn’t the indexicality of human values lead to Calvin problems, if that’s the kind of mind you’re copying?
My own intuition is that high fidelity value sharing (the kind made possible by mind copying / resets) would be a major breakthrough, and not just an incremental improvement as Robin suggests.
We do have high fidelity copying today. We can accurately copy anything we can represent as digital information—including values. While we can copy values, one problem is that we can’t easily convince people to “install” them. Instead the values of others often get rejected by people’s memetic immune system as attempts at manipulation.
If we can copy values, or represent them as digital information, I haven’t heard about it.
The closest thing I’ve seen is tools for exporting values into an intersubjective format like speech, writing, art, or behavior. As you point out, the subsequent corresponding import often fails… whether that’s because of explicit defense mechanisms, or because the exported data structure lacks key data, or because the import process is defective in some way, or for some other reason, is hard to tease out.
Maybe you mean something different from me by the term ‘values’. The values I was referring to are fairly simple to write down. Many of them are so codified in legal systems and religious traditions.
If I tell you that I like mulberries more than blackberries, then that’s information about my values represented digitally. The guts of the value information really is in there. Consequently, you can accurately make predictions about what I will do if presented with various food choices—without actually personally adopting my values.
Yeah, we apparently mean different things. To my mind the statement “I like mulberries more than blackberries” is not even close to being a high-fidelity copy of my relative preferences for mulberries and blackberries; I could not reconstruct the latter given only the former.
I would classify it as being value information which can be accurately copied. I never meant to suggest that it was an accurate copy of what was in your mind. For instance, your mental representation could be considered to include additional information about what mulberries and blackberries are—broadly related to what might be found in an encyclopedia of berries. The point is that we can represent people’s values digitally—to the point where we can make quite good predictions about what choices they will make under controlled conditions involving value-derived choices. Values aren’t especially mysterious, they are just what people want—and we have a big mountain of information about that which can be represented digitally.
You mean “values aren’t especially mysterious”, I expect.
I agree that they’re not mysterious. More specifically, I agree that what it means to capture my value information about X and Y is to capture the information someone else would need in order to accurately and reliably predict my relative preferences for X and Y under a wide range of conditions. And, yes, a large chunk of that information is what you describe here as encyclopedic knowledge.
So for (X,Y)=(mulberry, blackberry) a high-fidelity copy of my values, in conjunction with a suitable encyclopedia of berries, would allow you to reliably predict which one I would prefer to eat with chocolate ice cream, which one I would prefer to spread as jam on rye bread, which one I would prefer to decorate a cake with, which one I would prefer to receive a pint of as a gift, how many pints of one I’d exchange for a pint of the other, etc., etc., etc.
Yes?
Assuming I’ve gotten that right… so, when you say:
We do have high fidelity copying today. We can accurately copy anything we can represent as digital information—including values ...do you mean to suggest that we can, today, create a high-fidelity copy of my values with respect to mulberries and blackberries as described above?
(Obviously, this is a very simple problem in degenerate cases like “I like blackberries and hate mulberries,” but that’s not all that interesting.)
If so, do you know of any examples of that sort of high-fidelity copy of someone’s values with respect to some non-degenerate (X,Y) pair actually having been created? Can you point me at one?
I can’t meet your “complex value extraction” challenge. I never meant to imply “complete” extraction—just that we can extract value information (like this) and then copy it around with high fidelity. Revealed preferences can be good, but I wouldn’t like to get into quantifying their accuracy here.
OK. I certainly agree that any information we know how to digitally encode in the first place, we can copy around with high fidelity. But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them. That’s not because value is some kind of mysterious abstract ethereal “whatness of the if”… we can define it concretely as the stuff that informs, and in principle allows an observer to predict, our revealed preferences… but because it’s complicated. I’m inclined to agree with Wei_Dai that high-fidelity value sharing would represent a significant breakthrough in our understanding of and our ability to engineer human psychology, and would likely be a game-changer.
But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them.
Well, we do have the idea of revealed preference. Also, if you want to know what people value, you can often try asking them. Between them, these ideas work quite well.
What we can’t do is build a machine that optimises them—so there is something missing, but it’s mostly not value information. We can’t automatically perform inductive inference very well, for one thing.
I suspect I agree with you about what information we can encode today, and you seem to agree with me that there’s additional information in our brains (for example, information about berries) that we use to make those judgments which revealed preferences (and to a lesser extent explicitly articulated preferences) report on, which we don’t yet know how to encode.
I don’t really care whether we call that additional information “value information” or not; I thought initially you were claiming that we could in practice encode it. Thank you for clarifying.
Also agreed that there are operations our brains perform that we don’t know how to automate.
This sounds like a semantic quibble to me. Okay, maybe the main problem is not in copying but in “installing”, but wouldn’t mind copying effectively make “installation” much easier, as well?
It wasn’t intended as a semantic quibble—the idea was more to say: is there really a “major breakthrough” here? If so, what does it consist of? I was arguing against it being “high fidelity value sharing”.
Mind copying would indeed bypass the “installation” issue.
Robin has a post that in part addresses the question of how much value sharing can improve cooperation:
My own intuition is that high fidelity value sharing (the kind made possible by mind copying / resets) would be a major breakthrough, and not just an incremental improvement as Robin suggests.
Wouldn’t the indexicality of human values lead to Calvin problems, if that’s the kind of mind you’re copying?
We do have high fidelity copying today. We can accurately copy anything we can represent as digital information—including values. While we can copy values, one problem is that we can’t easily convince people to “install” them. Instead the values of others often get rejected by people’s memetic immune system as attempts at manipulation.
If we can copy values, or represent them as digital information, I haven’t heard about it.
The closest thing I’ve seen is tools for exporting values into an intersubjective format like speech, writing, art, or behavior. As you point out, the subsequent corresponding import often fails… whether that’s because of explicit defense mechanisms, or because the exported data structure lacks key data, or because the import process is defective in some way, or for some other reason, is hard to tease out.
Maybe you mean something different from me by the term ‘values’. The values I was referring to are fairly simple to write down. Many of them are so codified in legal systems and religious traditions.
If I tell you that I like mulberries more than blackberries, then that’s information about my values represented digitally. The guts of the value information really is in there. Consequently, you can accurately make predictions about what I will do if presented with various food choices—without actually personally adopting my values.
Yeah, we apparently mean different things. To my mind the statement “I like mulberries more than blackberries” is not even close to being a high-fidelity copy of my relative preferences for mulberries and blackberries; I could not reconstruct the latter given only the former.
I would classify it as being value information which can be accurately copied. I never meant to suggest that it was an accurate copy of what was in your mind. For instance, your mental representation could be considered to include additional information about what mulberries and blackberries are—broadly related to what might be found in an encyclopedia of berries. The point is that we can represent people’s values digitally—to the point where we can make quite good predictions about what choices they will make under controlled conditions involving value-derived choices. Values aren’t especially mysterious, they are just what people want—and we have a big mountain of information about that which can be represented digitally.
You mean “values aren’t especially mysterious”, I expect.
I agree that they’re not mysterious. More specifically, I agree that what it means to capture my value information about X and Y is to capture the information someone else would need in order to accurately and reliably predict my relative preferences for X and Y under a wide range of conditions. And, yes, a large chunk of that information is what you describe here as encyclopedic knowledge.
So for (X,Y)=(mulberry, blackberry) a high-fidelity copy of my values, in conjunction with a suitable encyclopedia of berries, would allow you to reliably predict which one I would prefer to eat with chocolate ice cream, which one I would prefer to spread as jam on rye bread, which one I would prefer to decorate a cake with, which one I would prefer to receive a pint of as a gift, how many pints of one I’d exchange for a pint of the other, etc., etc., etc.
Yes?
Assuming I’ve gotten that right… so, when you say:
(Obviously, this is a very simple problem in degenerate cases like “I like blackberries and hate mulberries,” but that’s not all that interesting.)
If so, do you know of any examples of that sort of high-fidelity copy of someone’s values with respect to some non-degenerate (X,Y) pair actually having been created? Can you point me at one?
I can’t meet your “complex value extraction” challenge. I never meant to imply “complete” extraction—just that we can extract value information (like this) and then copy it around with high fidelity. Revealed preferences can be good, but I wouldn’t like to get into quantifying their accuracy here.
OK.
I certainly agree that any information we know how to digitally encode in the first place, we can copy around with high fidelity.
But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them. That’s not because value is some kind of mysterious abstract ethereal “whatness of the if”… we can define it concretely as the stuff that informs, and in principle allows an observer to predict, our revealed preferences… but because it’s complicated.
I’m inclined to agree with Wei_Dai that high-fidelity value sharing would represent a significant breakthrough in our understanding of and our ability to engineer human psychology, and would likely be a game-changer.
Well, we do have the idea of revealed preference. Also, if you want to know what people value, you can often try asking them. Between them, these ideas work quite well.
What we can’t do is build a machine that optimises them—so there is something missing, but it’s mostly not value information. We can’t automatically perform inductive inference very well, for one thing.
I suspect I agree with you about what information we can encode today, and you seem to agree with me that there’s additional information in our brains (for example, information about berries) that we use to make those judgments which revealed preferences (and to a lesser extent explicitly articulated preferences) report on, which we don’t yet know how to encode.
I don’t really care whether we call that additional information “value information” or not; I thought initially you were claiming that we could in practice encode it. Thank you for clarifying.
Also agreed that there are operations our brains perform that we don’t know how to automate.
This sounds like a semantic quibble to me. Okay, maybe the main problem is not in copying but in “installing”, but wouldn’t mind copying effectively make “installation” much easier, as well?
It wasn’t intended as a semantic quibble—the idea was more to say: is there really a “major breakthrough” here? If so, what does it consist of? I was arguing against it being “high fidelity value sharing”.
Mind copying would indeed bypass the “installation” issue.