You mean “values aren’t especially mysterious”, I expect.
I agree that they’re not mysterious. More specifically, I agree that what it means to capture my value information about X and Y is to capture the information someone else would need in order to accurately and reliably predict my relative preferences for X and Y under a wide range of conditions. And, yes, a large chunk of that information is what you describe here as encyclopedic knowledge.
So for (X,Y)=(mulberry, blackberry) a high-fidelity copy of my values, in conjunction with a suitable encyclopedia of berries, would allow you to reliably predict which one I would prefer to eat with chocolate ice cream, which one I would prefer to spread as jam on rye bread, which one I would prefer to decorate a cake with, which one I would prefer to receive a pint of as a gift, how many pints of one I’d exchange for a pint of the other, etc., etc., etc.
Yes?
Assuming I’ve gotten that right… so, when you say:
We do have high fidelity copying today. We can accurately copy anything we can represent as digital information—including values ...do you mean to suggest that we can, today, create a high-fidelity copy of my values with respect to mulberries and blackberries as described above?
(Obviously, this is a very simple problem in degenerate cases like “I like blackberries and hate mulberries,” but that’s not all that interesting.)
If so, do you know of any examples of that sort of high-fidelity copy of someone’s values with respect to some non-degenerate (X,Y) pair actually having been created? Can you point me at one?
I can’t meet your “complex value extraction” challenge. I never meant to imply “complete” extraction—just that we can extract value information (like this) and then copy it around with high fidelity. Revealed preferences can be good, but I wouldn’t like to get into quantifying their accuracy here.
OK. I certainly agree that any information we know how to digitally encode in the first place, we can copy around with high fidelity. But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them. That’s not because value is some kind of mysterious abstract ethereal “whatness of the if”… we can define it concretely as the stuff that informs, and in principle allows an observer to predict, our revealed preferences… but because it’s complicated. I’m inclined to agree with Wei_Dai that high-fidelity value sharing would represent a significant breakthrough in our understanding of and our ability to engineer human psychology, and would likely be a game-changer.
But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them.
Well, we do have the idea of revealed preference. Also, if you want to know what people value, you can often try asking them. Between them, these ideas work quite well.
What we can’t do is build a machine that optimises them—so there is something missing, but it’s mostly not value information. We can’t automatically perform inductive inference very well, for one thing.
I suspect I agree with you about what information we can encode today, and you seem to agree with me that there’s additional information in our brains (for example, information about berries) that we use to make those judgments which revealed preferences (and to a lesser extent explicitly articulated preferences) report on, which we don’t yet know how to encode.
I don’t really care whether we call that additional information “value information” or not; I thought initially you were claiming that we could in practice encode it. Thank you for clarifying.
Also agreed that there are operations our brains perform that we don’t know how to automate.
You mean “values aren’t especially mysterious”, I expect.
I agree that they’re not mysterious. More specifically, I agree that what it means to capture my value information about X and Y is to capture the information someone else would need in order to accurately and reliably predict my relative preferences for X and Y under a wide range of conditions. And, yes, a large chunk of that information is what you describe here as encyclopedic knowledge.
So for (X,Y)=(mulberry, blackberry) a high-fidelity copy of my values, in conjunction with a suitable encyclopedia of berries, would allow you to reliably predict which one I would prefer to eat with chocolate ice cream, which one I would prefer to spread as jam on rye bread, which one I would prefer to decorate a cake with, which one I would prefer to receive a pint of as a gift, how many pints of one I’d exchange for a pint of the other, etc., etc., etc.
Yes?
Assuming I’ve gotten that right… so, when you say:
(Obviously, this is a very simple problem in degenerate cases like “I like blackberries and hate mulberries,” but that’s not all that interesting.)
If so, do you know of any examples of that sort of high-fidelity copy of someone’s values with respect to some non-degenerate (X,Y) pair actually having been created? Can you point me at one?
I can’t meet your “complex value extraction” challenge. I never meant to imply “complete” extraction—just that we can extract value information (like this) and then copy it around with high fidelity. Revealed preferences can be good, but I wouldn’t like to get into quantifying their accuracy here.
OK.
I certainly agree that any information we know how to digitally encode in the first place, we can copy around with high fidelity.
But we don’t know how to digitally encode our values in the first place, so we don’t know how to copy them. That’s not because value is some kind of mysterious abstract ethereal “whatness of the if”… we can define it concretely as the stuff that informs, and in principle allows an observer to predict, our revealed preferences… but because it’s complicated.
I’m inclined to agree with Wei_Dai that high-fidelity value sharing would represent a significant breakthrough in our understanding of and our ability to engineer human psychology, and would likely be a game-changer.
Well, we do have the idea of revealed preference. Also, if you want to know what people value, you can often try asking them. Between them, these ideas work quite well.
What we can’t do is build a machine that optimises them—so there is something missing, but it’s mostly not value information. We can’t automatically perform inductive inference very well, for one thing.
I suspect I agree with you about what information we can encode today, and you seem to agree with me that there’s additional information in our brains (for example, information about berries) that we use to make those judgments which revealed preferences (and to a lesser extent explicitly articulated preferences) report on, which we don’t yet know how to encode.
I don’t really care whether we call that additional information “value information” or not; I thought initially you were claiming that we could in practice encode it. Thank you for clarifying.
Also agreed that there are operations our brains perform that we don’t know how to automate.