Incidentally, I also don’t believe that equating “procrastination” with “our evolutionary desire to rest” is even remotely correct.
I got that from here. For everything else see this comment, would love to hear your opinion. I am not sure what exactly it is that I am confused about. Thank you.
I might agree that we consciously convince ourselves that procrastination is a form of rest, as one of many ways of rationalizing procrastination. But not that we’ve evolved procrastination as a way of satisfying our desire to rest, which is what I understood you to be implying.
Then again, I mostly didn’t buy that post at all, so take my reaction for what it’s worth.
As for the terminal/instrumental goal thing… here, also, I’m probably the wrong guy to ask, as I don’t really buy into the standard LW position.
As I’ve said a few times (most coherently here), I’m not actually sold on the idea that terminal goals exist in the first place.
So, yeah, I would agree that goals change over time, or at least can change, and utility functions (insofar as such things exist) change, and that this whole idea of human values being a fixed fulcrum against which we can measure the motion of the universe isn’t quite right. We are part of the world we change, not some kind of transcendent Unmoved Mover that stands outside of it.
Which is not to say I oppose the enterprise of building optimizing agents in a way that preserves our “terminal values”: that’s the right direction to go in. It’s just that I expect the result of that enterprise will in fact be that we preserve our most stable and mutually-reinforcing “instrumental” values, and they will be somewhat modified by the process.
Growing up is like that sometimes… we become something we couldn’t have conceived of and wouldn’t have approved of, had we been consulted.
To put this in LW terms: I expect that what a sufficiently powerful seed AI extracts from an analysis of humanity’s coherent extrapolated volition will not be, technically speaking, a set of terminal values… rather, I expect it will be a set of particularly stable and mutually reinforcing instrumental values, which are the closest approximation human minds contain to terminal values.
And I don’t expect that implementing that CEV will be a one-time operation where we do it and we Win and nothing more needs to be done ever… rather, I expect that it will be a radical improvement in our environment, which we will become accustomed to, which will alter the balance of our values, which will cause us to identify new goals that we optimize for.
All that said, I don’t object to people making the simplifying assumption that their current targets are actually universal terminal points. For some people (and I’m often one of them!), simplicity is an important motivational factor, and the alternative is to just sit in a paralyzed puddle of inconceivable alternatives.
I got that from here. For everything else see this comment, would love to hear your opinion. I am not sure what exactly it is that I am confused about. Thank you.
I might agree that we consciously convince ourselves that procrastination is a form of rest, as one of many ways of rationalizing procrastination. But not that we’ve evolved procrastination as a way of satisfying our desire to rest, which is what I understood you to be implying.
Then again, I mostly didn’t buy that post at all, so take my reaction for what it’s worth.
As for the terminal/instrumental goal thing… here, also, I’m probably the wrong guy to ask, as I don’t really buy into the standard LW position.
As I’ve said a few times (most coherently here), I’m not actually sold on the idea that terminal goals exist in the first place.
So, yeah, I would agree that goals change over time, or at least can change, and utility functions (insofar as such things exist) change, and that this whole idea of human values being a fixed fulcrum against which we can measure the motion of the universe isn’t quite right. We are part of the world we change, not some kind of transcendent Unmoved Mover that stands outside of it.
Which is not to say I oppose the enterprise of building optimizing agents in a way that preserves our “terminal values”: that’s the right direction to go in. It’s just that I expect the result of that enterprise will in fact be that we preserve our most stable and mutually-reinforcing “instrumental” values, and they will be somewhat modified by the process.
Growing up is like that sometimes… we become something we couldn’t have conceived of and wouldn’t have approved of, had we been consulted.
To put this in LW terms: I expect that what a sufficiently powerful seed AI extracts from an analysis of humanity’s coherent extrapolated volition will not be, technically speaking, a set of terminal values… rather, I expect it will be a set of particularly stable and mutually reinforcing instrumental values, which are the closest approximation human minds contain to terminal values.
And I don’t expect that implementing that CEV will be a one-time operation where we do it and we Win and nothing more needs to be done ever… rather, I expect that it will be a radical improvement in our environment, which we will become accustomed to, which will alter the balance of our values, which will cause us to identify new goals that we optimize for.
All that said, I don’t object to people making the simplifying assumption that their current targets are actually universal terminal points. For some people (and I’m often one of them!), simplicity is an important motivational factor, and the alternative is to just sit in a paralyzed puddle of inconceivable alternatives.