I pretty strongly agree with this review (and jtbc it was written without any input from me, even though Daniel and I are both at CHAI).
I think of ‘coherence arguments’ as including things like ‘it’s not possible for you to agree to give me a limitless number of dollars in return for nothing’, which does imply some degree of ‘goal-direction’.
Yeah, maybe I should say “coherence theorems” to be clearer about this? (Like, it isn’t a theorem that I shouldn’t give you limitless number of dollars in return for nothing; maybe I think that you are more capable than me and fully aligned with me, and so you’d do a better job with my money. Or maybe I value your happiness, and the best way to purchase it is to give you money no strings attached.)
Responses from outside this camp
Fwiw, I do in fact worry about goal-directedness, but (I think) I know what you mean. (For others, I think Daniel is referring to something like “the MIRI camp”, though that is also not an accurate pointer, and it is true that I am outside that camp.)
Depends on the distribution over utility functions, the action space, etc, but e.g. if it uniformly selects a numeric reward value for each possible trajectory (state-action sequence) where the actions are low-level (e.g. human muscle control), astronomically low.
That will probably be a good model for some (many?) powerful AI systems that humans build.
I don’t know. (I think it depends quite strongly on the way in which we train powerful AI systems.)
Not likely at low levels of intelligence, plausible at higher levels of intelligence, but really the question is not specified enough.
Well, I didn’t consult you in the process of writing the review, but we’ve had many conversations on the topic which presumably have influenced how I think about the topic and what I ended up writing in the review.
I think of ‘coherence arguments’ as including things like ‘it’s not possible for you to agree to give me a limitless number of dollars in return for nothing’, which does imply some degree of ‘goal-direction’.
Yeah, maybe I should say “coherence theorems” to be clearer about this?
Sorry, I meant theorems taking ‘no limitless dollar sink’ as an axiom and deriving something interesting from that.
I pretty strongly agree with this review (and jtbc it was written without any input from me, even though Daniel and I are both at CHAI).
Yeah, maybe I should say “coherence theorems” to be clearer about this? (Like, it isn’t a theorem that I shouldn’t give you limitless number of dollars in return for nothing; maybe I think that you are more capable than me and fully aligned with me, and so you’d do a better job with my money. Or maybe I value your happiness, and the best way to purchase it is to give you money no strings attached.)
Fwiw, I do in fact worry about goal-directedness, but (I think) I know what you mean. (For others, I think Daniel is referring to something like “the MIRI camp”, though that is also not an accurate pointer, and it is true that I am outside that camp.)
My responses to the questions:
The ones in Will humans build goal-directed agents?, but if you want arguments that aren’t about humans, then I don’t know.
Depends on the distribution over utility functions, the action space, etc, but e.g. if it uniformly selects a numeric reward value for each possible trajectory (state-action sequence) where the actions are low-level (e.g. human muscle control), astronomically low.
That will probably be a good model for some (many?) powerful AI systems that humans build.
I don’t know. (I think it depends quite strongly on the way in which we train powerful AI systems.)
Not likely at low levels of intelligence, plausible at higher levels of intelligence, but really the question is not specified enough.
Well, I didn’t consult you in the process of writing the review, but we’ve had many conversations on the topic which presumably have influenced how I think about the topic and what I ended up writing in the review.
Sorry, I meant theorems taking ‘no limitless dollar sink’ as an axiom and deriving something interesting from that.