Putting my cards on the table, this is my guess at the answers to the questions that I raise:
I don’t know.
Low.
Frequent if it’s an ‘intelligent’ one.
Relatively. You probably don’t end up with systems that resist literally all changes to their goals, but you probably do end up with systems that resist most changes to their goals, barring specific effort to prevent that.
Probably.
That being said, I think that a better definition of ‘goal-directedness’ would go a long way in making me less confused by the topic.
“random utility-maximizer” is pretty ambiguous; if you imagine the space of all possible utility functions over action-observation histories and you imagine a uniform distribution over them (suppose they’re finite, so this is doable), then the answer is low.
Heh, looking at my comment it turns out I said roughly the same thing 3 years ago.
Putting my cards on the table, this is my guess at the answers to the questions that I raise:
I don’t know.
Low.
Frequent if it’s an ‘intelligent’ one.
Relatively. You probably don’t end up with systems that resist literally all changes to their goals, but you probably do end up with systems that resist most changes to their goals, barring specific effort to prevent that.
Probably.
That being said, I think that a better definition of ‘goal-directedness’ would go a long way in making me less confused by the topic.
I have no idea why I responded ‘low’ to 2. Does anybody think that’s reasonable and fits in with what I wrote here, or did I just mean high?
“random utility-maximizer” is pretty ambiguous; if you imagine the space of all possible utility functions over action-observation histories and you imagine a uniform distribution over them (suppose they’re finite, so this is doable), then the answer is low.
Heh, looking at my comment it turns out I said roughly the same thing 3 years ago.