I agree with almost every element of your response, especially the part about how desires emerges in an AI.
Yet I do find the concept of productive and unproductive desires interesting. It seems to point at a distinction between types of goals. For example, the difference between a robot that twitches (it’s code literally just makes it twitch) and a robot that wants to twitch (optimizer trying to maximize the number of times it twitches before getting destroyed). The latter is clearly more goal-directed than the other, and I only fear instrumental subgoals for the latter, not the former. I guess you could say that the former just doesn’t have a goal, and that’s alright if there is really a binary switch “productive” and “unproductive”. But if there is a continuum between the two, then thinking about the end points in terms of goals is relevant in interpreting the degrees of productiveness of goals in the middle.
But if there is a continuum between the two, then thinking about the end points in terms of goals is relevant in interpreting the degrees of productiveness of goals in the middle.
I don’t see why this is the case—you can just think about the continuum from non-goal to goal instead, which should get you the same benefits.
Yeah, rereading myself, you’re right. I think the important thing I wanted to say is just that the productive/unproductive desires or goals seems an interesting idea to formalize an aspect of goal-directedness.
I agree with almost every element of your response, especially the part about how desires emerges in an AI.
Yet I do find the concept of productive and unproductive desires interesting. It seems to point at a distinction between types of goals. For example, the difference between a robot that twitches (it’s code literally just makes it twitch) and a robot that wants to twitch (optimizer trying to maximize the number of times it twitches before getting destroyed). The latter is clearly more goal-directed than the other, and I only fear instrumental subgoals for the latter, not the former. I guess you could say that the former just doesn’t have a goal, and that’s alright if there is really a binary switch “productive” and “unproductive”. But if there is a continuum between the two, then thinking about the end points in terms of goals is relevant in interpreting the degrees of productiveness of goals in the middle.
I don’t see why this is the case—you can just think about the continuum from non-goal to goal instead, which should get you the same benefits.
Yeah, rereading myself, you’re right. I think the important thing I wanted to say is just that the productive/unproductive desires or goals seems an interesting idea to formalize an aspect of goal-directedness.