Yes, I’m saying that AIs are very likely to have (in a broad sense, including e.g. having subagents that have) long-term goals.
Now why couldn’t an agent by motivated to maximize short-term paperclips?
It *could*, but I’m saying that making an AI like that isn’t like choosing a loss function for training, because long-term thinking is convergent.
Your original comment said:
I can’t see anything unnatural about an agent that has both consequentialist reasoning capabilities and a high time preference.
This is what I’m arguing against. I’m saying it’s very unnatural. *Possible*, but very unnatural.
And:
This means that it would never sacrifice reward now for reward later, and so it would essentially be exempt from instrumental convergence.
This sounds like you’re saying that myopia *makes* there not be convergent instrumental goals. I’m saying myopia basically *implies* there not being convergent instrumental goals, and therefore is at least as hard as making there not be CIGs.
Yes, I’m saying that AIs are very likely to have (in a broad sense, including e.g. having subagents that have) long-term goals.
It *could*, but I’m saying that making an AI like that isn’t like choosing a loss function for training, because long-term thinking is convergent.
Your original comment said:
This is what I’m arguing against. I’m saying it’s very unnatural. *Possible*, but very unnatural.
And:
This sounds like you’re saying that myopia *makes* there not be convergent instrumental goals. I’m saying myopia basically *implies* there not being convergent instrumental goals, and therefore is at least as hard as making there not be CIGs.