I personally didn’t feel confused, so I think I mostly turn that question around to you? (i.e. it seemed natural to me to use “consequentalist” in this way, and insofar as any confusion came up, specifying ‘oh, no I didn’t mean it as an ethical theory’ seems like it should address it. But, you might disagree)
I think my personal take is basically “yeah it seems like almost everything routes through a near-consequentialist theory” and “calling this theory ‘consequentialism’ seems fair to me”.
I spend a lot of time with people that are working on AI / AI Alignment who aren’t in the rationality community, and I don’t think this is the take for all of them. In particular I imagine from the “words have meaning, dammit” camp a lot of disagreement about ‘consequentialism’ the term, but if you taboo’d it, there’s a lot of broad agreement here.
In particular, I think this belief is super common and super strong in researchers focused on aligning AGI, or otherwise focused on long-term alignment.
I do think there’s a lot of disagreement in the more near-term alignment research field.
This is why this article felt weird to me—it’s not clear that there is a super wide mistake being made, and to the extent Raemon/John think there is, there’s also a lot of people who are uncertain (again c/f moral uncertainty) even if updating in the ‘thinking/predicting’ direction.
E.g. for this bit:
I… guess what I think Eliezer thinks is that Thoughful Researcher isn’t respecting inner optimizers enough.
My take is median Thoughtful Researcher is more uncertain about inner optimizers—instead of being certain that EY is wrong here.
And pointing at another bit:
Consequentialism is a (relatively) simple, effective process for accomplishing goals, so things that efficiently optimize for goals tend to approximate it.
I think people would disagree with this as consequentialism.
It’s important to maybe point at another term that’s charged with a nontraditional meaning in this community: rationality.
We mean something closer to skeptical empiricism that the actual term, but if you taboo it I think you end up with a lot more agreement about what we’re talking about.
I personally didn’t feel confused, so I think I mostly turn that question around to you? (i.e. it seemed natural to me to use “consequentalist” in this way, and insofar as any confusion came up, specifying ‘oh, no I didn’t mean it as an ethical theory’ seems like it should address it. But, you might disagree)
I think my personal take is basically “yeah it seems like almost everything routes through a near-consequentialist theory” and “calling this theory ‘consequentialism’ seems fair to me”.
I spend a lot of time with people that are working on AI / AI Alignment who aren’t in the rationality community, and I don’t think this is the take for all of them. In particular I imagine from the “words have meaning, dammit” camp a lot of disagreement about ‘consequentialism’ the term, but if you taboo’d it, there’s a lot of broad agreement here.
In particular, I think this belief is super common and super strong in researchers focused on aligning AGI, or otherwise focused on long-term alignment.
I do think there’s a lot of disagreement in the more near-term alignment research field.
This is why this article felt weird to me—it’s not clear that there is a super wide mistake being made, and to the extent Raemon/John think there is, there’s also a lot of people who are uncertain (again c/f moral uncertainty) even if updating in the ‘thinking/predicting’ direction.
E.g. for this bit:
My take is median Thoughtful Researcher is more uncertain about inner optimizers—instead of being certain that EY is wrong here.
And pointing at another bit:
I think people would disagree with this as consequentialism.
It’s important to maybe point at another term that’s charged with a nontraditional meaning in this community: rationality.
We mean something closer to skeptical empiricism that the actual term, but if you taboo it I think you end up with a lot more agreement about what we’re talking about.