So I wouldn’t say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant
I don’t know if I would call it “mis-”attribute. My point, confirmed by spxtr and some other commenters, is that agency is relative to the observer, that there is no absolute difference between a “true” agency and an “apparent agency”.
Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us [...]
I think most of this statement follows from its last part, “ability to explore solution-space in a way that will end up surprising us”. Once that happens, we assign the rest of the agenty attributes to whatever has surprised us.
But that’s a bug of our own ability to detect agency, not a feature of agency itself.
I guess this is the crux of our disagreement. To a superintelligence, we are CPUs without agency.
I would argue that Theologians have used the wide idea-space of their mythology to cover a lot of questions some of which also applicable outside of their theology.
I mean, it’s not as though religion has a monopoly on that idea. It is mentioned in the article how it has applications in any care-taking role.
Now if you can find someone talking about the possibility of a virgin getting pregnant through her ear or nose, that I will grant you is pretty unique to Christianity in specific time periods when social mores say pregnancy is good, vaginas are bad, and virginity is good..
I don’t know if I would call it “mis-”attribute. My point, confirmed by spxtr and some other commenters, is that agency is relative to the observer, that there is no absolute difference between a “true” agency and an “apparent agency”.
I think most of this statement follows from its last part, “ability to explore solution-space in a way that will end up surprising us”. Once that happens, we assign the rest of the agenty attributes to whatever has surprised us.
I guess this is the crux of our disagreement. To a superintelligence, we are CPUs without agency.
As often happens, LW discusses theology without realizing it.
aka “Do you really have free will if God knows everything you will decide?” X-)
I would argue that Theologians have used the wide idea-space of their mythology to cover a lot of questions some of which also applicable outside of their theology.
I mean, it’s not as though religion has a monopoly on that idea. It is mentioned in the article how it has applications in any care-taking role.
Now if you can find someone talking about the possibility of a virgin getting pregnant through her ear or nose, that I will grant you is pretty unique to Christianity in specific time periods when social mores say pregnancy is good, vaginas are bad, and virginity is good..