I see your point, but I think you’re confusing a partial overlapping with an identity.
There are many bugs/uncertainty that appear as agency, but there are also many bugs/uncertainty which doesn’t appear as agency (as you said about true randomness), and there are also behavior that are actually smart and that appear as agency because of smartness (like the way I was delighted with Emacs the first time I realized that if I asked it to replace “blue” with “red”, it would replace “Blue” with “Red” and “BLUE” with “RED”), I got the same “feeling of agency” there that I could have on bugs.
So I wouldn’t say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant (because it’s safest to mis-attribute agency to nothing that doesn’t have it, than to not attribute it to something that does have it), the same way our ancestors used to see the sun, storms, volcanoes, … as having agency.
Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us, not because of bugs, but because of its inherent complexity. And we have been evolved to mis-attribute agency to things which behave in unexpected ways. But that’s a bug of our own ability to detect agency, not a feature of agency itself.
So I wouldn’t say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant
I don’t know if I would call it “mis-”attribute. My point, confirmed by spxtr and some other commenters, is that agency is relative to the observer, that there is no absolute difference between a “true” agency and an “apparent agency”.
Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us [...]
I think most of this statement follows from its last part, “ability to explore solution-space in a way that will end up surprising us”. Once that happens, we assign the rest of the agenty attributes to whatever has surprised us.
But that’s a bug of our own ability to detect agency, not a feature of agency itself.
I guess this is the crux of our disagreement. To a superintelligence, we are CPUs without agency.
I would argue that Theologians have used the wide idea-space of their mythology to cover a lot of questions some of which also applicable outside of their theology.
I mean, it’s not as though religion has a monopoly on that idea. It is mentioned in the article how it has applications in any care-taking role.
Now if you can find someone talking about the possibility of a virgin getting pregnant through her ear or nose, that I will grant you is pretty unique to Christianity in specific time periods when social mores say pregnancy is good, vaginas are bad, and virginity is good..
I see your point, but I think you’re confusing a partial overlapping with an identity.
There are many bugs/uncertainty that appear as agency, but there are also many bugs/uncertainty which doesn’t appear as agency (as you said about true randomness), and there are also behavior that are actually smart and that appear as agency because of smartness (like the way I was delighted with Emacs the first time I realized that if I asked it to replace “blue” with “red”, it would replace “Blue” with “Red” and “BLUE” with “RED”), I got the same “feeling of agency” there that I could have on bugs.
So I wouldn’t say that agency is bugs, but that we have evolved to mis-attribute attribute agency to things that are dangerous/unpleasant (because it’s safest to mis-attribute agency to nothing that doesn’t have it, than to not attribute it to something that does have it), the same way our ancestors used to see the sun, storms, volcanoes, … as having agency.
Agency is something different, hard to exactly pinpoint (philosophers have been going at it for centuries), but that involves ability to have a representation of reality, to plan ahead for a goal, a complexity of representation and ability to explore solution-space in a way that will end up surprising us, not because of bugs, but because of its inherent complexity. And we have been evolved to mis-attribute agency to things which behave in unexpected ways. But that’s a bug of our own ability to detect agency, not a feature of agency itself.
I don’t know if I would call it “mis-”attribute. My point, confirmed by spxtr and some other commenters, is that agency is relative to the observer, that there is no absolute difference between a “true” agency and an “apparent agency”.
I think most of this statement follows from its last part, “ability to explore solution-space in a way that will end up surprising us”. Once that happens, we assign the rest of the agenty attributes to whatever has surprised us.
I guess this is the crux of our disagreement. To a superintelligence, we are CPUs without agency.
As often happens, LW discusses theology without realizing it.
aka “Do you really have free will if God knows everything you will decide?” X-)
I would argue that Theologians have used the wide idea-space of their mythology to cover a lot of questions some of which also applicable outside of their theology.
I mean, it’s not as though religion has a monopoly on that idea. It is mentioned in the article how it has applications in any care-taking role.
Now if you can find someone talking about the possibility of a virgin getting pregnant through her ear or nose, that I will grant you is pretty unique to Christianity in specific time periods when social mores say pregnancy is good, vaginas are bad, and virginity is good..