I don’t think people who make this argument explicitly reject brain physicalism—they won’t come to you straight up saying “I believe we have a God-given immortal soul and these mechanical husks will never possess one”. However, if hard-pressed, they’ll probably start giving out arguments that kind of sound like dualism (in which the machines can’t possibly ever catch up to human creativity for reasons). Mostly, I think it’s just the latest iteration of existential discomfort at finding out we’re perhaps not so special after all, like with “the Earth is not the centre of the Universe” and “we’re just descendants of apes, ruled by the same laws as all other animals”.
That said, I think then an interesting direction to take the discussion would be “ok, let’s say these machines can NEVER experience anything and that conscious experience is a requirement to be able to express certain feelings and forms of creativity. Do you think it is also necessary to prove mathematical theorems, make scientific discoveries or plan a deception?”. Because in the end, that’s what would make an AI really dangerous, even if its art remained eternally kinda mid.
I don’t think people who make this argument explicitly reject brain physicalism—they won’t come to you straight up saying “I believe we have a God-given immortal soul and these mechanical husks will never possess one”. However, if hard-pressed, they’ll probably start giving out arguments that kind of sound like dualism (in which the machines can’t possibly ever catch up to human creativity for reasons). Mostly, I think it’s just the latest iteration of existential discomfort at finding out we’re perhaps not so special after all, like with “the Earth is not the centre of the Universe” and “we’re just descendants of apes, ruled by the same laws as all other animals”.
That said, I think then an interesting direction to take the discussion would be “ok, let’s say these machines can NEVER experience anything and that conscious experience is a requirement to be able to express certain feelings and forms of creativity. Do you think it is also necessary to prove mathematical theorems, make scientific discoveries or plan a deception?”. Because in the end, that’s what would make an AI really dangerous, even if its art remained eternally kinda mid.
Good point. I think I will add it later.