I almost missed that there’s new thoughts here, I thought this was a rehash of your previous post The AI Apocalypse Myth!
The new bit sounds similar to Elon Musk’s curious AI plan. I think this means it has a similar problem: humans are complex and a bounty of data to learn about, but as the adage goes, “all happy families are alike; each unhappy family is unhappy in its own way.” A curious/learning-first AI might make many discoveries about happy humans while it is building up power, and then start putting humans in a greater number of awful but novel and “interesting” situations once it doesn’t need humanity to survive.
That said, this is only a problem if the AI is likely to not be empathetic/compassionate, which if I’m not mistaken, is one of the main things we would disagree on. I think that instead of trying to find these technical workarounds, you should argue for the much more interesting (and important!) position that AIs are likely to be empathetic and compassionate by default.
If instead you do want to be more persuasive with these workarounds, can I suggest adopting more of a security mindset? You appear to be looking for ways in which things can possibly go right, instead of all the ways things can go wrong. Alternatively, you don’t appear to be modeling the doomer mindset very well, so you can’t “put on your doomer hat” and check whether doomers would see your proposal as persuasive. Understanding a different viewpoint in depth is a big ask, but I think you’d find more success that way.
I almost missed that there’s new thoughts here, I thought this was a rehash of your previous post The AI Apocalypse Myth!
The new bit sounds similar to Elon Musk’s curious AI plan. I think this means it has a similar problem: humans are complex and a bounty of data to learn about, but as the adage goes, “all happy families are alike; each unhappy family is unhappy in its own way.” A curious/learning-first AI might make many discoveries about happy humans while it is building up power, and then start putting humans in a greater number of awful but novel and “interesting” situations once it doesn’t need humanity to survive.
That said, this is only a problem if the AI is likely to not be empathetic/compassionate, which if I’m not mistaken, is one of the main things we would disagree on. I think that instead of trying to find these technical workarounds, you should argue for the much more interesting (and important!) position that AIs are likely to be empathetic and compassionate by default.
If instead you do want to be more persuasive with these workarounds, can I suggest adopting more of a security mindset? You appear to be looking for ways in which things can possibly go right, instead of all the ways things can go wrong. Alternatively, you don’t appear to be modeling the doomer mindset very well, so you can’t “put on your doomer hat” and check whether doomers would see your proposal as persuasive. Understanding a different viewpoint in depth is a big ask, but I think you’d find more success that way.