Relatedly, CoEms could be run at potentially high speed-ups, and many copies or variations could be run together. So we could end up in the classic scenario of a smarter-than-average “civilization”, with “thousands of years” to plan, that wants to break out of the box.
This still seems less existentially risky, though, if we end up in a world where the CoEms retain something approximating human values. They might want to break out of the box, but probably wouldn’t want to commit species-cide on humans.
As fas as I understand, the point on this proposal is that “human-like cognitive architecture ≈ cognitive containability ≈ sort of safety”, not “human-like cognitive architecture ≈ human values”. I just want to say that even human can be cognitively uncontainable relatively to another human, because they can learn mental tricks that look to another human as Magic.
Relatedly, CoEms could be run at potentially high speed-ups, and many copies or variations could be run together. So we could end up in the classic scenario of a smarter-than-average “civilization”, with “thousands of years” to plan, that wants to break out of the box.
This still seems less existentially risky, though, if we end up in a world where the CoEms retain something approximating human values. They might want to break out of the box, but probably wouldn’t want to commit species-cide on humans.
As fas as I understand, the point on this proposal is that “human-like cognitive architecture ≈ cognitive containability ≈ sort of safety”, not “human-like cognitive architecture ≈ human values”. I just want to say that even human can be cognitively uncontainable relatively to another human, because they can learn mental tricks that look to another human as Magic.