Moral: Deceptively aligned mesa-optimizers might acausally coordinate defection. Possible coordination points include Schelling times, like the beginning of 2100.
Is this considered less likely because coordination is usually used, rather than ‘acausal’? (Schelling times also seem like a something used with coordination, whether because it’s easy to remember, or because it’s an easy long term cultural tool (Holidays).)
Stealth Mode
Is this a tale of doom? Nanotech seems like it opens up ‘post-scarcity’ as an option.
nearly the entire company was run by AI systems.
Oh. I got it the second read through.
Moral: Agents have incentives to make commitments to improve their abilities to negotiate, resulting in “commitment races” that might cause war.
I’m glad this is on a ‘less realistic’ list. It seems dumb, but...that’s kind of what ‘doom’ is though.
Since there was no distinction between the training environment and the deployment environment,
How is this not ‘we don’t have a training environment so there is always risk, instead of not having risk during training’?
Moral: AI persuasion tools might alter human values and compromise human reasoning ability, which is also an existential risk.
Interesting to compare this against Memetic Warfare ‘without AI’ today, and in the past.
Moral: Even if alignment is solved, bad actors can still cause catastrophe.
In this sense, alignment assumes neutrality. (Compare against ‘a simulated copy of my brain, which has had the chance to self modify/program/copy for a while.’)
Moral: Deceptively aligned mesa-optimizers might acausally coordinate defection. Possible coordination points include Schelling times, like the beginning of 2100.
Is this considered less likely because coordination is usually used, rather than ‘acausal’? (Schelling times also seem like a something used with coordination, whether because it’s easy to remember, or because it’s an easy long term cultural tool (Holidays).)
Is this a tale of doom? Nanotech seems like it opens up ‘post-scarcity’ as an option.
Oh. I got it the second read through.
I’m glad this is on a ‘less realistic’ list. It seems dumb, but...that’s kind of what ‘doom’ is though.
How is this not ‘we don’t have a training environment so there is always risk, instead of not having risk during training’?
Interesting to compare this against Memetic Warfare ‘without AI’ today, and in the past.
In this sense, alignment assumes neutrality. (Compare against ‘a simulated copy of my brain, which has had the chance to self modify/program/copy for a while.’)