I agree that non-universal-optimizers are not necessarily safe. There’s a reason I wrote “many” not “all” canonical arguments. In addition to gaming the system, there’s also the time honored technique of rewriting the rules. I’m concerned about possible feedback loops. Evolution brought about the values we know and love in a very specific environment. If that context changes while evolution accelerates, I foresee a problem.
I agree that non-universal-optimizers are not necessarily safe. There’s a reason I wrote “many” not “all” canonical arguments. In addition to gaming the system, there’s also the time honored technique of rewriting the rules. I’m concerned about possible feedback loops. Evolution brought about the values we know and love in a very specific environment. If that context changes while evolution accelerates, I foresee a problem.