Yeah, the “self-improving agent”, “simulate powerful agents”, “self-fulfilling prophecies” and “oracles or predictors may become agents” were all meant to be examples of computation hazards, and those doesn’t necessarily involved simulating people.
Ah, I was thinking of “computational hazard” as meaning the computation itself is bad, not its consequences on the computing substrate or outside environment. I thought a “self-improving agent” was an example of something that might compute a hazard as a result of computing lots of stuff, some of which turns out to be hazardous. But short of instantiating that particular computational hazard, I don’t think it does bad merely by computation, rather the computation helps it direct its actions to achieve bad consequences.
Yeah, the “self-improving agent”, “simulate powerful agents”, “self-fulfilling prophecies” and “oracles or predictors may become agents” were all meant to be examples of computation hazards, and those doesn’t necessarily involved simulating people.
Ah, I was thinking of “computational hazard” as meaning the computation itself is bad, not its consequences on the computing substrate or outside environment. I thought a “self-improving agent” was an example of something that might compute a hazard as a result of computing lots of stuff, some of which turns out to be hazardous. But short of instantiating that particular computational hazard, I don’t think it does bad merely by computation, rather the computation helps it direct its actions to achieve bad consequences.
I think I agree.