I agree that it’s a tricky problem, but I think it’s probably tractable. The way PreDCA tries to deal with these difficulties is:
The AI can tell that, even before the AI was turned on, the physical universe was running certain programs.
Some of those programs are “agentic” programs.
Agentic programs have approximately well-defined utility functions.
Disassembling the humans doesn’t change anything, since it doesn’t affect the programs that were already running[1] before the AI was turned on.
Since we’re looking at agent-programs rather than specific agent-actions, there is much more ground for inference about novel situations.
Obviously, the concepts I’m using here (e.g. which programs are “running” or which programs are “agentic”) are non-trivial to define, but infra-Bayesian physicalism does allow us the define them (not without some caveats, but hopefully at least to a 1st approximation).
I agree that it’s a tricky problem, but I think it’s probably tractable. The way PreDCA tries to deal with these difficulties is:
The AI can tell that, even before the AI was turned on, the physical universe was running certain programs.
Some of those programs are “agentic” programs.
Agentic programs have approximately well-defined utility functions.
Disassembling the humans doesn’t change anything, since it doesn’t affect the programs that were already running[1] before the AI was turned on.
Since we’re looking at agent-programs rather than specific agent-actions, there is much more ground for inference about novel situations.
Obviously, the concepts I’m using here (e.g. which programs are “running” or which programs are “agentic”) are non-trivial to define, but infra-Bayesian physicalism does allow us the define them (not without some caveats, but hopefully at least to a 1st approximation).
More precisely, I am looking at agents which could prevent the AI from becoming turned on, this is what I call “precursors”.