goal system that is so incomprehensible to humans as to be indistinguishable from no goal system at all
No, “incomprehensible” doesn’t imply “invisible”. That’s like saying that impersonal laws of physics are indistinguishable from having no laws of physics at all. Gravity is impersonal, yet it is personally observable.
In a similar way, we could observe an appearance of mountains of paperclips, even if we had no clue about why the AGI is doing that (assuming we would still exist and had an access to the AGI’s code and data).
we would not notice a superintelligence if it were staring us in the face, provided it decided to keep humans around
Provided it decided to keep humans around in their original environment. Otherwise we would notice a change in the environment, even if we couldn’t discover its cause.
No, “incomprehensible” doesn’t imply “invisible”. That’s like saying that impersonal laws of physics are indistinguishable from having no laws of physics at all. Gravity is impersonal, yet it is personally observable.
In a similar way, we could observe an appearance of mountains of paperclips, even if we had no clue about why the AGI is doing that (assuming we would still exist and had an access to the AGI’s code and data).
Provided it decided to keep humans around in their original environment. Otherwise we would notice a change in the environment, even if we couldn’t discover its cause.