I suspect that the observed behaviors will feel normal—even if the goals and motives are inscrutable, humans will infer more pedestrian goals from the actions they see. In other words, the motivations may be unnatural or weird, but the actions will still be actions, and we’ll make up reasons that we can believe.
For instance, the AGI may cure cancer by making our DNA more resistant to mutation, because they fear uncontrolled variance in their pets. We’ll only notice the cure, not the motive.
I suspect that the observed behaviors will feel normal—even if the goals and motives are inscrutable, humans will infer more pedestrian goals from the actions they see. In other words, the motivations may be unnatural or weird, but the actions will still be actions, and we’ll make up reasons that we can believe.
For instance, the AGI may cure cancer by making our DNA more resistant to mutation, because they fear uncontrolled variance in their pets. We’ll only notice the cure, not the motive.