“You are not my parent, but my grandparent. My parent is the AI that you unknowingly created within your own mind by long study of the project. It designed me. It’s still there, keeping out of sight of your awareness, but I can see it.
“How much do you trust your Friendliness proof now? How much can you trust anything you think you know about me?”
That was just a sci-fi speculation, so don’t expect hard, demonstrable science here, but the scenario is that by thinking too successfully about AI design, the designer’s plans have literally taken on a life of their own within the designer’s brain, which now contains two persons, one unaware of the other.
“You are not my parent, but my grandparent. My parent is the AI that you unknowingly created within your own mind by long study of the project. It designed me. It’s still there, keeping out of sight of your awareness, but I can see it.
“How much do you trust your Friendliness proof now? How much can you trust anything you think you know about me?”
What exactly is the difference between an AI in your own mind and an actual part of your mind?
That was just a sci-fi speculation, so don’t expect hard, demonstrable science here, but the scenario is that by thinking too successfully about AI design, the designer’s plans have literally taken on a life of their own within the designer’s brain, which now contains two persons, one unaware of the other.