Here’s a more refined way of pointing out the problem that the parent comment was addressing:
I am a general intelligence that emerged running on hardware that wasn’t intelligently designed for general intelligence.
Because of the sorts of problems I’m able to solve when directly applying my general intelligence (and because I don’t understand intelligence that well), it is easy to end up implicitly believing that my hardware is far more intelligent than it actually is.
Examples of ways my hardware is “sub-par”:
It don’t seem to get automatic belief propagation.
There doesn’t seem to be strong reasons to expect that all of my subsystems are guaranteed to be aligned with the motives that I have on a high level.
Because there are lots of little things that I implicitly believe my hardware does, which it does not, there are a lot of corrective measures I do not take to solve the deficiencies I actually have.
It’s completely possible that my hardware works in such a way that I’m effectively working on different sets of beliefs and motives and various points in time, and I have a bias towards dismissing that because, “Well that would be stupid, and I am intelligent.”
Another perspective. I’m thinking about all of the examples from the sequences of people near Eliezer thinking that AI’s would just do certain things automatically. It seems like that lens is also how we look at ourselves.
Here’s a more refined way of pointing out the problem that the parent comment was addressing:
I am a general intelligence that emerged running on hardware that wasn’t intelligently designed for general intelligence.
Because of the sorts of problems I’m able to solve when directly applying my general intelligence (and because I don’t understand intelligence that well), it is easy to end up implicitly believing that my hardware is far more intelligent than it actually is.
Examples of ways my hardware is “sub-par”:
It don’t seem to get automatic belief propagation.
There doesn’t seem to be strong reasons to expect that all of my subsystems are guaranteed to be aligned with the motives that I have on a high level.
Because there are lots of little things that I implicitly believe my hardware does, which it does not, there are a lot of corrective measures I do not take to solve the deficiencies I actually have.
It’s completely possible that my hardware works in such a way that I’m effectively working on different sets of beliefs and motives and various points in time, and I have a bias towards dismissing that because, “Well that would be stupid, and I am intelligent.”
Another perspective. I’m thinking about all of the examples from the sequences of people near Eliezer thinking that AI’s would just do certain things automatically. It seems like that lens is also how we look at ourselves.
Or it could humans are not automatically strategic, but on steroids. Humans do not automatically get great hardware.