A mind can perform original seeing (to various degrees), and it can also use cached thoughts.
Cache thoughts are more “Procedural instruction manuals” and original seeing is more “Your true anticipations of reality”.
Both reality and social reality (social improv web) apply pressures and rewards that shape your cached thoughts.
It often looks like people can be said to have motives/agendas/goals, because their cached thoughts have been formed by the pressures of the social improv web.
Ex. Tom has a cached thought, the execution of which results in “People Like Tom”, which makes it look reasonable to assert “Tom’s motives are for people to like him”.
People are Cached-thought-executors, not Utility-maximizers/agenda-pursuers.
One can switch from acting from cached thoughts, to acting from original seeing without ever realizing a switch happened.
Motte and bailey doesn’t have to be intentional.
When talking with someone and applying pressure to their beliefs, it no longer becomes effective to chase down their “motives”/cached thoughts, because they’ve switched to a weak form of original seeing, andin that moment effectively no longer have the “motives” they had a few moments ago.
Tentatively dubbing this the Schrodinger’s Agenda.
Just wanted to say I liked the core insight here (that people seem more-like-hidden-agenda executors when they’re running on cached thoughts). I think it probably makes more sense to frame it as a hypothesis than a “this is a true thing about how social reality and motivation work”, but a pretty good hypothesis. I’d be interested in the essay exploring what evidence. might falsify it or reinforce it.
(This is something that’s not currently a major pattern among rationalist thinkpieces on psychology but probably should be)
hmmmmm, ironically my immediate thought was, “Well of course I was considering it as a hypothesis which I’m examining the evidence for”, though I’d bet that the map/territory separation was not nearly as emphasized in my mind when I was generating this idea.
Yeah, I think your framing is how I’ll take the essay.
Here’s a more refined way of pointing out the problem that the parent comment was addressing:
I am a general intelligence that emerged running on hardware that wasn’t intelligently designed for general intelligence.
Because of the sorts of problems I’m able to solve when directly applying my general intelligence (and because I don’t understand intelligence that well), it is easy to end up implicitly believing that my hardware is far more intelligent than it actually is.
Examples of ways my hardware is “sub-par”:
It don’t seem to get automatic belief propagation.
There doesn’t seem to be strong reasons to expect that all of my subsystems are guaranteed to be aligned with the motives that I have on a high level.
Because there are lots of little things that I implicitly believe my hardware does, which it does not, there are a lot of corrective measures I do not take to solve the deficiencies I actually have.
It’s completely possible that my hardware works in such a way that I’m effectively working on different sets of beliefs and motives and various points in time, and I have a bias towards dismissing that because, “Well that would be stupid, and I am intelligent.”
Another perspective. I’m thinking about all of the examples from the sequences of people near Eliezer thinking that AI’s would just do certain things automatically. It seems like that lens is also how we look at ourselves.
I’m in the process of turning this thought into a full essay.
Ideas that are getting mixed together:
Cached thoughts, Original Seeing, Adaption Executors not Fitness Maximizes, Motte and Bailey, Double Think, Social Improv Web.
A mind can perform original seeing (to various degrees), and it can also use cached thoughts.
Cache thoughts are more “Procedural instruction manuals” and original seeing is more “Your true anticipations of reality”.
Both reality and social reality (social improv web) apply pressures and rewards that shape your cached thoughts.
It often looks like people can be said to have motives/agendas/goals, because their cached thoughts have been formed by the pressures of the social improv web.
Ex. Tom has a cached thought, the execution of which results in “People Like Tom”, which makes it look reasonable to assert “Tom’s motives are for people to like him”.
People are Cached-thought-executors, not Utility-maximizers/agenda-pursuers.
One can switch from acting from cached thoughts, to acting from original seeing without ever realizing a switch happened.
Motte and bailey doesn’t have to be intentional.
When talking with someone and applying pressure to their beliefs, it no longer becomes effective to chase down their “motives”/cached thoughts, because they’ve switched to a weak form of original seeing, and in that moment effectively no longer have the “motives” they had a few moments ago.
Tentatively dubbing this the Schrodinger’s Agenda.
Just wanted to say I liked the core insight here (that people seem more-like-hidden-agenda executors when they’re running on cached thoughts). I think it probably makes more sense to frame it as a hypothesis than a “this is a true thing about how social reality and motivation work”, but a pretty good hypothesis. I’d be interested in the essay exploring what evidence. might falsify it or reinforce it.
(This is something that’s not currently a major pattern among rationalist thinkpieces on psychology but probably should be)
hmmmmm, ironically my immediate thought was, “Well of course I was considering it as a hypothesis which I’m examining the evidence for”, though I’d bet that the map/territory separation was not nearly as emphasized in my mind when I was generating this idea.
Yeah, I think your framing is how I’ll take the essay.
Here’s a more refined way of pointing out the problem that the parent comment was addressing:
I am a general intelligence that emerged running on hardware that wasn’t intelligently designed for general intelligence.
Because of the sorts of problems I’m able to solve when directly applying my general intelligence (and because I don’t understand intelligence that well), it is easy to end up implicitly believing that my hardware is far more intelligent than it actually is.
Examples of ways my hardware is “sub-par”:
It don’t seem to get automatic belief propagation.
There doesn’t seem to be strong reasons to expect that all of my subsystems are guaranteed to be aligned with the motives that I have on a high level.
Because there are lots of little things that I implicitly believe my hardware does, which it does not, there are a lot of corrective measures I do not take to solve the deficiencies I actually have.
It’s completely possible that my hardware works in such a way that I’m effectively working on different sets of beliefs and motives and various points in time, and I have a bias towards dismissing that because, “Well that would be stupid, and I am intelligent.”
Another perspective. I’m thinking about all of the examples from the sequences of people near Eliezer thinking that AI’s would just do certain things automatically. It seems like that lens is also how we look at ourselves.
Or it could humans are not automatically strategic, but on steroids. Humans do not automatically get great hardware.