I’m writing a book about epistemology. It’s about The Problem of the Criterion, why it’s important, and what it has to tell us about how we approach knowing the truth.
I’ve also written a lot about AI safety. Some of the more interesting stuff can be found at the site of my currently-dormant AI safety org, PAISRI.
Yes, this is why I like the movie better than the short story. PKD did more of what Total Recall did in other stories, like Ubik and A Scanner Darkly and The Man Who Japed, but never sends it fully the way Total Recall does.
Quest (1984)
This movie was written by Ray Bradbury.
It’s about people who have 8 day lifespans, and follows the story of a boy who grows up to fulfill a great quest. I like it from a rationalist standpoint because it has themes similar to those we have around AI, life extension, and more: we have a limited to achieve something, and if we don’t pull it off we are at least personally doomed, and maybe societally, too.
PT Barnum (1999)
This is a made for TV movie that can easily be found for free on YouTube.
I like it because it tells a somewhat fictionalized account of PT Barnum’s life that shows him as an expert in understanding the psychology of people and figuring out how to give them products they’ll love. Some might say what he does is exploitative, but the movie presents him as not much different than modern social media algorithms that give us exactly what we want, even if we regret it in hindsight.
The rationalist angle is coming away with a sense of what’s it’s like to be a live player who is focused on achieving something and in deep contact with reality to achieve it, willing to ignore social scripts in order to get there.
Total Recall (1990)
Based on the Phillip K. Dick short story “We Can Remember It For You Wholesale”. The movie is better than the short story.
I can’t tell you why this is a rationality movie without spoilers...
The movie masterfully sucks you into a story where you don’t know if you’re watching what’s really happening, or if you’re watching the false memories inserted into the protagonists mind at the start of the film. Much of the fun for rationalists would be trying to figure out if the film was reality or implanted memory.
It’s not quite like the dot com bust. Bottom of the market is very soft, with new grads basically having no options, but the top of the market is extremely tight, with the middle doing about like normal. Employers feel they can be more choosy right now for all roles, though, so they are being. That will change if roles sit unfilled for longer.
How would you compare your ideas here to Asimov’s fictional science of psychohistory? I ask because while reading this post I kept getting flashbacks to Foundation.
Yes, red is perhaps the most useful to color to be able to see! That’s why I chose to use it in this example.
I don’t know, but I can say that after a lot of hours of Alexander lessons my posture and movement improved in ways that would be described as “having less muscle tension” and this having less tension happened in conjunction with various sorts of opening and being more awake and moving closer to PNSE.
Thank you for sharing your thoughts, and sorry for your losses. It’s often hard to talk about death, especially about the deaths of those we love. I don’t really have anything other to say than that I found this moving to read, and I’m glad you shared it with us.
Here’s more answer than you probably wanted.
First up, the word “epistemic” solves a limitation of the word “knowledge” in that it doesn’t easily turn into an adjective. Yes, like all nouns in English it can be used like an adjective in the creation of noun phrases, but “knowledge state” and “knowledge status” don’t sound as good.
But more importantly there’s a strong etymological reason to prefer the word “epistemic” in these cases. “Epistemic” comes from “episteme”, one of Greek’s words for knowledge[1]. Episteme is knowledge that is justified by observation and reason, and importantly is known because the knower was personally convinced of the justification, as opposed to gnosis, where the only justification is experience, or doxa, which is second-hand knowledge[2].
Thus “epistemic” carries with it the connotation of being related to justified beliefs. An “epistemic state” or “epistemic status” implies a state or status related to how justified one’s beliefs are.
“Knowledge” is cognate with another Greek word for knowledge, “gnosis”, but the two words evolved along different paths from PIE *gno-, meaning “know”.
We call doxa “hearsay” in English, but because of that word’s use in legal contexts, it carries some pejorative baggage related to how hearsay is treated in trials. To get around this we often avoid the word “hearsay” and instead focus on our level of trust in the person we learned something from, but won’t make a clear distinction between hearsay and personally justified knowledge.
I’m sure my allegiance to these United States was not created just by reciting the Pledge thousands of times. In fact, I resented the Pledge for a lot of my life, especially once I learned more about its history.
But if I’m honest with myself, I do feel something like strong support for the ideals of the United States, much stronger than would make sense if someone had convinced me as an adult that its founding principals were a good idea. The United States isn’t just my home. I yearn for it to be great, to embody its values, and to persist, even as I disagree with many of the details of how we’re implementing the dream of the founders today.
Why do I think the Pledge mattered? It helped me get the feeling right. Once I had positive feelings about the US, of course I wanted to actually like the US. I latched onto the part of it that resonates with me: the founding principals. Someone else might be attracted to something else, or maybe would even find they don’t like the United States, but stay loyal to it because they have to.
I’m also drawing on my experience with other fake-it-until-you-make-it rituals. For example, I and many people really have come to feel more grateful for the things we have in life by explicitly acknowledge that gratitude. At the start it’s fake: you’re just saying words. But eventually those words start to carry meaning, and before long it’s not fake. You find the gratitude that was already inside you and learn how to express it.
In the opening example, I bet something similar could work for getting kids to appologize. No need to check if they are really sorry, just make them say sorry. Eventually the sadness at having caused harm will become real and flow into the expression of it. It’s like a kind of reverse training, where you create handles for latent behaviors to crystalize around, and by creating the right conditions when the ritual is performed, you stand a better-than-chance possibility of getting the desired association.
Some cultures used to, and maybe still do, have a solution to the hostile telepaths problem you didn’t list: perform rituals even if you don’t mean them.
If a child breaks their mom’s glasses, the mom doesn’t care if they are really sorry or not. All she cares about is if they perform the sorry-I-broke-your-glasses ritual, whatever that looks like. That’s all that’s required.
The idea is that the meaning comes later. We have some non-central instances of this in Western culture. For example, most US school children recite the Pledge of Allegiance every day (or at least they used to). I can remember not fully understanding what the words meant until I was in middle school, but I just went along with it. And wouldn’t you know it, it worked! I do have an allegiance to the United States as a concept.
The world used to be more full of these rituals and strategies for appeasing hostile telepaths, who just chose not to use their telepathy because everyone agreed it didn’t matter so long as the rituals were performed. But the spread of Christianity and Islam has brought a demand for internalized control of behaviors to much of the world, and with it we get problems like shame and guilt.
Now I’m not saying that performing rituals even if you don’t mean them is a good solution. There are a lot of tradeoffs to consider, and guilt and shame offer some societal benefits that enable higher trust between strangers. But it is an alternative solution, and one that, as my Pledge of Allegiance example suggests, does sometimes work.
Many ideas are hard to fully express in words. Maybe no idea can be precisely and accurately captured. Something is always left out when we use our words.
What I think makes some people faster (and arguably better) writers is that they natively think in terms of communication with others, whereas I natively think in terms of world modeling, and then try to come up with words that explain the word model. They don’t have to go through a complex thought process to figure out how to transmit their world model to others, because they just say thing that convey the messages that exist in their head, and those messages are generated based on their model of the world.
Yep! In fact, an earlier draft of this post included a mention of Paul Graham, because he’s a popular and well-liked example of someone who has a similar process to the one I use (though I don’t know if he does it for the same reasons).
In that earlier draft, I contrasted Graham with Scott Alexander, who I vaguely recall mentioning that he basically sits down at his computer and a couple hours later a finish piece of writing has appeared. But I couldn’t find a good reference of this being Scott’s process, so maybe it’s just a thing I talked with him about in person one time.
In the end I decided this was an unnecessary tangent for the body of the text, but I’m very glad to have a chance to talk about it in the comments! Thanks!
As of late July last year, “I” am in PNSE. A few comments.
First, no major errors or concerns when reading the post. I might have missed something, but nothing triggered the “this is misunderstanding what PNSE is fundamentally like” alarm.
Second, there’s a lot of ways PNSE is explained. I like this short version: “I am me”. That is, “I”, the subject of experience, no longer experiences itself as subject, but rather as object, i.e. “me”. It’s like having a third-person experience of the self. I also like to describe it as thought becoming a sense, like vision or hearing, because “I” no longer do the thinking; instead this person does the thinking to me.
Third, not everyone describes it this way, but in Zen we call the transition into PNSE the Great Death because it literally feels like dying. It’s not dissimilar from the ego death people experience on drugs like LSD, but ego “death” is better described as ego “sleep” because it comes back and, after it’s happened once, the mind knows the ego is going to come back, whereas in the Great Death the sense of separate self is gone and not coming back. All that said, many with PNSE don’t experience a violent transition like this, so the Great Death or something like it may be a contingent feature of some paths to PNSE and not others.
Fourth, I don’t remember if the paper discusses this, and this is controversial among some Buddhist traditions, but PNSE doesn’t mean the mind is totally liberated from belief in a separate self. You said the homunculus concept lies dormant, but I’d say it does more than that. The mind is filled with many beliefs that presupposed the existence of the homunculus, and even if the homunculus is no longer part of experiences of the world, it’s still baked into habits of behavior, and it takes significant additional work once in PNSE to learn new habits to replace the old ones that don’t have the homunculus baked into them. Very few people ever become free of all of them, and maybe literally no one does as long as they continue to live.
Fifth and finally, PNSE is great, I’m glad it’s how I am now. It’s also fine not to be in it, because even if you believe you have a homunculus, in an absolute sense you already don’t, you’re just confused about how the world works, and that’s okay, we’re all confused. PNSE is also confused, but in different ways, and with fewer layers of confusion. So if you read this post and are now excited to try for PNSE, great, do it, but be careful. Lots of people Goodhart on what they think PNSE is because they try too hard to get it. If PNSE doesn’t sneak up on you, then be extra suspect of Goodharting! (Actually, just always be suspicious that you’ve Goodharted yourself!)
The information/assurance split feels quite familiar to me as an engineering manager.
My work life revolves around projects, especially big projects that takes months to complete. Other parts of the business depend on when these projects will be done. In some cases, the entire company’s growth plans may hinge on my team completing a project by a certain time. And so everyone wants as much assurance as possible about when projects will complete.
This makes it really hard to share information, because people are so hungry for assurance they will interpret almost any sharing of information as assurance. A typical conversation I used to have when I was naive to this fact:
Sales manager: Hey, Gordon, when do you think that project will be done?
Me: Oh, if things go according to plan, probably next month.
Sales manager: Cool, thanks for the update!
If the project ships next month, no problem. But as often happens in software engineering, if the project gets delayed, now the sales manager is upset:
Them: Hey, you said it would be ready next month. What gives?
Me: I said if things went according to plan, but there were surprises, so it took us longer than we initially though it would.
Them: Dammit. I sold a customer on the assumption that the project was shipping this month! What am I supposed to tell them now?
Me: I don’t know, why did you do that? I was giving you an internal estimate, not a promise of delivery.
Them: You said this month. I’m tired of Engineering always having some excuse about why stuff is delayed.
What did I do wrong? I failed to understand that Sales, and most other functions in a software business, are so dependent and hungry for information from Engineering, that they saw the assurance they wanted to see rather than the information I was giving.
I’ve (mostly) learned my lesson. I have to carefully control how much I say to anyone not directly involved in the project, lest they get the wrong idea.
Someone: Hey, Gordon, when do you think that project will be done?
Me: We’re working on it. We set a goal of having it complete by end of next quarter.
Do I actually expect it to take all the way to next quarter? No. Most likely it’ll be done next month. But if anything unexpected happens, now I’ve given a promise I can keep.
This isn’t exactly just “underpromise, overdeliver”. That’s part of it, but it’s also about noticing when you’re accidentally making a promise, even when you think you’re not, even if you say really explicitly that you’re not making a promise, someone will interpret as a promise and now you’ll have to deal with that.
I defined tool AI specifically as controllable, so AI without a quantitative guarantee that it’s controllable (or “safe”, as you write) wouldn’t meet the safety standards and its release would be prohibited.
If your stated definition is really all you mean by tool AI, then you’ve defined tool AI in a very nonstandard way that will confuse your readers.
When most people hear “tool AI”, I expect them to think of AI like hammers: tools they can use to help them achieve a goal, but aren’t agentic and won’t do anything on their own they weren’t directly asked to do.
You seem to have adopted a definition of “tool AI” that actually means “controllable and goal-achieving AI”, but give no consideration to agency, so I can only conclude from your writing that you would mean for AI agents to be included as tools, even if they operated independently, so long as they could be controlled in some sense (what sense control takes exactly you never specify). This is not what I expect most people to expect someone to mean by a “tool”.
Again, I like all the reasoning about entente, but this use of the word “tool AI” is confusing, maybe even deceptive (I assume that was not the intent!). It also leaves me felling like your “solution” of tool AI is nothing other than a rebrand of what we’ve already been talking about in the field variously as safe, aligned, or controllable AI, which I guess is fine, but “tool AI” is a confusing name for that. This also further downgrades my opinion of the solution section, since as best I can tell it’s just saying “build AI safely” without enough details to be actionable.
What do you make of the extensive arguments that tool AI are not actually safer than other forms of AI, and only look that way on the surface by ignoring issues of instrumental convergence to power-seeking and the capacity for tool AI to do extensive harm even if under human control? (See the Tool AI page for links to many posts tackling this question from different angles.)
(Also, for what it’s worth, I was with you until the Tool AI part. I would have liked this better if it had been split between one post arguing what’s wrong with entente and one post arguing what to do instead.)
I agree with the main claim of this post, mostly because I came to the same conclusion several years ago and have yet to have my mind changed away from it in the intervening time. If anything, I’m even more sure that values are after-the-fact reifications that attempt to describe why we behave the way we do.
I like this question a lot, but I’m more interested in its opposite, so I asked it!